Biapy's Bookmarks

Reader

https://reader.dev/

ai apache2-licensed crawler foss markdown open-source readability scraping self-hosted web-app

Added 1 month ago

Crawl4AI

https://docs.crawl4ai.com/

ai apache2-licensed crawler mcp open-source scraping

Added 2 months ago

wxpath

https://github.com/rodricios/wxpath

crawler development foss mit-licensed open-source python scraping

Added 6 months ago

LibreCrawl

https://librecrawl.com/

crawler foss mit-licensed open-source self-hosted seo web-app

Added 8 months ago

SiteOne Crawler

https://crawler.siteone.io/

command-line crawler foss mit-licensed open-source seo

Added 10 months ago

SEOnaut

https://seonaut.org/

audit crawler foss mit-licensed open-source self-hosted seo web-app

Added 10 months ago

A Vocabulary For Expressing AI Usage Preferences

https://ietf-wg-aipref.github.io/drafts/draft-ietf-aipref-vocab.html?cf_target_id=_blank

ai crawler robot scraping standard

Added 11 months ago

The Web Robots Pages

https://www.robotstxt.org/

crawler documentation robot scraping seo standard web

Added 1 year ago

Photon

https://github.com/s0md3v/Photon

command-line crawler foss open-source osint

Added 1 year ago

Nepenthes

https://zadzmo.org/code/nepenthes/

crawler foss honeypot mit-licensed open-source scraping self-hosted web-application-firewall

Added 1 year ago

Common Crawl

https://commoncrawl.org/

crawler llm machine-learning non-profit rag scraping web-service

Added 1 year ago

Crawl4AI

https://crawl4ai.com/mkdocs/

ai crawler foss llm open-source rag scraping

Added 1 year ago

Crawlee

https://crawlee.dev/

crawler foss javascript library open-source scraping typescript web

Added 1 year ago

Crawlee for Python

https://crawlee.dev/python/

crawler development foss library open-source python scraping web

Added 1 year ago

Firecrawl

https://www.firecrawl.dev/

agpl3-licensed crawler data-science foss llm machine-learning open-source scraping

Added 1 year ago

Fess

https://fess.codelibs.org/

crawler elasticsearch enterprise opensearch open-source search-engine self-hosted

Added 2 years ago

Katana

https://github.com/projectdiscovery/katana

crawler data-science framework golang open-source software spider

Added 3 years ago

Greenflare SEO Web Crawler

https://greenflare.io/

crawler seo software

Added 4 years ago

Screaming Frog SEO Spider Website Crawler

https://www.screamingfrog.co.uk/seo-spider/

crawler development linux macos seo spider ubuntu web-design windows

Added 4 years ago

Scrapy

http://scrapy.org/

crawler development foss framework library open-source python scraping web

Added 13 years ago