Biapy's Bookmarks

Figranium

https://figranium.dev/

api-server browser-automation gpl3-licensed open-source scraping web-app

Added 3 days ago

Kitesurf

https://kitesurf.cloudflare.app/

browser-automation cloudflare cloudflare-workers scraping web-browser

Added 1 week ago

Obscura

https://obscura.sh/

ai apache2-licensed foss headless open-source scraping web-browser

Added 1 week ago

Accept: text/markdown

https://acceptmarkdown.com/

markdown scraping standard

Added 2 weeks ago

BetterWright

https://betterwright.com/

ai foss mit-licensed open-source playwright scraping web-browser

Added 4 weeks ago

Moli Browser

https://browser.lexmount.com/

ai apache2-licensed browser-automation foss mit-licensed open-source playwright puppeteer scraping web-browser

Added 1 month ago

wigolo

https://knockoutez.github.io/wigolo/

agpl3-licensed foss mcp open-source scraping self-hosted web

Added 1 month ago

👁️ Agent Reach :cn:

https://github.com/Panniantong/Agent-Reach/tree/main

china command-line foss mit-licensed open-source reddit scraping twitter youtube

Added 2 months ago

mimic

https://github.com/littledivy/mimic

ai foss library mit-licensed mitm open-source python scraping

Added 2 months ago

kage

https://kage.tamnd.com/

archive clone command-line foss html mit-licensed open-source scraping web

Added 2 months ago

.MD this page

https://github.com/Ademking/MD-This-Page

browser-addon chrome firefox foss markdown mit-licensed open-source scraping

Added 3 months ago

SerpApi

https://serpapi.com/

commercial google scraping web-service

Added 3 months ago

Reader

https://reader.dev/

ai apache2-licensed crawler foss markdown open-source readability scraping self-hosted web-app

Added 3 months ago

Crawl4AI

https://docs.crawl4ai.com/

ai apache2-licensed crawler mcp open-source scraping

Added 3 months ago

curl.md

https://curl.md/

agent-skills markdown readability scraping web-server

Added 4 months ago

Obscura

https://github.com/h4ckf0r0day/obscura

ai-agent apache2-licensed browser-automation foss headless lightweight open-source scraping web-browser

Added 4 months ago

Is Your Site Agent-Ready?

https://isitagentready.com/

ai-agent llm llms.txt robots.txt scraping web-service

Added 4 months ago

skim

https://github.com/noblepayne/skim

clojure converter foss html library markdown mit-licensed open-source scraping

Added 5 months ago

Defuddle

https://defuddle.md/

converter foss library markdown mit-licensed open-source readability scraping

Added 5 months ago

Scrapy

https://www.scrapy.org/

bsd3-licensed foss framework open-source python scraping

Added 5 months ago

CloakBrowser

https://cloakbrowser.dev/

browser-automation chromium foss mit-licensed open-source scraping web-browser

Added 5 months ago

haphash

https://github.com/dgl/haphash

ai bsd-licensed foss haproxy open-source scraping web-application-firewall

Added 6 months ago

Scrapling

https://scrapling.readthedocs.io/en/latest/

bsd3-licensed foss framework open-source python scraping

Added 6 months ago

Dembrandt

https://www.dembrandt.com/

color command-line css foss mit-licensed open-source scraping typography web-design

Added 6 months ago

Doppelganger

https://doppelgangerdev.com/

automation playwright scraping self-hosted source-available sul-licensed web-app web-browser

Added 7 months ago

wxpath

https://github.com/rodricios/wxpath

crawler development foss mit-licensed open-source python scraping

Added 7 months ago

shot-scraper

https://shot-scraper.datasette.io/en/stable/index.html

apache2-licensed command-line foss open-source scraping screenshot web

Added 8 months ago

iocaine

https://iocaine.madhouse-project.org/

ai firewall foss open-source scraping

Added 9 months ago

Markdown Printer

https://github.com/levz0r/markdown-printer

browser-addon chrome edge firefox foss markdown mit-licensed open-source scraping

Added 9 months ago

Herr Bischoff's Bot Database

https://badbot.org/

blocklist robot robots.txt scraping web-service

Added 10 months ago

ai.robots.txt

https://github.com/ai-robots-txt/ai.robots.txt

ai ai-agent apache blocklist caddy firewall foss mit-licensed open-source robots.txt scraping

Added 11 months ago

The /llms.txt file

https://llmstxt.org/

ai llm llms.txt scraping standard web

Added 1 year ago

mdream

https://github.com/harlan-zw/mdream

converter foss llm markdown mit-licensed open-source scraping

Added 1 year ago

dirsearch

https://github.com/maurosoria/dirsearch

foss gpl2-licensed open-source osint scraping web

Added 1 year ago

Byparr

https://github.com/ThePhaseless/Byparr

cloudflare foss gpl3-licensed open-source prowlarr scraping seedbox

Added 1 year ago

🤖 Botasaurus Framework 🤖

https://www.omkar.cloud/botasaurus/

foss framework mit-licensed open-source python scraping

Added 1 year ago

A Vocabulary For Expressing AI Usage Preferences

https://ietf-wg-aipref.github.io/drafts/draft-ietf-aipref-vocab.html?cf_target_id=_blank

ai crawler robot scraping standard

Added 1 year ago

Meka Agent

https://withmeka.com/

ai-agent foss mit-licensed open-source scraping web

Added 1 year ago

Fireplexity

https://tools.firecrawl.dev/fireplexity

ai firecrawl foss llm mit-licensed open-source scraping search-engine self-hosted

Added 1 year ago

The Web Robots Pages

https://www.robotstxt.org/

crawler documentation robot scraping seo standard web

Added 1 year ago

Maxun Cloud

https://www.maxun.dev/

agpl3-licensed foss no-code open-source scraping self-hosted web-app

Added 1 year ago

Scraperr

https://scraperr-docs.pages.dev/

foss mit-licensed no-code open-source scraping self-hosted unmaintained web-app

Added 1 year ago

Defuddle

https://kepano.github.io/defuddle/

foss javascript mit-licensed open-source readability scraping

Added 1 year ago

CleverBee

https://cleverb.ee/

agpl3-licensed ai ai-agent llm open-source playwright python scraping

Added 1 year ago

TeleGraphite

https://github.com/hamodywe/telegram-scraper-TeleGraphite

foss json mit-licensed open-source scraping telegram

Added 1 year ago

PriceBuddy

https://github.com/jez500/pricebuddy

e-commerce foss gpl3-licensed open-source scraping self-hosted web-app

Added 1 year ago

Anubis

https://anubis.techaro.lol/

firewall foss mit-licensed open-source scraping self-hosted web-application-firewall

Added 1 year ago

Fetcher MCP

https://github.com/jae-jae/fetcher-mcp

ai-agent foss llm mcp mit-licensed open-source playwright scraping web web-browser

Added 1 year ago

Lightpanda Browser

https://lightpanda.io/

agpl3-licensed automation browser-automation chrome foss headless lightweight open-source scraping web-browser

Added 1 year ago

Nepenthes

https://zadzmo.org/code/nepenthes/

crawler foss honeypot mit-licensed open-source scraping self-hosted web-application-firewall

Added 1 year ago

Common Crawl

https://commoncrawl.org/

crawler llm machine-learning non-profit rag scraping web-service

Added 1 year ago

DrissionPage官网 :cn:

https://drissionpage.cn/

automation china open-source python scraping web web-browser

Added 1 year ago

Crawl4AI

https://crawl4ai.com/mkdocs/

ai crawler foss llm open-source rag scraping

Added 1 year ago

AIL-Framework

https://github.com/supdevinci/ail-framework-docker

data-analytics foss open-source osint scraping security

Added 1 year ago

DataFuel

https://www.datafuel.dev/

commercial llm rag scraping web-service

Added 1 year ago

Darkdump

https://github.com/josh0xA/darkdump

command-line deep-web foss mit-licensed open-source osint python scraping

Added 1 year ago

Discount Bandit

https://discount-bandit.cybrarist.com/

e-commerce foss open-source product-tracker scraping self-hosted web-app

Added 1 year ago

simple-cloudflare-solver

https://github.com/nlevee/simple-cloudflare-solver

cloudflare foss open-source scraping seedbox self-hosted web-app

Added 1 year ago

GitDorker

https://github.com/obheda12/GitDorker

audit command-line github open-source pentest scraping secret security

Added 1 year ago

Scrapling

https://github.com/D4Vinci/Scrapling

foss library open-source python scraping web

Added 1 year ago