<?xml version="1.0" encoding="UTF-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
    <title>robots.txt</title>
    <link rel="self" type="application/atom+xml" href="https://links.biapy.com/guest/tags/3237/feed"/>
    <updated>2026-06-18T06:30:51+00:00</updated>
    <id>https://links.biapy.com/guest/tags/3237/feed</id>
            <entry>
            <id>https://links.biapy.com/links/12618</id>
            <title type="text"><![CDATA[Is Your Site Agent-Ready?]]></title>
            <link rel="alternate" href="https://isitagentready.com/" />
            <link rel="via" type="application/atom+xml" href="https://links.biapy.com/links/12618"/>
            <author>
                <name><![CDATA[Biapy]]></name>
            </author>
            <summary type="text">
                <![CDATA[Scan your website to see how ready it is for AI agents. We check multiple emerging standards — from robots.txt and Markdown negotiation to MCP, OAuth, Agent Skills and agentic commerce.]]>
            </summary>
            <updated>2026-04-24T13:34:20+00:00</updated>
        </entry>
            <entry>
            <id>https://links.biapy.com/links/10818</id>
            <title type="text"><![CDATA[Herr Bischoff&amp;#039;s Bot Database]]></title>
            <link rel="alternate" href="https://badbot.org/" />
            <link rel="via" type="application/atom+xml" href="https://links.biapy.com/links/10818"/>
            <author>
                <name><![CDATA[Biapy]]></name>
            </author>
            <summary type="text">
                <![CDATA[Please find below a manually curated and researched list of users agents I came across. It&amp;#039;s impressive to see how many of the bots active today flat out do not respect robots.txt settings — or claim to do it but ignore them. This list is updated regularly, whenever I spot new user agents and look into their behavior. There is no JavaScript, here no fancy search.

Related contents:

- [Comment protéger vos serveurs et lutter efficacement contre les crawlers d’IA @ Bearstech :fr:](https://bearstech.com/societe/blog/comment-proteger-vos-serveurs-et-lutter-efficacement-contre-les-crawlers-dia).]]>
            </summary>
            <updated>2025-10-30T06:48:53+00:00</updated>
        </entry>
            <entry>
            <id>https://links.biapy.com/links/10495</id>
            <title type="text"><![CDATA[ai.robots.txt]]></title>
            <link rel="alternate" href="https://github.com/ai-robots-txt/ai.robots.txt" />
            <link rel="via" type="application/atom+xml" href="https://links.biapy.com/links/10495"/>
            <author>
                <name><![CDATA[Biapy]]></name>
            </author>
            <summary type="text">
                <![CDATA[A list of AI agents and robots to block. 

This list contains AI-related crawlers of all types, regardless of purpose. We encourage you to contribute to and implement this list on your own site. See information about the listed crawlers and the FAQ.

Related contents:

- [\#119: Les news sur le développement web et l&amp;#039;IA pour septembre 2025 RC2 @ Double Slash :fr:](https://double-slash.dev/podcasts/news-sept25-rc2/).
- [Comment bloquer les crawlers IA qui pillent votre site sans vous demander la permission ? @ Korben :fr:](https://korben.info/bloquer-crawlers-ia-robots-txt-htaccess-nginx.html).]]>
            </summary>
            <updated>2025-12-16T10:41:09+00:00</updated>
        </entry>
            <entry>
            <id>https://links.biapy.com/links/10167</id>
            <title type="text"><![CDATA[Really Simple Licensing (RSL)]]></title>
            <link rel="alternate" href="https://rslstandard.org/" />
            <link rel="via" type="application/atom+xml" href="https://links.biapy.com/links/10167"/>
            <author>
                <name><![CDATA[Biapy]]></name>
            </author>
            <summary type="text">
                <![CDATA[The open content licensing standard
for the AI-first Internet

Really Simple Licensing (RSL) is an evolution of the early ideas behind the widely adopted RSS standard, which provided a machine-readable framework for publishers to syndicate content to third-party clients and crawlers in exchange for traffic.

- [RSL Collective homepage](https://rslcollective.org/).

Related contents:

- [Pay-per-output? AI firms blindsided by beefed up robots.txt instructions. @ Ars Technica](https://arstechnica.com/tech-policy/2025/09/pay-per-output-ai-firms-blindsided-by-beefed-up-robots-txt-instructions/).]]>
            </summary>
            <updated>2025-09-12T12:33:43+00:00</updated>
        </entry>
    </feed>
