Skip to content

AI visibility

Should I block AI crawlers in robots.txt

Permitting training crawlers while refusing retrieval crawlers is backwards from any stance anyone actually holds: your content is absorbed into models, and the crawlers that would send readers back to you with attribution are turned away. Whichever side you want, both rules should point the same way.

What 13x checks

This is rule aeo.crawler-policy-inversion in the public registry: Retrieval and training policy point the same way. It runs on every audit, against the pages we actually fetched, and its result is derived from the response rather than estimated.

Surface
AI visibility
Score weight
5 of the readiness score
Scope
Runs on every audited page
Applies
Only where the market or the page shape makes it relevant

Registry version 2026-07-30. Every rule is published, and the audit is deterministic — the same page produces the same finding every time.

The fix

The same text the audit hands you when this check fails on your own site.

Your robots.txt allows to collect training data, and blocks from fetching pages to cite.

That is the opposite of both common stances. The agents are different jobs and they are usually confused for each other:

| Agent | What it does | Blocking it means | | --- | --- | --- | | GPTBot | Collects corpus for training | Do not train on my content | | OAI-SearchBot | Builds ChatGPT's search index | Do not cite me in ChatGPT | | ChatGPT-User | Fetches a link during a live chat | Do not follow links to me | | PerplexityBot | Builds Perplexity's answer index | Do not cite me in Perplexity | | Google-Extended | Gemini grounding and training | No effect on Google Search rank |

"Do not train on me, but do cite me" — the stance most publishers want:

Configuration
User-agent: *
Allow: /

# Retrieval: how readers find you and how you get attributed
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

# Training: opted out
User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Sitemap: https://example.com/sitemap.xml

"Open to everything" — swap the second block to `Allow: /`.

Either is defensible. What you have now is neither, and it reads as a copied block list rather than a decision. Note that robots.txt is honoured voluntarily; it states intent, it does not enforce it.

Does your site have this problem?

13x checks this and 112 others against your live URL in about 30 seconds. No account, and every finding comes with the fix for your framework.

No signup. Results in 30 seconds.

More ai visibility checks