Skip to content

AI visibility

How to let ChatGPT and Claude crawl my site

Answer engines can only cite pages their retrieval crawlers are allowed to fetch. Blocking GPTBot, ClaudeBot or PerplexityBot in robots.txt removes you from the answers they generate, and the block is usually copied from a template rather than chosen.

What 13x checks

This is rule aeo.ai-crawler-access in the public registry: Answer engines are allowed to retrieve. It runs on every audit, against the pages we actually fetched, and its result is derived from the response rather than estimated.

Surface
AI visibility
Score weight
14 of the readiness score
Scope
Runs on every audited page
Applies
To every site

Registry version 2026-07-30. Every rule is published, and the audit is deterministic — the same page produces the same finding every time.

The fix

The same text the audit hands you when this check fails on your own site.

Your robots.txt blocks these AI crawlers: .

Decide deliberately, then be explicit. To allow them, remove the blocking rules and state it:

Configuration
User-agent: *
Allow: /

# Answer engines — allowing these is what makes you citable
User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

Sitemap: https://example.com/sitemap.xml

Two things worth knowing before you decide:

- Google-Extended controls Gemini and AI Overviews only. It does not affect normal Google Search ranking, so blocking it costs AI visibility and gains nothing. - CCBot is Common Crawl, which most models train on. Blocking it is a reasonable choice if you object to training use; it is a separate decision from wanting to be cited today.

A rule naming a specific crawler overrides the * block entirely, so you can keep a restrictive default and allow only the ones you want.

Framework-specific versions

Where the change differs enough to be worth writing out separately.

Does your site have this problem?

13x checks this and 112 others against your live URL in about 30 seconds. No account, and every finding comes with the fix for your framework.

No signup. Results in 30 seconds.

More ai visibility checks