Skip to content

AI visibility

Why AI crawlers get a different response than Google

robots.txt permission is only half the question — what matters is the status code your server actually returns to each agent. A bot-protection rule or a CDN filter can allow a crawler in robots.txt and still answer it with a 403, which is invisible unless something probes it as that agent.

What 13x checks

This is rule aeo.crawler-reachability in the public registry: AI crawlers actually receive the page. It runs on every audit, against the pages we actually fetched, and its result is derived from the response rather than estimated.

Surface
AI visibility
Score weight
12 of the readiness score
Scope
Runs on every audited page
Applies
Only where the market or the page shape makes it relevant

Registry version 2026-07-30. Every rule is published, and the audit is deterministic — the same page produces the same finding every time.

The fix

The same text the audit hands you when this check fails on your own site.

These agents did not receive your page: .

This is not a markup problem and you will not find it in robots.txt — the request never reached your application. Something in front of it answered first.

Where to look, in order:

1. Cloudflare → Security → Bots → *AI Scrapers and Crawlers*. This is enabled by default on many zones and blocks AI crawlers regardless of what your robots.txt says. If you want to be cited, turn it off, or add a WAF skip rule for the agents you allow. 2. Vercel → Firewall → *AI Bots* managed ruleset. Same behaviour, same fix. 3. AWS WAF → the AWSManagedRulesBotControlRuleSet CategoryAI labels. 4. Your own middleware. Search the repo for user-agent matching:

bash code
rg -i "user-agent|userAgent" --type ts -l | head

Verify with a real request rather than a dashboard toggle:

bash code
for ua in "PerplexityBot/1.0" "OAI-SearchBot/1.0" "ClaudeBot/1.0" "ChatGPT-User/1.0"; do
  printf '%-24s %s\n' "$ua" \
    "$(curl -s -o /dev/null -w '%{http_code}' -A "$ua" https://example.com)"
done

Every line should print 200. A 403, 429 or 503 is the block. Note that a challenge page often returns 200 with almost no text, so compare the byte count too:

bash code
curl -s -A "PerplexityBot/1.0" https://example.com | wc -c
curl -s https://example.com | wc -c

If the first number is a fraction of the second, the agent is being served an interstitial rather than your page.

Decide deliberately. Blocking AI crawlers is a legitimate stance, and bandwidth from aggressive crawlers is a real cost. The problem this finding reports is the mismatch: your robots.txt publishes one policy and your edge enforces another, so you are neither protected nor cited — you are just unpredictable.

Does your site have this problem?

13x checks this and 112 others against your live URL in about 30 seconds. No account, and every finding comes with the fix for your framework.

No signup. Results in 30 seconds.

More ai visibility checks