Free tool
Check which AI crawlers your robots.txt allows
Reads your robots.txt the way each answer engine reads it, and separates the two questions everyone conflates: whether you can be cited, and whether you can be used as training data.
Fetches one file — your robots.txt — and reads it the way each agent does.
Retrieval and training are not the same decision
A retrieval crawler fetches your page when somebody asks a question, and the answer cites you. A training crawler collects corpus. Blocking the second is a licensing choice more publishers make every year. Blocking the first removes you from answers people are reading today — usually by accident, because both were disallowed by a template.
| Agent | What blocking it costs |
|---|---|
| Googlebotgooglebot | Google Search and AI Overviews |
| OAI-SearchBotoai-searchbot | ChatGPT search results |
| ChatGPT-Userchatgpt-user | ChatGPT following a link to you in a live conversation |
| PerplexityBotperplexitybot | Perplexity |
| ClaudeBotclaudebot | Claude |
| GPTBotgptbot | OpenAI training corpora |
| Google-Extendedgoogle-extended | Gemini grounding |
| Applebot-Extendedapplebot-extended | Apple Intelligence |
| CCBotccbot | Common Crawl, which most models train on |
robots.txt is a claim. The audit measures the outcome.
The full audit requests your page as each agent and records the status code it actually receives, because bot management at the edge can allow a crawler in robots.txt and answer it with a 403 anyway. It also checks whether your content exists without JavaScript, which is what most AI crawlers see.
Run the full auditAI visibility checks
- My content only renders with JavaScriptContent is readable without JavaScript
- How to let ChatGPT and Claude crawl my siteAnswer engines are allowed to retrieve
- Why AI crawlers get a different response than GoogleAI crawlers actually receive the page
- How to write a description AI can quoteA citable description near the top
- Why my content is not quoted in AI answersPassages stand on their own
- Why my pages contradict each other about pricingThe site states the same facts everywhere
- Should I block AI crawlers in robots.txtRetrieval and training policy point the same way
- How to write headings that get cited in AI answersHeadings phrased as questions
- How to add FAQPage schemaFAQPage schema where there are questions
- Why lists and tables should be real HTMLReal lists and tables, not styled divs
- Do authorship and dates affect AI citationsAuthorship and freshness signals
- Does content freshness affect AI searchDeclared dates are recent
- What is llms.txt and do I need onellms.txt is present