Discoverability · Astro
How to add a robots.txt file in Astro
Without one, crawlers apply their own defaults and you have no way to point them at your sitemap or keep them out of routes that should never be indexed. A malformed line is worse than a missing file: crawlers skip lines they cannot parse, so a typo silently drops the rule you were relying on.
The fix for Astro
13x detects your framework from the response and hands you this version rather than the generic one — below 50% confidence it hedges and gives you the generic one instead.
Add public/robots.txt:
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap-index.xmlIf you use @astrojs/sitemap, the generated file is sitemap-index.xml, not sitemap.xml.
On a different stack? The general version of this fix explains what 13x checks and why it matters, without assuming a framework.
Check your Astro site
113 deterministic checks against your live URL, in about 30 seconds. Framework detected from the response, so every fix comes back in the form your stack actually uses.
No signup. Results in 30 seconds.