Discoverability · Next.js (Pages Router)
How to add a robots.txt file in Next.js (Pages Router)
Without one, crawlers apply their own defaults and you have no way to point them at your sitemap or keep them out of routes that should never be indexed. A malformed line is worse than a missing file: crawlers skip lines they cannot parse, so a typo silently drops the rule you were relying on.
The fix for Next.js (Pages Router)
13x detects your framework from the response and hands you this version rather than the generic one — below 50% confidence it hedges and gives you the generic one instead.
Put a static robots.txt in public/:
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xmlFiles in public/ are served from the root, so this becomes /robots.txt.
On a different stack? The general version of this fix explains what 13x checks and why it matters, without assuming a framework.
Check your Next.js (Pages Router) site
113 deterministic checks against your live URL, in about 30 seconds. Framework detected from the response, so every fix comes back in the form your stack actually uses.
No signup. Results in 30 seconds.