Discoverability · Next.js (App Router)
How to add a robots.txt file in Next.js (App Router)
Without one, crawlers apply their own defaults and you have no way to point them at your sitemap or keep them out of routes that should never be indexed. A malformed line is worse than a missing file: crawlers skip lines they cannot parse, so a typo silently drops the rule you were relying on.
The fix for Next.js (App Router)
13x detects your framework from the response and hands you this version rather than the generic one — below 50% confidence it hedges and gives you the generic one instead.
Add app/robots.ts. Next generates /robots.txt from it at build time:
import type { MetadataRoute } from "next";
export default function robots(): MetadataRoute.Robots {
return {
rules: [{ userAgent: "*", allow: "/" }],
sitemap: "https://example.com/sitemap.xml",
};
}On a different stack? The general version of this fix explains what 13x checks and why it matters, without assuming a framework.
Check your Next.js (App Router) site
113 deterministic checks against your live URL, in about 30 seconds. Framework detected from the response, so every fix comes back in the form your stack actually uses.
No signup. Results in 30 seconds.