Skip to content

AI visibility · Next.js (App Router)

How to let ChatGPT and Claude crawl my site in Next.js (App Router)

Answer engines can only cite pages their retrieval crawlers are allowed to fetch. Blocking GPTBot, ClaudeBot or PerplexityBot in robots.txt removes you from the answers they generate, and the block is usually copied from a template rather than chosen.

The fix for Next.js (App Router)

13x detects your framework from the response and hands you this version rather than the generic one — below 50% confidence it hedges and gives you the generic one instead.

Your robots.txt blocks: . In app/robots.ts:

ts code
import type { MetadataRoute } from "next";

const AI_CRAWLERS = [
  "GPTBot",
  "OAI-SearchBot",
  "ChatGPT-User",
  "ClaudeBot",
  "anthropic-ai",
  "PerplexityBot",
  "Google-Extended",
  "Applebot-Extended",
];

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      { userAgent: "*", allow: "/" },
      // Explicit beats inherited: state the decision rather than leaving it to a default.
      { userAgent: AI_CRAWLERS, allow: "/" },
    ],
    sitemap: "https://example.com/sitemap.xml",
  };
}

Note that Google-Extended affects Gemini and AI Overviews only, not Search ranking.

On a different stack? The general version of this fix explains what 13x checks and why it matters, without assuming a framework.

Check your Next.js (App Router) site

113 deterministic checks against your live URL, in about 30 seconds. Framework detected from the response, so every fix comes back in the form your stack actually uses.

No signup. Results in 30 seconds.

The same fix, other stacks