Skip to content

Discoverability

How to add a robots.txt file

Without one, crawlers apply their own defaults and you have no way to point them at your sitemap or keep them out of routes that should never be indexed. A malformed line is worse than a missing file: crawlers skip lines they cannot parse, so a typo silently drops the rule you were relying on.

What 13x checks

This is rule discoverability.robots-txt in the public registry: robots.txt exists and is valid. It runs on every audit, against the pages we actually fetched, and its result is derived from the response rather than estimated.

Surface
Discoverability
Score weight
6 of the readiness score
Scope
Runs on every audited page
Applies
To every site

Registry version 2026-07-30. Every rule is published, and the audit is deterministic — the same page produces the same finding every time.

The fix

The same text the audit hands you when this check fails on your own site.

Create robots.txt at the web root so it is served from https://example.com/robots.txt:

Configuration
User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Verify it returns content-type: text/plain and not your app's HTML shell.

Framework-specific versions

Where the change differs enough to be worth writing out separately.

Does your site have this problem?

13x checks this and 112 others against your live URL in about 30 seconds. No account, and every finding comes with the fix for your framework.

No signup. Results in 30 seconds.

More discoverability checks