Rules
What changed in the rules.
Separated AI retrieval from AI training, and started measuring what crawlers actually receive rather than what robots.txt claims.
- Changed
Blocking a training crawler no longer counts against the score.
Refusing to be training corpus is a licensing decision that a growing number of publishers make deliberately. Refusing retrieval crawlers removes you from answers people are reading today. Scoring both the same way told editors their considered stance was a defect, so the two are now separate questions — and a robots.txt that permits training while blocking retrieval is reported as the inversion it almost always is.
- Added
The page is now requested as each AI agent, not just parsed for permission.
robots.txt is a claim about what should happen. On a growing share of origins, bot management at the edge answers first and returns a 403 to an agent robots.txt explicitly allows. That disagreement is invisible to every tool that only reads the file, so we make the request and record the status code.
- Added
Two checks for whether your content survives being quoted on its own.
Retrieval lifts a passage out of the page and shows it alone. A paragraph opening with “it” or “this feature” means nothing once separated, and a site that states two different prices on two pages gets hedged rather than quoted. Both are copy problems rather than markup problems, which is why nothing else was catching them.
- Added
Secret and sandbox keys in the page source.
A build-time prefix such as NEXT_PUBLIC_ or VITE_ inlines whatever it is given into the client bundle. The quieter half of this check is the live sandbox key: the payment form accepts test cards, declines real ones, and logs nothing as a failure — the money simply never arrives.
Dates are the day a change shipped and are never regenerated. A changelog derived from the build moves on every deploy while the rules sit still — which is the fake-freshness signal our own content-freshness check reports on other people’s sites. The full registry is published with weights and scope.
Run the current rules against your product
113 checks, about 30 seconds, no account. Every finding comes with its evidence and the fix for the framework we detect.
No signup. Results in 30 seconds.