Should I block AI crawlers in robots.txt?

It depends on your goals. If you do not want your content used for AI training, add Disallow rules for user-agents like GPTBot, ClaudeBot, and CCBot. The auditor flags when none of these are addressed so the choice is at least deliberate.

Does robots.txt keep secret paths private?

No. robots.txt is public and only requests that compliant crawlers skip listed paths. Listing a sensitive directory there actually advertises it to anyone reading the file. Use real authentication and the noindex header for genuine privacy.

Why flag a Disallow on /admin?

A Disallow rule names the path in a public file, so it can hint attackers toward sensitive areas. The auditor notes paths like /admin, /api, and /private so you remember that listing them is a disclosure, not protection.

What counts as a syntax error?

Common issues include directives before any User-agent line, an Allow or Disallow line with no path, and unknown field names. The auditor reports the line number so you can correct it.

Is my robots.txt sent anywhere?

No. The file is parsed entirely in your browser. Nothing is transmitted, so you can audit unpublished or internal robots.txt content safely.

robots.txt Privacy Auditor

A robots.txt file controls which crawlers may visit which paths, and in 2026 it is also where you state your stance on AI training crawlers. It is easy to misconfigure: forgetting AI bots entirely, accidentally allowing admin paths, or — most importantly — assuming the file hides anything when it is fully public. This auditor parses your robots.txt in the browser, groups the rules, and reports privacy and syntax findings with fixes.

How it works

The tool parses the file line by line, building groups keyed by User-agent:

AI crawler coverage — it checks whether any rules address known AI bots (GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot). If none are mentioned, it flags that your AI-training stance is undefined.
Path exposure — Disallow rules naming sensitive paths (/admin, /api, /private, /.git) are noted, because listing them publicly is a disclosure, not protection.
Permissive Allow — Allow rules that open up admin or API paths are flagged.
Syntax — directives appearing before any User-agent line, empty Allow/Disallow values, and unknown fields are reported with line numbers.

Each finding comes with a short remediation note.

Tips and example

Consider:

User-agent: *
Disallow: /admin/
Allow: /api/public/

The auditor notes that /admin/ is exposed by being named, that no AI crawler directive exists, and confirms the Allow is scoped to a public sub-path. The privacy-correct approach is to protect /admin with real authentication (not robots.txt), add explicit User-agent: GPTBot rules if you want to opt out of AI training, and never rely on Disallow to keep anything secret — the file itself is world-readable.