Where does robots.txt go?

It must live at the root of your domain, e.g. https://example.com/robots.txt. Crawlers look for it there before crawling your site.

Does robots.txt make pages private?

No. It only requests that well-behaved crawlers skip certain paths — it is not a security control. Use authentication or noindex headers to truly protect content.

Is my configuration uploaded anywhere?

No. The file is generated entirely in your browser from the values you enter — nothing is sent to a server.

What does User-agent mean in robots.txt?

User-agent names the crawler a rule block applies to. * means all crawlers; you can also target a specific bot like Googlebot. Each block lists Disallow and Allow rules for that user-agent.

Does Disallow remove a page from Google?

No. Disallow stops compliant crawlers from fetching a path, but a blocked URL can still appear in search results if linked elsewhere. To keep a page out of the index, allow crawling and use a noindex meta tag or header instead.

What does crawl-delay do?

Crawl-delay asks crawlers to wait a set number of seconds between requests, easing load on your server. Google ignores it (use Search Console crawl-rate settings), but some other crawlers honour it.

How do Allow and Disallow interact?

You can block a folder with Disallow and then re-open a specific file or subpath inside it with Allow. Most crawlers apply the most specific (longest) matching rule, so Allow can carve an exception out of a broader Disallow.

robots.txt Generator

A robots.txt file tells search engines and crawlers which parts of your site they may and may not crawl. This generator builds a valid one from a preset — allow all or block all — or from custom rules with user-agent, disallow and allow paths, a crawl-delay, and your sitemap URL.

How it works

The tool assembles standard robots.txt directives in the correct order. A User-agent line names the crawler (* for all), followed by Disallow lines for paths to block and Allow lines for exceptions. An optional Crawl-delay asks crawlers to pace their requests, and a Sitemap line points crawlers to your sitemap URL. The presets are shortcuts: “allow all” emits User-agent: * with an empty Disallow, and “block all” emits Disallow: /. You then copy the output or download a ready-to-deploy file. Remember robots.txt is an instruction to well-behaved crawlers, not a security control.

Example

A custom config blocking the admin and cart but allowing a public sub-path, with a sitemap:

User-agent: *
Disallow: /admin/
Disallow: /cart/
Allow: /cart/help/
Crawl-delay: 10

Sitemap: https://example.com/sitemap.xml

Directive	Meaning
`User-agent: *`	Applies to all crawlers
`Disallow: /admin/`	Do not crawl /admin/
`Allow: /cart/help/`	Exception within a blocked folder
`Sitemap:`	Location of your sitemap

Place the file at the root of your domain (https://example.com/robots.txt). Everything is generated in your browser, so nothing is uploaded.