Where does robots.txt go?

It must live at the root of your domain, e.g. https://example.com/robots.txt. Crawlers look for it there before crawling your site.

Does robots.txt make pages private?

No. It only requests that well-behaved crawlers skip certain paths — it is not a security control. Use authentication or noindex headers to truly protect content.

Is my configuration uploaded anywhere?

No. The file is generated entirely in your browser from the values you enter — nothing is sent to a server.

What does User-agent mean in robots.txt?

User-agent names the crawler a rule block applies to. * means all crawlers; you can also target a specific bot like Googlebot. Each block lists Disallow and Allow rules for that user-agent.

Does Disallow remove a page from Google?

No. Disallow stops compliant crawlers from fetching a path, but a blocked URL can still appear in search results if linked elsewhere. To keep a page out of the index, allow crawling and use a noindex meta tag or header instead.

What does crawl-delay do?

Crawl-delay asks crawlers to wait a set number of seconds between requests, easing load on your server. Google ignores it (use Search Console crawl-rate settings), but some other crawlers honour it.

How do Allow and Disallow interact?

You can block a folder with Disallow and then re-open a specific file or subpath inside it with Allow. Most crawlers apply the most specific (longest) matching rule, so Allow can carve an exception out of a broader Disallow.

What is the robots.txt Generator?

Free robots.txt generator — allow or block crawlers, set per-path disallow and allow rules, crawl-delay and sitemap URL, then copy or download. It runs entirely in your browser, nothing is uploaded. It runs free in your browser on Gera Tools, with nothing uploaded.

robots.txt Generator

Name: robots.txt Generator
Creator: Gera Tools
License: https://creativecommons.org/licenses/by/4.0/

A robots.txt file tells search engines and crawlers which parts of your site they may and may not crawl. This generator builds a valid one from a preset — allow all or block all — or from custom rules with user-agent, disallow and allow paths, a crawl-delay, and your sitemap URL.

How it works

The tool assembles standard robots.txt directives in the correct order. A User-agent line names the crawler (* for all), followed by Disallow lines for paths to block and Allow lines for exceptions. An optional Crawl-delay asks crawlers to pace their requests, and a Sitemap line points crawlers to your sitemap URL. The presets are shortcuts: “allow all” emits User-agent: * with an empty Disallow, and “block all” emits Disallow: /. You then copy the output or download a ready-to-deploy file. Remember robots.txt is an instruction to well-behaved crawlers, not a security control.

Example

A custom config blocking the admin and cart but allowing a public sub-path, with a sitemap:

User-agent: *
Disallow: /admin/
Disallow: /cart/
Allow: /cart/help/
Crawl-delay: 10

Sitemap: https://example.com/sitemap.xml

Directive	Meaning
`User-agent: *`	Applies to all crawlers
`Disallow: /admin/`	Do not crawl /admin/
`Allow: /cart/help/`	Exception within a blocked folder
`Sitemap:`	Location of your sitemap

Place the file at the root of your domain (https://example.com/robots.txt). Everything is generated in your browser, so nothing is uploaded.

What to Disallow in practice

Most sites benefit from blocking a handful of well-known internal paths that waste crawl budget or leak implementation detail without adding search value:

/admin/ — login and admin panels should never be indexed.
/search? or /?s= — internal search result pages are often thin or duplicate content; Google prefers you block them.
/checkout/ and /account/ — account flows do not belong in search results and blocking them is tidy.
/wp-admin/ (WordPress) — a standard block for WordPress sites; also reduces bot login attempts.
/*.json$ or /*.xml$ — API response paths exposed on the same domain.

Per-bot targeting

The User-agent field accepts specific bot names alongside the * wildcard. Common named agents are Googlebot, Bingbot, DuckDuckBot, Yandex, Baiduspider and AhrefsBot. Each named block overrides the wildcard block for that bot, so you can let Googlebot through a path while blocking all others, or set a gentler Crawl-delay for one bot and a stricter one for another.

Crawl-delay: who honours it?

Crawl-delay asks crawlers to wait a set number of seconds between successive requests. Bing and Yandex honour it. Google ignores it — use Search Console’s crawl-rate tool instead. If your server is resource-constrained, set a reasonable delay for non-Google bots to reduce peak load without affecting Googlebot.

Deploying your file

Once you copy or download the generated file, upload it to the web root of your domain so it is accessible at https://yourdomain.com/robots.txt. On most hosting platforms this means placing the file in the public/, static/, or www/ folder at the top level. Verify the URL loads in a browser before submitting the sitemap in Search Console.