What is a robots meta tag?

A robots meta tag is an HTML element placed in the head of a page that tells search-engine crawlers how to treat that page. Unlike robots.txt, which controls whether a crawler can access a URL at all, the meta robots tag controls what the crawler does once it has accessed the page — whether to index it, follow its links, show a cached copy, display a text snippet, and so on.

What is the difference between noindex in robots.txt and in the meta tag?

robots.txt disallow tells a well-behaved crawler not to visit the URL; it cannot see or honour a noindex directive on a page it never fetches. The meta robots tag is read after the page is fetched, so noindex there is the authoritative way to remove a page from an index. If you block a URL in robots.txt, search engines may still list it as a URL-only result with no snippet because they know the URL exists from external links.

When should I use nofollow on a page?

nofollow on the meta robots tag applies to every link on the page, which is a blunt instrument. It is useful on pages like login walls, legal disclaimers, or internal search results where you want none of the links followed. For selective control on individual links, use the rel="nofollow" attribute on those anchor elements instead.

What does max-snippet do?

max-snippet limits how many characters Google may use for the text snippet shown beneath a result. Setting max-snippet:0 is equivalent to nosnippet. Setting max-snippet:-1 gives Google permission to show the longest snippet it considers useful. Values in between (for example 160) cap the snippet length.

Does noarchive remove my page from Google Cache?

Yes. noarchive (or its older synonym nocache) tells search engines not to store and display a cached copy of the page. The page can still be indexed and appear in results — it just will not have a cached-copy link. Use it on pages with frequently changing content such as price lists or live scores.

What are per-bot directives and when do I need them?

Most sites only need the generic robots tag. Per-bot directives (using names like googlebot, bingbot, or slurp) let you apply different rules to specific crawlers. For example, you might want Google to index a page but prevent Bing from caching it. The more-specific tag takes priority over the generic robots tag for that crawler.

Robots Meta Tag Generator

The Robots Meta Tag Generator builds the correct <meta name="robots"> tag — and optional per-bot variants — for any page. Instead of memorising directive names and worrying about syntax, you pick the settings you want and copy the finished HTML directly into your page head. The effect summary panel shows you at a glance what each configuration means for indexing, link-following, cached copies, snippets, image previews and translation.

How it works

A robots meta tag is read by every major crawler after it fetches a page. The tag is a comma-separated list of directives inside a content attribute:

<meta name="robots" content="index, follow" />

This generator assembles the directive list from your selections and escapes every value correctly — no risk of unquoted characters breaking your markup.

The primary robots tag applies to all crawlers. If you need a specific crawler (say googlebot) to behave differently, you add a per-bot tag with the bot name as the name attribute; the more-specific rule takes precedence for that crawler while all others still follow the generic tag.

Directive reference

Directive	Effect
`index` / `noindex`	Whether to list the page in search results
`follow` / `nofollow`	Whether to crawl the links on the page
`noarchive`	Suppress the cached-copy link in results
`nosnippet`	Show no text or video snippet in results
`max-snippet:N`	Cap snippet length at N characters (-1 = unlimited, 0 = none)
`max-image-preview:SIZE`	Limit thumbnail size (none / standard / large)
`max-video-preview:N`	Cap video preview length in seconds (-1 = unlimited)
`notranslate`	Suppress Google’s offer to translate the page
`noodp`	Do not use ODP/DMOZ description (legacy, mostly obsolete)
`unavailable_after: DATE`	Treat page as noindex after the given ISO 8601 datetime

Worked example

Suppose you run a legal-terms page that must be crawlable (so links work) but should never appear as a search result and must not be cached:

Set indexing to noindex
Set following to follow (links on the page may still pass PageRank)
Tick noarchive

The generator produces:

<meta name="robots" content="noindex, follow, noarchive" />

Or imagine a news article you want fully indexed with the largest possible Google snippet and a large image preview:

Keep indexing as index, following as follow
Set text snippet to max-snippet:-1
Set image preview to max-image-preview:large

Result:

<meta name="robots" content="index, follow, max-snippet:-1, max-image-preview:large" />

Formula note

There is no mathematical formula here — the tag is purely declarative. The generator’s job is accurate syntax and completeness. Every value is HTML-escaped so characters like &, ", < and > cannot break the surrounding markup, and the effect-summary badges give you an instant human-readable confirmation of what you have configured before you copy.

FAQs

See the six questions below for common situations including the difference between robots.txt and the meta tag, how to use nofollow correctly, what max-snippet does, and when per-bot directives are worth adding.