• Use noindex to keep a page out of search results. The page must remain crawlable for the tag to be seen.
• X-Robots-Tag does the same job in an HTTP header and works for PDFs, images and other non-HTML files.
• When directives conflict, the most restrictive one wins.
• Long-term noindexed pages are crawled less often and their links eventually stop counting.
• Snippet directives such as nosnippet and max-snippet also limit what AI features can show.

Robots meta directives are page-level instructions for search engines. They can be delivered as a meta tag in the HTML head or as an X-Robots-Tag HTTP response header. Both do the same job, so the choice comes down to what is easier to deploy.

Syntax

As a meta tag:

 <meta name="robots" content="noindex, follow" /> 

Targeting one crawler only:

 <meta name="googlebot" content="noindex" /> 

As an HTTP header, which is the only option for non-HTML files:

 X-Robots-Tag: noindex, nofollow 

The directives

• index, follow: the default. You do not need to state it.
• noindex: do not show this page in search results.
• nofollow: do not follow any links on this page.
• none: shorthand for noindex, nofollow.
• nosnippet: do not show a text snippet or video preview. This also stops the content being used as a direct input for Google AI Overviews and AI Mode.
• max-snippet:[number]: limit the snippet to a number of characters. 0 means none and -1 means no limit.
• max-image-preview:[none|standard|large]: set the largest image preview allowed. Large is needed for Discover.
• max-video-preview:[number]: limit video preview length in seconds.
• noimageindex: do not index images on this page.
• notranslate: do not offer a translation in results.
• unavailable_after:[date]: stop showing the page after a given date. Handy for events and offers.
• indexifembedded: allow content to be indexed when embedded through an iframe, even if the source URL is noindexed.

The older noarchive directive no longer does anything in Google, which retired its cache link in 2024. Bing still uses it, including as a signal for how content may be used in Copilot.

For finer control within a page, Google also supports the data-nosnippet HTML attribute on span, div and section elements.

Noindex and robots.txt do not mix

For noindex to work, the crawler has to fetch the page and read the instruction. If the URL is disallowed in robots.txt, the tag is never seen and the URL may stay in the index. Pick one: block crawling to save crawl resources, or allow crawling and noindex to remove from results.

Noindex over the long term

Google has said that pages left on noindex for a long time are crawled less frequently and are eventually treated much like noindex, nofollow, because the links on a page it rarely visits stop being processed. So "noindex, follow" is not a reliable way to pass link equity through a page indefinitely.

Common mistakes

• Site-wide noindex carried over from a staging environment.
• Noindex combined with a canonical to another URL, which sends conflicting signals.
• Noindexed URLs listed in the XML sitemap.
• Injecting noindex with JavaScript, or removing it with JavaScript. If the raw HTML says noindex, Google may not render the page at all.
• Two robots meta tags with different values. The most restrictive wins.
• Applying nofollow at page level when you only meant to qualify one link.

How to test

• URL Inspection reports "Excluded by noindex tag" and shows the fetched HTML and headers.
• Check headers directly with your browser's network panel or curl -I.
• A site crawler will list every URL's meta robots and X-Robots-Tag values so you can spot patterns.

The GEO angle

Snippet controls are the main lever for limiting how Google's AI features use your content without leaving search altogether. The trade-off is real: restrict snippets and you also lose ordinary snippets and reduce your chance of being cited. Other AI platforms do not consistently honour these tags, so robots.txt remains the primary control for them.

Add your title here

This is a paragraph. Writing in paragraphs lets visitors find what they are looking for quickly and easily. Make sure the title suits the content of this text.