Meta Robots Tag Tester
Understand exactly what your meta robots and X-Robots-Tag directives tell search engines — including the conflicts between them.
Paste the value of the content attribute, or the whole <meta> tag — either works.
Leave empty for <meta name="robots">, which every crawler obeys. Name a bot to test a tag only that crawler reads.
Nothing to check yet. With no robots tag at all, search engines treat a page as index, follow — the summary below is what that default looks like.
What Google will do
- Index this page?
- Yes
- Follow links on this page?
- Yes
- Show a text snippet?
- Yes, Google picks the length
- Show an image preview?
- Standard size (the default)
- Show a video preview?
- Yes, Google picks the length
- Index images on this page?
- Yes
- Allow a cached copy?
- Yes, where the engine still offers one
- Offer a translation?
- Yes
The default a page gets with no robots tag at all.
The tag to paste
<meta name="robots" content="index, follow">Goes in <head>. A robots tag in the body is ignored, and so is one added by JavaScript after Google has already rendered the page.
How to use it
- Paste the content value of a robots tag — or the whole <meta> tag — and the breakdown updates as you type. Nothing is submitted anywhere.
- Use the pills to build a tag from scratch. Clicking a directive that is already there removes it, and picking a different value for the same directive swaps it.
- Switch to Fetch from URL to read a live page. It returns every robots meta tag on the page plus the X-Robots-Tag response header, which is the half a browser cannot show you.
Getting robots directives right
- Where index and noindex conflict, Google takes the more restrictive side. That rule applies across sources too: a meta tag saying index cannot rescue a page whose header says noindex.
- X-Robots-Tag does everything the meta tag does, and works on files that have no HTML to put a tag in — PDFs, images, feeds. It is the only way to noindex a PDF short of blocking it.
- noindex, follow keeps link equity moving through a page you do not want in the index. Google has said it eventually treats long-lived noindex pages as nofollow anyway, so it is a soft guarantee — but it costs nothing on the pages that matter.
- max-image-preview:large is what makes a page eligible for Google Discover. Leave it off and you get the standard thumbnail, which is not enough to be shown there.
- Never noindex a page you have also blocked in robots.txt. Google has to crawl the page to see the noindex, and robots.txt stops it doing so — the URL can stay indexed as a bare link for years.
- none is shorthand for noindex, nofollow. Use it only when you mean both, and never alongside other directives — the mix is where accidental deindexing comes from.
Frequently asked questions
What is the difference between meta robots and robots.txt?
robots.txt controls crawling; meta robots controls indexing and how a result is displayed. A robots.txt Disallow tells a crawler not to fetch the URL at all, which means it never sees the page or any tag inside it. A meta robots tag lives in the page's HTML, so the page has to be crawled for the directive to be read. That is why blocking a URL in robots.txt and adding noindex to it is self-defeating: the crawler never gets far enough to find the noindex, and the URL can still appear in results as a title-only listing.
What happens if I use both noindex and index?
Google resolves conflicting directives by taking the more restrictive one, so noindex wins and the page is dropped from the index. The same applies when the conflict is spread across sources — a meta tag saying index and an X-Robots-Tag header saying noindex resolve to noindex. The one exception is specificity: if you have a generic robots tag and a googlebot-specific tag, Googlebot obeys the googlebot one and ignores the generic tag entirely, even where the generic tag is stricter.
What is the X-Robots-Tag HTTP header?
It is the same directive vocabulary as meta robots, delivered as an HTTP response header instead of a tag in the HTML. Because it travels with the response rather than the markup, it works on anything a server can send: PDFs, images, XML feeds, plain text. It can also name a crawler — "X-Robots-Tag: googlebot: noindex" applies to Google alone. A browser cannot read that header on a cross-origin page, which is why this tool fetches the URL server-side to show it to you.
Does nofollow in meta robots affect PageRank?
A page-level nofollow tells Google not to follow any link on the page, so no PageRank flows out through them. Since 2019 Google treats nofollow as a hint rather than a directive for crawling and indexing purposes, meaning it may still crawl those links — but it does not restore the link equity that nofollow drops. The equity is not redistributed to your other links either; it is simply lost. On pages you are keeping out of the index, noindex, follow is almost always the better trade.
How long does noindex take to remove a page from search?
The page has to be recrawled before Google sees the directive, so removal takes anywhere from a day to several weeks depending on how often that URL is crawled. Submitting the URL for reindexing in Search Console speeds it up; the Removals tool hides it within hours while the noindex does its work permanently. Keep the page crawlable the whole time — if you block it in robots.txt while waiting, the noindex will never be read.