A hidden noindex is an indexing block you can't see on the page itself. It sits either in the HTML head as a robots meta tag or in the HTTP response as an X-Robots-Tag header, and Google follows both. If a page you want in search shows URL marked 'noindex' in Search Console, check the response headers, the original HTML and robots.txt, in that order.
Two places a noindex can hide
Google's guide to blocking indexing with noindex describes two ways to set the rule, with the same effect:
- In the HTML:
<meta name="robots" content="noindex">in the head, or a crawler-specific version such as<meta name="googlebot" content="noindex">. - In the HTTP response header:
X-Robots-Tag: noindex. It never appears in the page source, so "view source" won't show it. For PDFs, images and video it is the only option, because those files have no HTML head.
The header version is the one people miss. The page renders, the source is clean, and the block lives in a server or CDN rule nobody remembers adding.
Where a hidden noindex usually comes from
| Source | What happens | Where to look |
|---|---|---|
| CMS staging setting left on | A "discourage search engines" option adds a robots meta tag to every page | CMS visibility settings |
| SEO plugin rules | A post type, tag archive or paginated page set to noindex | Plugin settings per content type |
| Server config | A rule that sets X-Robots-Tag, often copied from a staging server | .htaccess, nginx config, response headers |
| CDN or proxy | An edge rule adds the header to some paths or file types | CDN rules; headers fetched from outside your network |
| Templates | A noindex written into one template and reused by new page types | Theme or page templates |
| JavaScript | A script adds or removes the robots meta tag after load | Rendered HTML in URL Inspection |
The JavaScript case needs care. Google's JavaScript SEO basics say that when Google finds noindex in the original code it may skip rendering and JavaScript execution, so removing a noindex with JavaScript may not work as expected. If you want a page indexed, keep noindex out of the original HTML.
One more trap: unescaped code samples. An article that explains noindex and pastes <meta name="robots" content="noindex"> as raw HTML, instead of escaped text, puts a real robots tag into its own page. SEO crawlers report it as noindex, and you can't count on search engines ignoring it.
Robots.txt can hide the noindex from Google
If robots.txt disallows a URL, Googlebot never fetches the page, so it never sees the noindex. Google's robots meta tag and X-Robots-Tag specification says indexing rules on a page blocked from crawling are not found and are ignored, and the URL can still appear in results if other pages link to it. To get a page out of the index, allow crawling and use noindex. To get a page into the index, remove both blocks.
How to check a page in five minutes
- Read the response headers. Run
curl -sI -A "Googlebot" https://example.com/page/and look forX-Robots-Tag. In Chrome DevTools, open Network, reload, click the document request and read Response Headers. - Search the original HTML. Use view-source (not Inspect, which shows the DOM after scripts ran) and search for
noindex, includinggooglebotmeta tags. - Check robots.txt for a Disallow rule that matches the path.
- Run URL Inspection in Search Console. The Indexing allowed? line shows whether a noindex was detected, and the live test shows the HTML and HTTP headers Googlebot received.
- Check many URLs at once. A bulk noindex checker that reads both the meta tag and the X-Robots-Tag header saves time; the SpeedyIndex tool takes up to 10 URLs per run and shows where the block sits.
Not every drop is a noindex
Pages also leave the index when the server fails. Google's documentation on HTTP status codes says 5xx and 429 responses make its crawlers slow down, and URLs that keep returning server errors are removed from the index. If Search Console shows Server error (5xx) next to pages you suspected of carrying a noindex, run them through a 5xx error checker before you touch any meta tags.
For planned maintenance, answer with 503 and a Retry-After header rather than switching on noindex. A noindex that stays in place after the work is done removes the page from the index.
After you remove the noindex
- Request the page again and confirm the header and the HTML are clean.
- Make sure robots.txt allows crawling of the path.
- Request indexing for the key URLs in URL Inspection.
- Watch the URL marked 'noindex' count in the Page indexing report fall over the following weeks.
- Add an X-Robots-Tag check to your release checklist, so a staging config doesn't reach production again.
FAQ
Does X-Robots-Tag work for PDFs and images?
Yes. Google's documentation says the response header can be used for non-HTML resources such as PDFs, video files and image files. For those files it is the only way to set noindex, because they have no HTML head.
If robots.txt blocks a page, will Google see its noindex?
No. Googlebot doesn't fetch a page that robots.txt disallows, so it never reads the noindex. The URL can still appear in results if other pages link to it. Allow crawling if you want the noindex to work.
Can I add or remove noindex with JavaScript?
Don't rely on it. Google says it may skip rendering when the original HTML contains noindex, so removing it with JavaScript may not work as expected. Set the rule you want in the original HTML or in the server response.
Sources
- Google Search Central: Block Search indexing with noindex
- Google Search Central: Robots meta tag and X-Robots-Tag specifications
- Google Search Central: JavaScript SEO basics
- Google Search Central: How HTTP status codes affect Google's crawlers
- Search Console Help: URL Inspection tool