X-Robots-Tag HTTP header contains noindex — the fix
The server sends an X-Robots-Tag HTTP header with a noindex directive. This instructs search engines not to index the page, even if the HTML contains no noindex meta tag.
Where this fits: crawlability in SEO
Crawlability is whether search engine bots can physically reach and read your pages. Before Google can rank anything, Googlebot has to fetch it: follow links to it, receive a successful response, and parse the HTML. Crawl problems are silent — the page looks fine in a browser while bots hit walls (robots rules, broken chains, server errors, infinite URL spaces). Because everything downstream depends on the crawl, fixing crawlability issues usually unlocks more value per hour than any other SEO work.
Why x-robots-tag http header contains noindex hurts your rankings
The X-Robots-Tag is processed before the page body is parsed, making it an authoritative signal that overrides on-page directives. If set unintentionally (often by server misconfiguration), it silently de-indexes pages without any visible indication in the HTML source, making it hard to diagnose.
This is a critical issue: it can remove pages from search results entirely or block them from being crawled at all. Treat it as an outage, not a backlog item — every day it persists is compounding lost visibility.
How to fix it
Remove the noindex directive from the X-Robots-Tag HTTP header in your server or CDN configuration. Check .htaccess, nginx config, and any middleware that modifies response headers. Verify with curl -I to confirm the header is no longer sent.
Crawlability best practices
- Keep robots.txt minimal and deliberate — block admin/duplicate spaces, never CSS/JS assets.
- Maintain an XML sitemap that lists only canonical, indexable, 200-status pages, and reference it in robots.txt.
- Keep every important page within 3 clicks of the homepage.
- Return proper status codes: 200 for content, 301 for permanent moves, 410 for gone — never soft-404s.
- Collapse redirect chains to a single hop; update internal links to point at final URLs.
- Keep URL parameters under control — infinite filter/sort combinations burn crawl budget.
The full library: SEO best practices, by category.
Frequently asked questions
What does "X-Robots-Tag HTTP header contains noindex" mean?
The server sends an X-Robots-Tag HTTP header with a noindex directive. This instructs search engines not to index the page, even if the HTML contains no noindex meta tag.
Why does x-robots-tag http header contains noindex matter for SEO?
The X-Robots-Tag is processed before the page body is parsed, making it an authoritative signal that overrides on-page directives. If set unintentionally (often by server misconfiguration), it silently de-indexes pages without any visible indication in the HTML source, making it hard to diagnose.
How do I fix x-robots-tag http header contains noindex?
Remove the noindex directive from the X-Robots-Tag HTTP header in your server or CDN configuration. Check .htaccess, nginx config, and any middleware that modifies response headers. Verify with curl -I to confirm the header is no longer sent.
How serious is this issue?
This is a critical issue: it can remove pages from search results entirely or block them from being crawled at all. Treat it as an outage, not a backlog item — every day it persists is compounding lost visibility. It belongs to the crawlability family of checks.
How do I find every page affected by this on my site?
Run a free Dr Urls audit: it crawls your site, detects x-robots-tag http header contains noindex on every affected page, shows example URLs, and generates a ready-to-use fix task. Re-scan after fixing to verify the issue is gone.
Does your site have this issue?
A free Dr Urls audit crawls your site, finds every page affected by x-robots-tag http header contains noindex, and hands you a ready-made fix task.
Check my site free