Free website check

Glossary

X-Robots-Tag

Written by
Outseen
Updated

X-Robots-Tag is an HTTP header that lets a server send the same instructions as a meta robots tag, such as noindex or nofollow. It also works for files without HTML, such as PDFs and images.

X-Robots-Tag is an HTTP header the server uses to send instructions about indexing and display. The rules are the same as for the meta robots tag, such as noindex and nofollow. Because the instruction sits in the HTTP response and not in the HTML, it also works for PDFs, images and other files. Google describes the header in its robots rules specification.

Example

HTTP/1.1 200 OK
X-Robots-Tag: noindex

You can also aim a rule at one crawler, for example X-Robots-Tag: googlebot: nofollow. The header is usually set in the web server configuration or by the hosting provider.

Common mistake

Only checking the page source. You cannot see an X-Robots-Tag on the page or in the HTML. It can be left behind after a test phase or come from a hosting setting, so a page drops out of Google without anyone noticing.

How do you check it?

Look at the response headers in your browser’s developer tools (Network tab) or use URL Inspection in Google Search Console. Atlas, the research engine developed by Outseen, checks this header too: on Outseen’s own test site, Atlas recorded a noindex in the X-Robots-Tag, with the source, URL and time of measurement. That example is in How Atlas investigates a website.

Sources

Read next