Robots.txt & Sitemap Checker

Check a website's robots.txt for mistakes, test whether Googlebot and other crawlers may visit a URL, and validate the XML sitemaps it lists.

Frequently asked questions

What is robots.txt?

robots.txt is a small text file at the root of a website (example.com/robots.txt) that tells search engine crawlers which parts of the site they may visit. It controls crawling, not indexing: a blocked page can still appear in Google if other sites link to it.

How do I stop a page from showing in Google?

Use a noindex meta robots tag or X-Robots-Tag header, and do not block the page in robots.txt. If robots.txt blocks the page, Google can never see the noindex instruction.

What happens if my site has no robots.txt?

If robots.txt returns 404, crawlers assume they may crawl everything, which is fine for most sites. If it returns a server error (5xx), Google may stop crawling the site until the file loads again.

What makes a sitemap valid?

An XML sitemap must be well-formed XML using the sitemaps.org format, list full URLs on the same host, contain at most 50,000 URLs and be no larger than 50 MB uncompressed. Larger sites split their URLs across several sitemaps listed in a sitemap index.