About this tool

What the Robots & Sitemap Checker does

The Robots & Sitemap Checker confirms the two files search engines look for first: a valid robots.txt and an XML sitemap. It shows whether each exists, how many crawl rules and URLs they contain, and whether they reference each other: the crawl basics that decide whether a site can be indexed at all.

Example resultwikipedia.org
  • robots.txtFound, valid
  • Crawl rules42
  • XML sitemapLinked
  • Sitemap URLs12

Illustrative example, not a live lookup.

A single wrong line in robots.txt can block an entire site from Google, and plenty of real sites do exactly that by accident. This check surfaces crawl blocks before they cost you traffic.

The sitemap side tells you whether search engines have a clean list of URLs to crawl, and whether robots.txt points to it the way it should.

Frequently asked questions

What is robots.txt for?
It tells crawlers which parts of a site they may or may not request. Used well it guides crawl budget; used carelessly it can accidentally block pages, or a whole site, from being indexed.
Do I need an XML sitemap?
It is strongly recommended. A sitemap gives search engines a complete, structured list of your URLs, which helps larger or newer sites get discovered and crawled more thoroughly.
Should robots.txt link to the sitemap?
Yes. A Sitemap: line in robots.txt is a standard way to point crawlers to your sitemap, and this tool checks whether that reference is present.