Skip to content

Menu

Follow Akif

SERP TimesSERP Times

    Type at least two letters to search news and guides.

    to open · ↑↓ to move · Esc to close

    Free tool · Technical

    Robots.txt & Sitemap Checker

    Fetch a site's robots.txt and XML sitemap, check they are valid and see which URLs are blocked.

    Checks public URLs from our server, up to 20 checks an hour.

    Public websites only. Try .

    This tool is being set up and will be available soon. The other free tools work right now.

    How to use it

    1. 1

      Enter any URL on the site, such as a page you want to test.

    2. 2

      Run the check. We fetch the site's robots.txt and the first sitemap it lists, or /sitemap.xml if it lists none.

    3. 3

      See whether Googlebot and other crawlers may crawl your URL, and which rule decides it.

    4. 4

      Review the warnings, fix them, and resubmit the sitemap in Search Console if it changed.

    Why it matters

    One wrong line in robots.txt can block crawling of a whole site, and a broken sitemap means Google learns about new pages more slowly. Both are easy to get wrong during a redesign or a move to a new CMS.

    Robots.txt controls crawling, not indexing. A blocked URL can still appear in results if other pages link to it. To keep a page out of Google, let it be crawled and add a noindex tag instead.

    The URL test follows Google's rules: the most specific user-agent group applies, the longest matching rule wins, and allow wins a tie. Google ignores directives such as crawl-delay and reads only the first 500 KiB of the file.

    Frequently asked questions

    Does robots.txt stop a page from being indexed?

    No, it stops crawling. Google can still index a blocked URL, without its content, if it finds links to it. Use a noindex meta tag on a crawlable page to keep it out of results.

    What happens if my site has no robots.txt?

    If it returns a 404, Google assumes there are no restrictions and crawls everything, which is fine for many sites. A server error (5xx) is worse: Google may pause crawling until it can read the file.

    Where should my sitemap be listed?

    Add a Sitemap: line with the full sitemap URL to robots.txt, and submit the sitemap in Google Search Console and Bing Webmaster Tools.

    How many URLs can one sitemap hold?

    Up to 50,000 URLs or 50 MB uncompressed per file. Larger sites split them up and list the files in a sitemap index.

    Does Google use priority and changefreq?

    No, Google ignores both. It uses lastmod when the dates are consistently accurate, so only change it when the content really changes.

    Can I check a staging or internal site?

    No. The tool only checks public websites. Sites on private network addresses are blocked for security.

    Related guides

    More SEO guides