Skip to content

Menu

Loading languages…

Machine translation by Google Translate. It can be wrong, so the English articles are the originals. Your choice is saved in a cookie on this device.

Follow Akif

SERP TimesSERP Times

Loading languages…

Machine translation by Google Translate. It can be wrong, so the English articles are the originals. Your choice is saved in a cookie on this device.

    Type at least two letters to search news and guides.

    to open · ↑↓ to move · Esc to close

    Free tool · Generators

    Robots.txt Generator

    Build a robots.txt file with preset rules, AI crawler controls and sitemap lines, then copy or download it.

    Runs in your browser: nothing you type is uploaded.

    Rules for all crawlers

    Every crawler may crawl the whole site.

    AI crawlers

    Tick the crawlers you want to keep out. Robots.txt is a request that well-behaved crawlers follow; it can't force anyone to comply.

    • Collects public pages that may be used to train OpenAI models.

    • Builds the index behind ChatGPT search results. Blocking it can hide you from ChatGPT search.

    • Fetches a page when a ChatGPT user asks it to.

    • Collects public pages that may be used to train Claude models.

    • Improves the quality of Claude's search results.

    • Fetches a page when a Claude user asks it to.

    • Indexes pages for Perplexity's answer engine.

    • Fetches a page when a Perplexity user asks it to.

    • A control token for Gemini training and grounding. Blocking it does not change how you rank in Google Search.

    • Controls whether Apple may use your pages to train its models. Apple Search still uses Applebot.

    • Builds the open Common Crawl dataset that many AI models train on.

    • Collects public pages that may be used to train Meta's AI models.

    Sitemaps

    Your robots.txt

    User-agent: *
Disallow:

    Upload the file to the root of your site, so it loads at https://example.com/robots.txt, then test it with our robots.txt checker.

    How to use it

    1. 1

      Pick a starting point, such as Allow everything, WordPress or Online store, and edit the rules for all crawlers.

    2. 2

      Tick any AI crawlers you want to keep out. The training preset blocks the bots that collect pages for model training.

    3. 3

      Add your sitemap URLs, one per line.

    4. 4

      Copy or download the file, upload it to the root of your site, then test it with the Robots.txt & Sitemap Checker.

    Why it matters

    Robots.txt controls crawling, not indexing. A blocked URL can still appear in search results if other pages link to it. To keep a page out of Google, let it be crawled and add a noindex tag instead.

    AI companies run several crawlers for different jobs. Training bots such as GPTBot and ClaudeBot collect pages for models. Search bots such as OAI-SearchBot and PerplexityBot build the indexes behind AI answers, so blocking them can keep you out of those answers. Google-Extended is a control token for Gemini training and doesn't change how you rank in Google Search.

    Robots.txt is a request. Reputable crawlers follow it, but it can't stop one that ignores it. Anything that must stay private needs a login or a firewall rule, not a robots.txt line.

    Frequently asked questions

    Does robots.txt keep a page out of Google?

    No. It stops crawling, and Google can still index a blocked URL, without its content, if it finds links to it. Use a noindex meta tag on a crawlable page to keep it out of results.

    Should I block AI crawlers?

    That's a business choice. Blocking training bots asks AI companies not to use your pages for model training and has no effect on Google rankings. Blocking AI search bots can keep your pages out of the answers those products give.

    Will every crawler obey my robots.txt?

    Well-behaved crawlers do, but there's no enforcement. If a bot ignores your rules, block it at the server or firewall.

    Where do I upload the file?

    To the root of your domain, so it loads at https://example.com/robots.txt. Each subdomain needs its own file, and the name must be lowercase.

    Should I block CSS and JavaScript files?

    No. Google needs them to render your pages the way visitors see them, and the generator warns you if a rule looks like it hides them.

    Related guides

    More SEO guides