Skip to content
SERP TimesSERP Times

Loading languages…

Machine translation by Google Translate. It can be wrong, so the English articles are the originals. Your choice is saved in a cookie on this device.

    Type at least two letters to search news and guides.

    to open · ↑↓ to move · Esc to close

    SEOGuide

    Robots.txt Disallow All: How to Block Every Crawler

    Two lines in robots.txt ask every well-behaved crawler to stay off your whole site. Here's the exact code, what it doesn't do (it won't deindex anything or keep a staging site private), and the right tool for each job people use it for.

    By

    Published 7 min read

    The Robots.txt Generator's Block everything preset, producing User-agent: * and Disallow: / with a warning that it blocks the whole site
    On this page

    A robots.txt disallow all rule is two lines, User-agent: * followed by Disallow: /, and it asks every crawler that follows the robots.txt standard to stay away from every URL on your site.

    It's the right tool for a narrow job: stopping well-behaved bots from crawling a site that isn't meant for search. It's the wrong tool for most of the jobs people reach for it to do, such as removing pages from Google, hiding a staging site or taking a site offline for a day. This guide shows the code, what it really does, and what to use instead.

    The disallow all code

    Create a plain text file named robots.txt with these two lines and upload it to the root of the host, so it loads at https://example.com/robots.txt:

    User-agent: *
    Disallow: /
    • User-agent: * starts a group of rules for every crawler that doesn't have a group of its own.
    • Disallow: / matches every path on the host, because every URL path starts with a slash.

    robots.txt works per host. www.example.com, example.com and staging.example.com each need their own file, and the rules apply only to the protocol, host and port where the file is served.

    Small changes flip the meaning completely:

    What robots.txt containsWhat Google does
    User-agent: * and Disallow: /Crawls nothing on the host
    User-agent: * and an empty Disallow:Crawls everything
    No robots.txt (a 404)Crawls everything, as if there were no rules
    robots.txt returns a server error (5xx)Stops crawling for 12 hours, then uses its last good copy for up to 30 days

    Google caches robots.txt for up to 24 hours, so a change usually takes up to a day to apply.

    What disallow all doesn't do

    It doesn't remove pages from Google

    Google's documentation is direct about this: robots.txt is "not a mechanism for keeping a web page out of Google." A blocked URL can still appear in search results, without a description, when other pages link to it.

    It also hides your noindex tags. For noindex to work, Google says the page "must not be blocked by a robots.txt file," because Google has to crawl the page to read the tag. Combining disallow all with noindex is a common mistake: the block wins, and the noindex is never seen.

    WordPress learned this the hard way. Its "Discourage search engines from indexing this site" setting used to add Disallow: / to robots.txt. In version 5.3 (2019) it switched to a robots meta tag, because blocked sites could still be listed in search results without content.

    It doesn't keep anything private

    robots.txt is a public file anyone can read, so listing secret paths in it advertises them. RFC 9309, the standard that formalized robots.txt in 2022, says the rules are "not a form of access authorization." Reputable crawlers follow them; others don't have to.

    Even among reputable companies, not everything follows the * group:

    • Some Google crawlers ignore it. For AdsBot-Google, Google's documentation says "the global user agent (*) is ignored." Block it by name if you need to.
    • Fetches a user asks for may skip robots.txt. OpenAI says that for ChatGPT-User, "robots.txt rules may not apply," and Perplexity says Perplexity-User "generally ignores robots.txt rules."

    Pick the right tool for the job

    Your goalUse thisWhy not disallow all
    Keep a staging or dev site out of searchA password (HTTP authentication) or an IP allowlist, plus disallow all as a backupLinks to the site can still get its URLs listed
    Keep a live page out of GoogleA noindex meta tag or X-Robots-Tag header on a crawlable pageGoogle can't read noindex on a blocked page
    Remove pages that are already indexednoindex, a 404 or 410, or a password; the Search Console Removals tool hides them for about six months meanwhileGoogle's removal guide says not to use robots.txt to block
    Take the site down for a day or twoA 503 status on your pagesGoogle's guidance says "Don't disallow all crawling in the robots.txt file"
    Stop AI model trainingBlock the training crawlers by nameDisallow all also blocks Google and Bing
    Reduce server load from crawlingTargeted rules for low-value URLs, such as filters and internal searchBlocking the whole site removes it from search over time

    On the 503 point, Google also warns not to return a 503 for robots.txt itself, "because this blocks all crawling."

    Variations you may need

    Block everyone except Google and Bing

    User-agent: *
    Disallow: /
    
    User-agent: Googlebot
    User-agent: Bingbot
    Allow: /

    Each crawler follows only the most specific group that names it, and Google says specific groups and the * group "are not combined." So Googlebot and Bingbot follow the Allow: / group, and everyone else is blocked. Remember that Google runs other crawlers, such as Googlebot-Image and AdsBot-Google, that you may also want to list.

    Block AI training crawlers only

    To opt out of model training while staying in Google and Bing, name the training bots instead of blocking everyone:

    User-agent: GPTBot
    User-agent: ClaudeBot
    User-agent: Google-Extended
    User-agent: Applebot-Extended
    User-agent: CCBot
    User-agent: Meta-ExternalAgent
    Disallow: /

    Google says Google-Extended "does not impact a site's inclusion in Google Search." Blocking search bots such as OAI-SearchBot or PerplexityBot is a different choice: it can keep you out of those products' answers. Our llms.txt vs robots.txt guide lists which bot does what.

    If your live site launched with disallow all

    This happens after redesigns and CMS moves, when the staging file goes live with the site. To recover:

    1. Replace the file with your real rules and confirm the new version loads at /robots.txt.
    2. Open Search Console's robots.txt report (in Settings), check the fetched version and choose Request a recrawl.
    3. Inspect your key URLs with the URL Inspection tool and request indexing for the most important ones.
    4. Resubmit your sitemap in the Sitemaps report so Google rediscovers pages faster. Our guide to finding the sitemap of a website helps if you don't know its address.
    5. Watch the Page indexing report for the next few weeks. Rankings often come back as pages are recrawled, but it isn't instant.

    Build and test the file with free tools

    Our free Robots.txt Generator has a Block everything preset for staging sites:

    1. Choose Block everything under "Rules for all crawlers." The tool writes User-agent: * and Disallow: /, and shows a warning that this blocks the whole site.
    2. To block only AI training instead, start from Allow everything, then click Block AI training crawlers or tick individual bots.
    3. Add your sitemap URL if the site has one, then copy or download the file and upload it to the root of the host.
    4. Test it with the Robots.txt & Sitemap Checker, which shows whether Googlebot, Bingbot, GPTBot and other crawlers may fetch a URL, and which line decides it.

    What the generator doesn't do: it writes one rule group for all crawlers plus one group for the AI bots you tick. For exceptions like "block everyone except Googlebot," copy the output and add the extra group by hand, as in the example above. It also can't add a password, a noindex tag or a 503; those are set on your server or in your CMS.

    Sources

    All accessed Nov 20, 2026:

    Frequently asked questions

    Is robots.txt case-sensitive?

    Partly. The file name must be lowercase (robots.txt). Field names such as User-agent and Disallow aren't case-sensitive, and neither are user-agent names, but paths are: Disallow: /Private/ doesn't block /private/.

    What does Allow: / do?

    It explicitly allows every URL for that group, which has the same effect as an empty Disallow line. It's useful inside a specific group, such as a Googlebot group in a file that otherwise blocks everyone, because it makes the exception easy to read.

    Does Disallow: / block Bing and the AI crawlers too?

    It asks every crawler that follows the robots.txt standard to stay out, including Bingbot and documented AI crawlers such as GPTBot, ClaudeBot and PerplexityBot. Some Google crawlers, such as AdsBot-Google, ignore the * group and must be named, and fetchers triggered by a user, such as ChatGPT-User and Perplexity-User, may not apply robots.txt rules.

    Should a staging site have a robots.txt file?

    Yes, as a backup. A disallow all file reduces crawling if the address leaks. The real protection is a password or an IP allowlist, and you need a launch step that replaces the file before the site goes live.

    Share

    X (opens in a new tab)LinkedIn (opens in a new tab)WhatsApp (opens in a new tab)

    About the author

    Akif Qureshi

    SEO specialist and founder of SERP Times · 2+ years in SEO

    Akif Qureshi has worked in SEO for more than two years, auditing sites, planning content and tracking how Google updates change real traffic. He started SERP Times to report search changes clearly and to say exactly what site owners should do about them.

    Certifications

    • Google Analytics (Google)
    • Google Ads Search (Google)
    • HubSpot
    • Digital Vidya
    • DSIM
    • Internship Studio

    Connect with Akif on LinkedIn, Instagram and Facebook.

    More SEO guides