Your robots.txt blocks Googlebot: the deindexing that is coming

In brief

Your robots.txt blocks Googlebot on part of the site: this is the most radical incident possible, the blocked sections are no longer crawled, no longer updated in the index, then degrade. You need to remove the Disallow rule immediately, unless the block is deliberate and documented.

CriticalP0 — Deindexing

What is it?

The robots.txt blocks Googlebot on part of the site.

Why it hurts SEO and AI

This is the most radical incident possible: the blocked sections will no longer be crawled, so no longer updated in the index, then degraded. The staging "Disallow" pushed to production is a classic cause of massive traffic losses.

When it is NOT a problem

  • The block is deliberate and documented on an area that should not be indexed (back office, cart, internal search results).
  • The blocked path contains no page meant for search: no organic traffic is at stake.
  • It is a staging site that must stay out of the index: a global Disallow is exactly what you want there.
Before
# robots.txt (staging pushed to prod by mistake)
User-agent: *
Disallow: /
After
# robots.txt
User-agent: *
Allow: /

# Block only what does not need indexing
Disallow: /cart
Disallow: /admin

"Disallow: /" under User-agent: * cuts Google off from the whole site: the line to never let reach production.

How to fix it

  1. Open the production robots.txt and find the Disallow rule that applies to Googlebot (under User-agent: Googlebot or User-agent: *).
  2. Confirm this block is not intended: in almost all cases, a broad Disallow on Google is an accident (staging pushed to prod).
  3. Remove or narrow the rule so it only blocks paths that truly should not be indexed, then redeploy immediately.
  4. Re-crawl and, in Search Console, test the robots.txt to confirm Googlebot can access the pages to be indexed again.

This scan checks the HTTP status, noindex and canonical of every page

Is this issue on your site?

Is your robots.txt blocking Google without you knowing? Enter a URL: Seogard reads your robots.txt and alerts you on the first Disallow that cuts off Googlebot. Free, no credit card.

By clicking Scan, I certify that I own this site or have permission to crawl it.

  • No credit card
  • Result in ~30 s

What you get back

Google keeps access to everything that should be indexed.

Seogard catches this regression automatically — through its monitoring for your metas and canonicals.

Frequently asked questions

How long before deindexing if Googlebot is blocked?

Crawling stops at once on the blocked paths; the index stops being refreshed, then pages degrade and eventually drop out of results. The longer the block lasts, the deeper the loss.

Is blocking in robots.txt enough to deindex a page?

No, and it is a trap: a page already indexed but blocked from crawling can stay in the index while Google cannot see a noindex. To deindex cleanly, let Google crawl the page and serve it a noindex.

Related rules

Noindex addedA noindex directive was added (robots/googlebot meta tag or X-Robots-Tag HTTP header). The page will disappear from Google results within days. HTTP status code changedThe HTTP status code changed (e.g. 200 to 404 or 301). Critical for server errors (4xx/5xx), warning for redirects (3xx). Canonical missingThe canonical tag is gone. Duplicate content risk — Google no longer knows which URL to index.

Updated July 10, 2026