You just blocked AI citation crawlers: what does it do?

In brief

Your robots.txt now blocks AI citation crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot...): these are the ones that let ChatGPT search, Perplexity and Claude consult and cite your pages. You need to remove them from the Disallow rules if you want to stay citable.

WarningGEO — AI visibility

What is it?

The robots.txt now blocks AI citation crawlers that were previously allowed.

Why it hurts SEO and AI

Citation crawlers (OAI-SearchBot and ChatGPT-User for OpenAI, PerplexityBot and Perplexity-User for Perplexity, Claude-SearchBot and Claude-User for Anthropic) are the ones that let ChatGPT search, Perplexity or Claude consult and CITE your pages. Blocking them removes you from AI answers. Not to be confused with training crawlers (GPTBot, ClaudeBot, Google-Extended), whose blocking does not affect citation.

When it is NOT a problem

  • You knowingly decided to opt out of AI answer engines: the block is a deliberate policy.
  • The bots involved are training crawlers (GPTBot, ClaudeBot, Google-Extended): blocking them does not affect citation, only model training.
  • The block targets a private or paywalled area not meant to be cited: consistent with your strategy.
Before
# robots.txt
User-agent: OAI-SearchBot
Disallow: /

User-agent: PerplexityBot
Disallow: /
After
# robots.txt
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# You may keep Disallow on training bots if needed
User-agent: GPTBot
Disallow: /

Citation crawlers (live search/browse) grant access to AI answers; training crawlers do not. The two are distinct.

How to fix it

  1. Open your robots.txt and find the new Disallow rules targeting citation crawlers (OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot, Claude-User).
  2. Determine whether this block is intended or inherited from an overly broad User-agent: * rule.
  3. Remove these citation crawlers from the Disallow rules while keeping, if that is your policy, the block on training bots.
  4. Redeploy the robots.txt then re-crawl to confirm the citation crawlers are allowed again.

This scan checks the signals AI reads: structured data, llms.txt, author

Is this issue on your site?

Did a deploy block the bots that cite you in ChatGPT or Perplexity? Enter a URL: Seogard reads your robots.txt and tells citation from training crawlers apart. Free, no credit card.

By clicking Scan, I certify that I own this site or have permission to crawl it.

  • No credit card
  • Result in ~30 s

What you get back

Your content becomes consultable and citable by AI answer engines again.

Seogard catches this regression automatically — through its GEO monitoring.

Frequently asked questions

What is the difference between a citation crawler and a training crawler?

A citation crawler (OAI-SearchBot, PerplexityBot, Claude-SearchBot) reads your pages live to answer and cite you; a training crawler (GPTBot, ClaudeBot) collects text to train a model. Blocking the latter does not affect citation.

Does blocking these crawlers have an immediate effect?

Yes, on their next attempt: a blocked crawler can no longer read your pages, so AI answer engines can no longer consult and cite them.

Related rules

llms.txt removedThe /llms.txt file was deleted. Optional file: no major LLM confirms consuming it to date (a proposal that has not been adopted) — mainly useful for documentation and agents, not a citation factor. FAQ schema removedThe FAQPage schema was removed. Google no longer shows FAQ rich results in the SERP: direct impact is low, but the removal may signal a component deleted by mistake. The visible Q&A content remains readable by Google and LLMs. Author removed (JSON-LD)The author disappeared from the structured data. AI systems cite anonymous content less — a credibility loss.

Related case: AI citation crawlers blocked (robots.txt)

Updated July 10, 2026