AI citation crawlers blocked: you drop out of ChatGPT and Perplexity answers

In brief

Blocking AI CITATION crawlers (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) removes you from ChatGPT search, Perplexity and Claude answers. Key nuance: blocking TRAINING crawlers (GPTBot, Google-Extended) does not affect citation, only whether your content is used to train the models.

WarningGEO recommendations

What is it?

The robots.txt blocks AI citation crawlers (OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot, Claude-User).

Why it hurts SEO and AI

These crawlers let answer engines (ChatGPT search, Perplexity, Claude) consult and cite your pages: blocking them removes you from their answers. Important nuance: blocking TRAINING crawlers (GPTBot, Google-Extended) does not affect citation — many tools conflate the two, Seogard does not.

When it is NOT a problem

  • You deliberately block training crawlers (GPTBot, ClaudeBot, Google-Extended) but let citation crawlers through: Seogard does not alert you, it is a legitimate choice.
  • The block targets a private or paid area you do not want cited anyway (back office, paywalled content).
  • Your strategy assumes not appearing in AI answer engines: that is a trade-off, not a technical error.
Before
# robots.txt
User-agent: PerplexityBot
Disallow: /

User-agent: OAI-SearchBot
Disallow: /
After
# robots.txt
# Citation crawlers allowed
User-agent: PerplexityBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

# Training policy stays your call
User-agent: GPTBot
Disallow: /

You can allow citation while refusing training: these are two distinct bots with two independent policies.

How to fix it

  1. Open your robots.txt and find any Disallow directives targeting OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot or Claude-User.
  2. Decide explicitly: do you want to be cited by AI answer engines? If yes, allow these crawlers.
  3. Tell them apart from training bots (GPTBot, Google-Extended): your policy on those can stay at blocked with no impact on citation.
  4. Deploy the fixed robots.txt and confirm it responds at /robots.txt without blocking itself.

This scan checks the signals AI reads: structured data, llms.txt, author

Is this issue on your site?

Are you unintentionally blocking the crawlers that cite you in ChatGPT and Perplexity? Enter a URL: Seogard reads your robots.txt and separates citation from training, where many tools conflate the two. Free, no credit card.

By clicking Scan, I certify that I own this site or have permission to crawl it.

  • No credit card
  • Result in ~30 s

What you get back

Possible presence in AI engine answers — a growing traffic channel.

Seogard catches this regression automatically — through its GEO monitoring.

Frequently asked questions

Should you block GPTBot and PerplexityBot?

They serve two different purposes. GPTBot is for model training; PerplexityBot is for citation in answers. Blocking GPTBot does not affect your presence in answers; blocking PerplexityBot removes you from them.

What is the difference between a training and a citation crawler?

A training crawler collects text to improve a model. A citation crawler reads your page live to answer a user and cite you. Only the second one influences your visibility in AI answers.

Related rules

Content visible only after JavaScriptMost of the body text only exists after JavaScript runs (absent from the raw HTML). AI crawlers (ChatGPT, Perplexity, Claude) read the raw HTML and will probably never see this content. Title missing from raw HTML (JS-only)The <title> is missing from the raw HTML sent by your server but filled in after JavaScript loads. Google may see it with a delay and LLMs (ChatGPT, Perplexity, Claude) probably never see it — they mostly read the raw HTML. Fix: render the title server-side (SSR). Description missing from raw HTML (JS-only)The meta description is missing from the raw HTML but present after JavaScript loads. Google and LLMs mostly read the raw HTML — your search snippet and AI visibility may suffer. Fix: render the meta description server-side (SSR).

Related case: New AI citation crawler blocking

Updated July 10, 2026