AI citation crawlers blocked: you drop out of ChatGPT and Perplexity answers

In brief

Blocking AI CITATION crawlers (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) removes you from ChatGPT search, Perplexity and Claude answers. Key nuance: blocking TRAINING crawlers (GPTBot, Google-Extended) does not affect citation, only whether your content is used to train the models.

WarningGEO recommendations

What is it?

The robots.txt blocks AI citation crawlers (OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot, Claude-User).

Why check this signal?

These crawlers let answer engines (ChatGPT search, Perplexity, Claude) consult and cite your pages: blocking them removes you from their answers. Important nuance: blocking TRAINING crawlers (GPTBot, Google-Extended) does not affect citation — many tools conflate the two, Seogard does not.

When it is NOT a problem

  • You deliberately block training crawlers (GPTBot, ClaudeBot, Google-Extended) but let citation crawlers through: Seogard does not alert you, it is a legitimate choice.
  • The block targets a private or paid area you do not want cited anyway (back office, paywalled content).
  • Your strategy assumes not appearing in AI answer engines: that is a trade-off, not a technical error.
Before
# robots.txt
User-agent: PerplexityBot
Disallow: /

User-agent: OAI-SearchBot
Disallow: /
After
# robots.txt
# Citation crawlers allowed
User-agent: PerplexityBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

# Training policy stays your call
User-agent: GPTBot
Disallow: /

You can allow citation while refusing training: these are two distinct bots with two independent policies.

How to fix it

  1. Open your robots.txt and find any Disallow directives targeting OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot or Claude-User.
  2. Decide explicitly: do you want to be cited by AI answer engines? If yes, allow these crawlers.
  3. Tell them apart from training bots (GPTBot, Google-Extended): your policy on those can stay at blocked with no impact on citation.
  4. Deploy the fixed robots.txt and confirm it responds at /robots.txt without blocking itself.

This scan checks robots directives and structured elements on the page

Is this issue on your site?

Are you unintentionally blocking the crawlers that cite you in ChatGPT and Perplexity? Enter a URL: Seogard reads your robots.txt and separates citation from training, where many tools conflate the two. Free, no credit card.

Create your account and try it free for 14 days. No credit card. By continuing, I confirm that I own this site or have permission to analyze it.

  • No credit card
  • Account required · 14-day trial

What you get back

Possible presence in AI engine answers — a growing traffic channel.

Check this signal with Seogard's GEO monitoring.

Frequently asked questions

Should you block GPTBot and PerplexityBot?

They serve two different purposes. GPTBot is for model training; PerplexityBot is for citation in answers. Blocking GPTBot does not affect your presence in answers; blocking PerplexityBot removes you from them.

What is the difference between a training and a citation crawler?

A training crawler collects text to improve a model. A citation crawler reads your page live to answer a user and cite you. Only the second one influences your visibility in AI answers.

Related rules

Content visible only after JavaScriptThe browser displays substantially more words after JavaScript than the text extracted from the HTTP response. Serve the main content in the initial HTML when appropriate, then verify it before and after rendering. Title missing from raw HTML (JS-only)The title tag is missing from the raw HTML but filled in after JavaScript executes. Description missing from raw HTML (JS-only)The meta description exists only in the JavaScript render, not in the raw HTML.

Related case: New AI citation crawler blocking

Updated July 10, 2026