AI companies now send several different crawlers to websites, and they don't all do the same job. Some collect content to train AI models. Some build indexes for AI search products that cite and link to sources. Others fetch a single page because a user asked a chatbot about it. Blocking all of them with one robots.txt rule is easy, but it may cost you visibility in AI search that you'd rather keep. This guide explains what the major AI crawlers do according to their operators' own documentation, what blocking each one means, and how to write robots.txt rules for the choice you make.

Three Kinds of AI Crawler

The key to making a sensible decision is understanding that AI crawlers generally fall into three categories:

  • Training crawlers collect content that may be used to train AI models. Blocking them is an opt-out from future training. It generally has no direct effect on whether AI search tools can cite you.
  • Search crawlers build the indexes AI search products use to find, cite, and link to sources. Blocking them reduces your chance of appearing in those answers.
  • User-initiated fetchers visit a specific page when a user asks the AI tool to read it or answer a question about it. Some of these don't follow robots.txt, because the visit is made on behalf of a person.

That separation means you don't have to choose between "all AI" and "no AI." You can, for example, opt out of training while still allowing AI search engines to cite your pages.

The Major AI Crawlers

The details below come from each company's published crawler documentation. User agents and policies change, so check the current documentation before relying on any list, including this one.

OpenAI

  • GPTBot: crawls content that may be used to train OpenAI's foundation models. OpenAI says disallowing GPTBot indicates your content shouldn't be used for training.
  • OAI-SearchBot: used to surface websites in ChatGPT's search features. OpenAI says sites that opt out of OAI-SearchBot won't be shown in ChatGPT search answers, though they may still appear as navigational links.
  • ChatGPT-User: used for actions a user initiates in ChatGPT, such as asking it to visit a page. OpenAI notes that robots.txt rules may not apply because these visits are user-initiated.

Anthropic

  • ClaudeBot: collects web content that may be used for model training. Blocking it signals that your future content should be excluded from training.
  • Claude-SearchBot: crawls to improve search result quality. Anthropic notes that blocking it may reduce your site's visibility in search results.
  • Claude-User: retrieves pages when a person asks Claude a question. Blocking it prevents your content from being retrieved in response to user queries.

Anthropic says its bots honor robots.txt and support the non-standard Crawl-delay directive.

Perplexity

  • PerplexityBot: surfaces and links websites in Perplexity's search results. Perplexity says it isn't used to crawl content for AI foundation models.
  • Perplexity-User: handles user-initiated requests. Perplexity says this fetcher generally ignores robots.txt rules because a user requested the visit.

Google

  • Google-Extended: not a separate crawler but a robots.txt product token. It controls whether content Google crawls may be used for training future Gemini models. Google states that Google-Extended doesn't affect a site's inclusion in Google Search and isn't used as a ranking signal.

Blocking Googlebot itself is a different matter. Google's AI Overviews and AI Mode are built on Google Search, so blocking Googlebot removes you from Google Search entirely. There's no way to block AI Overviews specifically while staying in regular search results.

Reasons to Block AI Crawlers

  • You don't want your content used for training. Blocking training crawlers is the main mechanism these companies offer for opting out.
  • Your content is your product. Publishers of paid or premium content may not want AI tools summarizing it.
  • Server load. Aggressive crawling can strain smaller servers. Rate limiting or a crawl delay, where supported, may be a better fix than a full block.

Reasons to Allow Them

  • AI search visibility. If ChatGPT, Claude, or Perplexity can't crawl your site, they're less likely to cite or link to you in their answers. For many sites, those citations are a growing source of referral traffic and brand exposure.
  • Brand presence. When people ask AI tools for recommendations in your space, you want your site to be part of the answer.
  • Competitive position. If your competitors allow AI search crawlers and you don't, their content is the content that gets cited.

The guides on SEO vs. AEO and optimizing for ChatGPT search cover why AI search visibility is worth protecting.

A Practical Middle Path

For many business sites, a sensible default is to block training crawlers and allow AI search crawlers. You opt out of model training while keeping the ability to be cited and linked in AI search answers. Here's what that looks like in robots.txt:

# Opt out of AI model training
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

# Allow AI search crawlers
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

Adjust it to your own priorities. If you're happy for your content to be used in training, leave the training crawlers allowed. If you want no AI involvement at all, block the search crawlers too, knowing you'll likely lose AI search citations.

The Allow lines are optional if your robots.txt has no broader rule blocking these user agents, but they make your intent explicit. The beginner's guide to robots.txt explains how rules and user-agent groups are matched.

Common Mistakes

  • Blocking everything with a wildcard. A User-agent: * with Disallow: / blocks every compliant crawler, including Googlebot. Always test before publishing.
  • Assuming robots.txt blocks user-initiated fetches. Some user-initiated fetchers don't follow robots.txt by design. If you need hard access control, you need authentication or server-level rules, not robots.txt.
  • Blocking by IP address. Anthropic notes that blocking its IPs isn't a reliable opt-out, because it also prevents the crawler from reading your robots.txt to learn your preferences.
  • Treating llms.txt as a blocking tool. llms.txt is a voluntary content summary, not an access control file. The llms.txt guide explains the difference.
  • Setting and forgetting. New AI crawlers appear and names change. Review your rules periodically against each company's current documentation.

Test and Measure

Before publishing new rules, test them. The Robots.txt Tester parses your file and lets you test any path against any user agent, so you can confirm GPTBot is blocked, OAI-SearchBot is allowed, and Googlebot is unaffected. Then check your server logs over the following weeks to see which AI crawlers visit and how often.

Finally, measure the outcome rather than guessing. If you change your AI crawler rules, you want to know whether your visibility in AI answers changed. OmniSEO tracks your brand's visibility across ChatGPT, Perplexity, Google AI Overviews, and other AI platforms, which shows whether you're being cited before and after a change. The OmniSEO review covers what it does. Disclosure: TopWebPositions may earn a commission if you sign up through this link, at no extra cost to you.