AI crawlers explained: search, training and user-triggered bots

Updated 2 November 2026

When people say “block the AI bots” they usually mean very different things. AI companies run several crawlers, and each has a different job. Treating them as one group is the most common reason a site disappears from AI search by accident.

Three kinds of bot

Search bots

These fetch pages so an assistant can show live answers with citations. OpenAI documents OAI-SearchBot as the crawler for ChatGPT search, and says to make sure you are not blocking it if you want your content to appear there. Other examples are Claude-SearchBot, PerplexityBot and Googlebot. Block these and you cannot be cited.

Training bots

These collect content to train future models: GPTBot and ClaudeBot. OpenAI says publishers who want content excluded from potential training should disallow GPTBot. Blocking training does not remove you from search, because search uses a different crawler.

User-triggered fetchers

ChatGPT-User, Claude-User and Perplexity-User visit a page only when a person asks an assistant to open or read it. They do not crawl on a schedule.

Google-Extended is not a crawler

Google's documentation says Google-Extended is a robots.txt token with no separate user-agent string. It controls whether crawled content may be used for Gemini training and grounding, and Google states it does not affect inclusion or ranking in Google Search. Crawling itself is done by Google's existing user agents.

A robots.txt example

This allows search crawlers and opts out of training. Whether to allow training is a business decision, not a technical one.

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Robots.txt is not the only gate

A site can allow a bot in robots.txt and still refuse it at the firewall or CDN. Cloudflare has announced that it is changing the default so AI crawlers are blocked unless they pay creators, so sites behind a CDN should confirm what their provider does by default. The only way to know is to request the page the way each bot would and look at the response code. A 200 means the page was served; 403, 429 or a challenge page means the bot was refused.

Two limits to remember. A user-agent test cannot catch blocking by IP address, and user-agent strings can be spoofed, which is why Google recommends verification. Review 30 days of server or CDN logs to see what real bots received.

Want to see how your own site looks to each bot? Run the free checker.