Crawlers
The AI user agents worth knowing, and what each one is for.
Search8
OAI-SearchBot Surfaces sites in ChatGPT search results OpenAI Claude-SearchBot Improves search result quality for Claude Anthropic Bingbot Bing's standard crawler; handles most of Bing's crawling Microsoft PerplexityBot Surfaces sites in Perplexity search results; not used for training Perplexity Applebot Search in Spotlight, Siri, and Safari; may also ground Apple AI answers Apple meta-webindexer Indexing for Meta AI search citations Meta Amzn-SearchBot Search experiences such as Alexa Amazon DuckAssistBot Fetches sources for AI-assisted answers DuckDuckGo
Training8
GPTBot Collects content for model training OpenAI ClaudeBot Collects content for model training Anthropic Google-Extended Gemini training and grounding; no effect on Google Search Google Applebot-Extended opts content out of Apple's AI training Apple Meta-ExternalAgent Training and indexing for Meta AI products Meta Amazonbot General crawling; content may be used for training Amazon CCBot Builds Common Crawl's open web archive Common Crawl
Bytespider Widely reported AI crawler; no official vendor doc found ByteDance
User-triggered7
ChatGPT-User Fetches pages for user actions in ChatGPT and Custom GPTs OpenAI Claude-User Fetches pages when a Claude user asks Anthropic Google-Agent Agents on Google infrastructure acting on a user's request Google Perplexity-User Fetches pages when a Perplexity user asks Perplexity Meta-ExternalFetcher Fetches pages on user request Meta Amzn-User Live fetches on behalf of users Amazon MistralAI-User User-requested fetches; no automatic crawling or training Mistral