AI crawler user agents and which to allow in robots.txt
The Awesome GEO crawler list holds 25 AI crawler user agents from 12 operators, read on 11 November 2026. Each of the 25 sits in one of four groups by what its operator says it does: search, training, user-triggered fetches or other jobs.
How this was measured
The readings come from the Awesome GEO catalog of 157 free platforms. A script fetches each platform’s robots.txt file and tests it against each AI crawler’s user agent. 147 of the 157 platforms were last checked on 11 November 2026. The other 10 were last checked on 26 March 2026: Chamber of Commerce, Clutch, CodeSandbox, MuckRack, Observable, Stack Overflow, Tumblr, XenForo Community, Yellow Pages and npm. The script re-checks every platform each Monday at 06:00 UTC.
“Blocked” means the platform’s robots.txt disallows that user agent for the profile page or home page that was tested. “Allowed” means it does not. A platform with no robots.txt file is counted as allowed. A robots.txt that could not be fetched is counted as no reading.
Awesome GEO is built and maintained by Dmitry (Dee) Kargaev, a design engineer and independent GEO researcher in Los Angeles. The list is published as data/crawlers.yaml in the deeflect/awesome-geo repository on GitHub. The raw readings are in the data/sites folder of that repository under a CC0-1.0 licence.
What the four groups mean
A search crawler builds the index an assistant cites from. Training crawlers collect text for model training. When a person asks the assistant about a page, a user-triggered fetcher loads it. The fourth group, other jobs, holds tasks outside those three.
The list has 8 user agents for search, 8 for training, 7 for user-triggered fetches and 2 for other jobs. A robots.txt rule names a crawler by its user agent token, written exactly as in the tables below. Two training tokens, Google-Extended and Applebot-Extended, send no requests of their own and exist only as robots.txt tokens.
User agents by group
The last two columns give the number of the 157 catalog platforms that allow or block each user agent.
Search
| User agent | Operator | What it does | Documentation | Allowed by | Blocked by |
|---|---|---|---|---|---|
| OAI-SearchBot | OpenAI | Surfaces sites in ChatGPT search results | Linked | 132 of 157 | 21 of 157 |
| Claude-SearchBot | Anthropic | Improves search result quality for Claude | None linked | 128 of 157 | 19 of 157 |
| Bingbot | Microsoft | Bing’s standard crawler; handles most of Bing’s crawling | Linked | 143 of 157 | 4 of 157 |
| PerplexityBot | Perplexity | Surfaces sites in Perplexity search results; not used for training | Linked | 130 of 157 | 23 of 157 |
| Applebot | Apple | Search in Spotlight, Siri and Safari; may also ground Apple AI answers | Linked | 139 of 157 | 8 of 157 |
| meta-webindexer | Meta | Indexing for Meta AI search citations | None linked | 128 of 157 | 19 of 157 |
| Amzn-SearchBot | Amazon | Search experiences such as Alexa | None linked | 127 of 157 | 20 of 157 |
| DuckAssistBot | DuckDuckGo | Fetches sources for AI-assisted answers | Linked | 128 of 157 | 19 of 157 |
Operator documentation: OAI-SearchBot, Bingbot, PerplexityBot, Applebot and DuckAssistBot.
Training
| User agent | Operator | What it does | Documentation | Allowed by | Blocked by |
|---|---|---|---|---|---|
| GPTBot | OpenAI | Collects content for model training | None linked | 108 of 157 | 45 of 157 |
| ClaudeBot | Anthropic | Collects content for model training | Linked | 110 of 157 | 43 of 157 |
| Google-Extended | Gemini training and grounding; no effect on Google Search; robots.txt token only | Linked | 113 of 157 | 40 of 157 | |
| Applebot-Extended | Apple | Opts content out of Apple’s AI training; robots.txt token only | None linked | 110 of 157 | 37 of 157 |
| Meta-ExternalAgent | Meta | Training and indexing for Meta AI products | Linked | 114 of 157 | 33 of 157 |
| Amazonbot | Amazon | General crawling; content may be used for training | Linked | 115 of 157 | 32 of 157 |
| CCBot | Common Crawl | Builds Common Crawl’s open web archive | Linked | 103 of 157 | 44 of 157 |
| Bytespider | ByteDance | Widely reported AI crawler; no official vendor doc found | None linked | 105 of 157 | 48 of 157 |
Operator documentation: ClaudeBot, Google-Extended, Meta-ExternalAgent, Amazonbot and CCBot.
User-triggered fetches
| User agent | Operator | What it does | Documentation | Allowed by | Blocked by |
|---|---|---|---|---|---|
| ChatGPT-User | OpenAI | Fetches pages for user actions in ChatGPT and Custom GPTs | None linked | 131 of 157 | 22 of 157 |
| Claude-User | Anthropic | Fetches pages when a Claude user asks | None linked | 129 of 157 | 18 of 157 |
| Google-Agent | Agents on Google infrastructure acting on a user’s request | Linked | 127 of 157 | 20 of 157 | |
| Perplexity-User | Perplexity | Fetches pages when a Perplexity user asks | None linked | 127 of 157 | 20 of 157 |
| Meta-ExternalFetcher | Meta | Fetches pages on user request | None linked | 127 of 157 | 20 of 157 |
| Amzn-User | Amazon | Live fetches on behalf of users | None linked | 127 of 157 | 20 of 157 |
| MistralAI-User | Mistral | User-requested fetches; no automatic crawling or training | Linked | 128 of 157 | 19 of 157 |
Operator documentation: Google-Agent and MistralAI-User.
Other jobs
| User agent | Operator | What it does | Documentation | Allowed by | Blocked by |
|---|---|---|---|---|---|
| OAI-AdsBot | OpenAI | Checks landing pages of ads submitted to ChatGPT | None linked | 128 of 157 | 19 of 157 |
| Google-CloudVertexBot | Crawls requested by site owners for Vertex AI Agents | None linked | 122 of 157 | 25 of 157 |
Which to allow
Search and user-triggered crawlers fetch pages for answers. Training crawlers collect text for training. If you want your site cited, that split is the one to read the tables by, and the choice between the groups is yours.
The robots.txt tool is a free generator and checker for AI crawler rules, with the crawlers grouped as Search, Training, User-triggered and Other. You can load a site’s current robots.txt, start from one of three presets and switch single crawlers between allowed and blocked.
Findings
- In the Awesome GEO catalog of 157 platforms, Bytespider is the most blocked AI crawler user agent, blocked by 48 platforms, followed by GPTBot at 45 and CCBot at 44.
- Bingbot is the least blocked AI crawler user agent in the catalog, blocked by 4 of 157 platforms, followed by Applebot at 8.
- Across the 157 platforms, the number blocking a user agent runs from 4 to 23 for search, 32 to 48 for training, 18 to 22 for user-triggered fetches and 19 to 25 for other jobs.
- OpenAI has the most user agents in the list with 4, ahead of Anthropic, Google, Meta and Amazon with 3 each.
- 12 of the 25 user agents have an operator documentation page linked in the list.
Limits of the readings
The check reads robots.txt only. It does not see blocks made by a firewall, a CDN, a login wall or rate limiting, and it does not show whether a crawler obeys the file.
robots.txt rules can differ by path, so a platform that blocks the tested page may allow other pages. A reading says nothing about whether an AI assistant cites the platform.
The counts describe the 157 catalog platforms, not your site. For a given user agent the allowed and blocked counts need not add up to 157, because a platform whose robots.txt could not be fetched has no reading.
Revisions
- 11 November 2026: first version of this page, read from the Awesome GEO catalog.