Skip to content
Awesome GEO
Platforms
Repos
Tools
Crawlers
GitHub
Platforms
Repos
Tools
Crawlers
About
Home
›
Tools
›
robots.txt
robots.txt
Choose which AI crawlers can read your site.
Load
Let AI search read me, keep training bots out
Allow all
Block all AI
Bots
Search
8
Allow
Block
OAI-SearchBot
override
Claude-SearchBot
override
Bingbot
override
PerplexityBot
override
Applebot
override
meta-webindexer
override
Amzn-SearchBot
override
DuckAssistBot
override
Training
8
Allow
Block
GPTBot
override
ClaudeBot
override
Google-Extended
override
Applebot-Extended
override
Meta-ExternalAgent
override
Amazonbot
override
CCBot
override
Bytespider
override
User-triggered
7
Allow
Block
ChatGPT-User
override
Claude-User
override
Google-Agent
override
Perplexity-User
override
Meta-ExternalFetcher
override
Amzn-User
override
MistralAI-User
override
Other
2
Allow
Block
OAI-AdsBot
override
Google-CloudVertexBot
override
robots.txt
example.com/robots.txt
Copy