Site check
Can AI crawlers reach a page, and can they read it.
Checks
- Crawler access
- robots.txt, 25 bots
- Live fetch as each bot
- HTTP status, 23 bots
- Indexing
- noindex, canonical
- Page
- title, h1, text without JS, date
- Structured data
- JSON-LD
- Files
- robots.txt, sitemap.xml, llms.txt
Problems
Crawler access
Search
OAI-SearchBot Claude-SearchBot Bingbot PerplexityBot Applebot meta-webindexer Amzn-SearchBot DuckAssistBot
Training
GPTBot ClaudeBot Google-Extended Applebot-Extended Meta-ExternalAgent Amazonbot CCBot Bytespider
User-triggered
ChatGPT-User Claude-User Google-Agent Perplexity-User Meta-ExternalFetcher Amzn-User MistralAI-User
Other
OAI-AdsBot Google-CloudVertexBot
Live fetch as each bot
Search
OAI-SearchBot Claude-SearchBot Bingbot PerplexityBot Applebot meta-webindexer Amzn-SearchBot DuckAssistBot
Training
GPTBot ClaudeBot Meta-ExternalAgent Amazonbot CCBot Bytespider
User-triggered
ChatGPT-User Claude-User Google-Agent Perplexity-User Meta-ExternalFetcher Amzn-User MistralAI-User
Other
OAI-AdsBot Google-CloudVertexBot
Indexing
Page
Structured data
Files
This tests the user agent only. Real crawlers also come from verified IP ranges, which your firewall may treat differently.