Awesome GEO

Platforms that block GPTBot and ClaudeBot, 11 November 2026

By · Updated

As of 11 November 2026, 45 of the 157 platforms in the Awesome GEO catalog block GPTBot, 43 block ClaudeBot and 39 block both. Another 108 allow GPTBot and 110 allow ClaudeBot, and 4 have no reading for each crawler.

How this was measured

The readings come from the Awesome GEO catalog of 157 free platforms. A script fetches each platform’s robots.txt file and tests it against each AI crawler’s user agent. Of the 157 platforms, 147 were last checked on 11 November 2026. The other 10 were last checked on 26 March 2026: Chamber of Commerce, Clutch, CodeSandbox, MuckRack, Observable, Stack Overflow, Tumblr, XenForo Community, Yellow Pages and npm.

Blocked means the platform’s robots.txt disallows that user agent for the profile page or home page that was tested. Allowed means it does not. A platform with no robots.txt file counts as allowed, and a robots.txt that could not be fetched counts as no reading. The script re-checks every platform each Monday at 06:00 UTC.

Awesome GEO is built and maintained by Dmitry (Dee) Kargaev, a design engineer and independent GEO researcher in Los Angeles.

Platforms that block GPTBot or ClaudeBot

The table lists the 49 platforms that block GPTBot, ClaudeBot or both, grouped by category. Each cell reads blocked, allowed or no reading. The columns show five of the crawlers the catalog tests.

PlatformCategoryGPTBotClaudeBotOAI-SearchBotClaude-SearchBotPerplexityBot
AlignableBusiness Directoriesblockedblockedblockedallowedblocked
CrunchbaseBusiness Directoriesblockedblockedallowedallowedallowed
F6SBusiness Directoriesblockedblockedallowedblockedallowed
FoursquareBusiness Directoriesblockedallowedallowedallowedallowed
GlassdoorBusiness Directoriesblockedblockedallowedallowedallowed
KompassBusiness Directoriesallowedblockedallowedallowedallowed
LinkedIn CompanyBusiness Directoriesblockedblockedallowedallowedblocked
YelpBusiness Directoriesblockedblockedblockedblockedblocked
IndieHackersCommunity Platformsblockedblockedblockedblockedblocked
LobstersCommunity Platformsblockedblockedblockedblockedblocked
QuoraCommunity Platformsblockedblockedblockedblockedblocked
RedditCommunity Platformsblockedblockedblockedblockedblocked
SpiceworksCommunity Platformsblockedblockedallowedallowedallowed
ArticleBizContent Syndicationblockedblockedallowedallowedallowed
Hacker NoonContent Syndicationblockedblockedallowedallowedallowed
MediumContent Syndicationblockedblockedallowedallowedallowed
Quora SpacesContent Syndicationblockedblockedblockedblockedblocked
ScribdContent Syndicationallowedblockedallowedallowedallowed
SitePointContent Syndicationblockedallowedallowedallowedallowed
SlideShareContent Syndicationallowedblockedallowedallowedallowed
TumblrContent Syndicationallowedblockedallowedno readingallowed
CoroflotDesign Portfoliosblockedallowedallowedallowedblocked
Figma CommunityDesign Portfoliosblockedblockedblockedblockedblocked
SiteInspireDesign Portfoliosblockedblockedallowedallowedallowed
CodePenDeveloper Profilesblockedblockedallowedallowedallowed
CodebergDeveloper Profilesblockedblockedblockedallowedblocked
GiteaDeveloper Profilesblockedblockedblockedallowedblocked
HackerRankDeveloper Profilesblockedblockedblockedblockedblocked
LaunchpadDeveloper Profilesblockedblockedblockedblockedblocked
Academia.eduKnowledge Basesblockedblockedallowedallowedallowed
MuckRackNiche Directoriesblockedblockedallowedno readingallowed
QwotedNiche Directoriesblockedblockedallowedallowedallowed
DiigoSocial Bookmarkingblockedblockedblockedblockedblocked
FlipboardSocial Bookmarkingblockedallowedblockedallowedallowed
StumbleUpon / MixSocial Bookmarkingblockedblockedblockedblockedblocked
BehanceSocial Profilesblockedblockedallowedallowedallowed
FlickrSocial Profilesblockedblockedallowedallowedallowed
GoodreadsSocial Profilesblockedallowedallowedallowedallowed
GravatarSocial Profilesblockedallowedallowedallowedallowed
InstagramSocial Profilesblockedblockedblockedblockedblocked
LetterboxdSocial Profilesblockedblockedallowedallowedallowed
MastodonSocial Profilesblockedblockedallowedallowedallowed
PinterestSocial Profilesblockedblockedblockedblockedblocked
Spotify for CreatorsSocial Profilesblockedblockedblockedblockedblocked
ThreadsSocial Profilesblockedblockedblockedblockedblocked
TikTokSocial Profilesblockedblockedblockedblockedblocked
VimeoSocial Profilesblockedblockedallowedallowedallowed
WhatsApp ChannelsSocial Profilesblockedblockedallowedallowedblocked
X / TwitterSocial Profilesblockedblockedblockedblockedblocked

MuckRack and Tumblr carry the 26 March 2026 reading. Every other row was checked on 11 November 2026.

The raw data is in the data/sites folder of the deeflect/awesome-geo repository on GitHub, under a CC0-1.0 licence.

Findings

17 of the 39 platforms that block both training crawlers still allow both OAI-SearchBot and Claude-SearchBot, the crawlers OpenAI and Anthropic use for search: Academia.edu, ArticleBiz, Behance, CodePen, Crunchbase, Flickr, Glassdoor, Hacker Noon, Letterboxd, LinkedIn Company, Mastodon, Medium, Qwoted, SiteInspire, Spiceworks, Vimeo and WhatsApp Channels.

Across the whole catalog, OAI-SearchBot is blocked by 21 of the 157 platforms and Claude-SearchBot by 19, against 45 for GPTBot and 43 for ClaudeBot.

Six platforms block GPTBot but not ClaudeBot (Coroflot, Flipboard, Foursquare, Goodreads, Gravatar and SitePoint), and 4 block ClaudeBot but not GPTBot (Kompass, Scribd, SlideShare and Tumblr).

Reddit and Spotify for Creators are the 2 platforms that block every AI crawler the catalog tests, which includes crawlers beyond the five in the table.

Among the other crawlers tested, 23 of the 157 platforms block PerplexityBot and 40 block Google-Extended.

Platforms that block both crawlers, by category

The table below gives, for each of the catalog’s 9 categories, how many platforms block both GPTBot and ClaudeBot.

CategoryBlock bothPlatforms in category
Developer Profiles529
Business Directories627
Content Syndication424
Community Platforms513
Social Profiles1227
Knowledge Bases18
Design Portfolios29
Niche Directories212
Social Bookmarking28

Limits

The check reads robots.txt only. It does not see blocks made by a firewall, a CDN, a login wall or rate limiting, and it does not show whether a crawler obeys the file.

Robots.txt rules can differ by path, so a platform that blocks the tested page may allow other pages.

A robots.txt reading says nothing about whether an AI assistant cites the platform.

Revisions

  • 11 November 2026: first version of this page, read from the Awesome GEO catalog.