Check if your site blocks AI crawlers with Site check

By Dmitry (Dee) Kargaev · Updated November 11, 2026

To check whether your site blocks AI crawlers, run the page address through Awesome GEO’s Site check, a free browser tool. It shows what your robots.txt says about each AI crawler and the HTTP status code your server returns when the page is requested under each crawler’s name.

Awesome GEO is a free, open-source catalog for Generative Engine Optimization, built and maintained by Dmitry (Dee) Kargaev, who wrote this page.

Before you start

You need a browser and the full address of a page you want AI assistants to read. For the last check you also need an access log from your server. The steps follow the tools as described on 11 November 2026.

Check the page with Site check

1. Go to Site check. It has two input modes, chosen with the buttons “URL” and “Paste HTML”.
2. Choose “URL”, then type the full address of the page into the field “Page URL”.
3. Press “Check”. The Awesome GEO server fetches the page, its headers, robots.txt, sitemap.xml and llms.txt. Afterwards the button reads “Check again”.
4. Look at “Problems” first. It shows a count and a list, or “No problems found”.
5. Then read “Crawler access”. It shows one chip per AI crawler, marked allowed, partial, blocked or unknown, read from your robots.txt.
6. Last, read “Live fetch as each bot”. Site check requests the page once per crawler, with that crawler’s name in the user agent, and shows the HTTP status code for each.

What it means when the two sections disagree

Section	Where the answer comes from	What you see
Crawler access	Your robots.txt	allowed, partial, blocked or unknown per crawler
Live fetch as each bot	Your server’s reply to a request carrying the crawler’s name	An HTTP status code per crawler

Crawler access tells you what your rules say. Live fetch tells you what your server did when asked. A crawler can be allowed in the first and still get an error code in the second. When that happens, the file is not the whole story.

A footnote in Site check sets a limit: the live fetch tests the user agent only, not the crawler’s verified IP ranges. The AEO Guy makes a related point in a guide on this check: the user agent header is easy to spoof, so real verification means the source IP falls inside the range the operator publishes. The same guide says a 403 to a crawler’s name is almost always a firewall rule or bot protection module, and that it overrides robots.txt entirely. It also warns that a challenge page can come back as a 200, because the status code says success.

Fix the rules with the robots.txt tool

7. In the robots.txt tool, type your domain into the field “Site” and press “Load”. A note says the file was loaded, or that the site has no robots.txt, so every bot is allowed.
8. Pick a preset: “Let AI search read me, keep training bots out”, which is selected by default, “Allow all” or “Block all AI”.
9. Adjust under “Bots” if you want. Crawlers are grouped as Search, Training, User-triggered and Other, with a count of how many are allowed. Each group has “Allow” and “Block” buttons, and you click a crawler’s chip to switch it. A crawler set differently from its group is tagged “override”.
10. Under “robots.txt”, press “Copy”. The generated file ends with “User-agent: *”, “Allow: /” and a “Sitemap:” line. Publish it as your site’s robots.txt.
11. Back in Site check, press “Check again” and read “Crawler access” and “Live fetch as each bot” once more.

Confirm which crawlers came

A rule check says what should happen. The Crawler log reader is a free tool that finds AI crawler requests in a server access log, so it shows what did happen.

12. Paste log lines into the Crawler log reader, drop a file or press “Choose file”. “Try the sample log” loads sample data. It reads combined-format lines from nginx and Apache and JSON lines from Caddy and Cloudflare. It runs in your browser and nothing is uploaded.
13. Read “AI crawlers”. You get a request total and, for each crawler, its operator, its three most requested paths, and the columns “Requests” and “Last seen”. If none appear, the tool says “No AI crawler requests in this log.”
14. Check “Files”. For /robots.txt, /sitemap.xml and /llms.txt it shows how many requests each got, which crawlers asked for it and the latest status code.

The AEO Guy calls a count of hits per AI agent over the last thirty days the only real proof that access is open. An empty log does not by itself mean a block: the same guide says new and low authority sites are crawled rarely. A guide from Yes AI reports that OpenAI says robots.txt changes can take about 24 hours to reach its search systems, and that Perplexity says up to 24 hours. Wait that long before judging the log.

If it does not work

Site check cannot fetch the page

If the fetch fails, Site check offers “Paste HTML instead” and “Try again”. Local and private addresses are refused, and so are responses over 1,000,000 bytes. In Paste HTML mode you paste the page source into the field “Page HTML”. That mode skips “Crawler access”, “Live fetch as each bot”, “X-Robots-Tag”, “In sitemap.xml” and “Files”, because it has only the HTML to read. Crawler access is not tested there.

Robots.txt allows the crawler but the server does not

When “Crawler access” says allowed and “Live fetch as each bot” shows an error code such as 403, look outside robots.txt. ShowUp Labs names three Cloudflare settings that can each block AI crawlers independently: “Block AI bots” under Security, the managed robots.txt under AI Crawl Control, and Bot Fight Mode. It adds that WordPress security plugins such as Wordfence, and bot rules on Vercel, Netlify and AWS WAF, can catch AI crawlers too. Change the setting at your CDN, host or plugin, then press “Check again”.

Check by hand

You can read the rules yourself. Open yourdomain.com/robots.txt in a browser; per Centium the file sits at the root of the domain. Look for “Disallow: /” under “User-agent: *”, which shuts every crawler out of the whole site. Look also for named groups such as “User-agent: GPTBot” followed by “Disallow: /”. A crawler with its own named group ignores the catch-all group. If there is no robots.txt, everything is open. Each hostname needs its own file, so repeat the check for every subdomain.