Are LLMs crawling my site?

There is a question that seems simple but isn’t: can language models read my website?

The intuitive answer is to check the robots.txt file. If it says Allow: /, everything is fine. However, robots.txt is a statement of intent, not a guarantee of access. Between that statement and the bot lie a firewall, network rules, and a bot detection system. For this analysis, I built a tool to answer that question with data rather than assumptions. It is free and available below.

What it checks

You enter a URL, and the tool performs two actions for each relevant AI crawler.

• First, it reads the domain’s robots.txt and determines whether that specific path is allowed for that user agent. It doesn’t just look for the bot’s name; it applies the longest-match criterion, understands wildcards and $ anchors, and resolves which rule group applies when multiple exist. A Disallow: /productos/ combined with an Allow: /productos/publico/ yields different results depending on the specific URL—and that matters.
• Next, it makes an actual request, identifying itself with each bot’s user agent, and observes the server’s response. This is where surprises often arise.
• It also checks X-Robots-Tag headers and <meta name="robots"> tags, and identifies the presence of an application firewall—the kind commonly used to protect corporate sites.

The three bot groups, and why they aren’t interchangeable

The results are presented in three separate blocks because blocking one group versus another has very different consequences.

• Search. Scheduled crawling. These bots visit on their own initiative, save your page, and cite it later when someone asks a related question. Examples include OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, bingbot, Applebot, and Amazonbot. Blocking them excludes you from the index used to generate answers.

• User-initiated. There is no scheduling here. Someone might be typing into ChatGPT right now and pasting your URL, or asking about your brand, prompting the assistant to check your website in real-time to provide an answer. These are ChatGPT-User, Claude-User, and Perplexity-User.

This is the group that is most painful to block—and the one most often blocked unintentionally. When blocked, someone explicitly looking for you receives an “unable to access this page” error. You don’t just lose a ranking position; you lose a person who had already sought you out. They get blocked by accident because anti-bot systems view them simply as automated traffic, lacking further context.

Training. Data collection for model training: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, and CCBot. Blocking them doesn’t affect whether you get cited today, but rather what models know about you tomorrow. This is the group most people choose to block intentionally, and it is a legitimate decision.

A technical note: Google-Extended and Applebot-Extended are not crawlers. They are usage-control tokens, so they are only checked within robots.txt. The tool indicates this status rather than pretending to have made an actual request.

The case that changed the tool

During testing, I analyzed a page hosted behind a corporate firewall. Result: sixteen accessible bots, zero blocked, all clear (green).

There was just one odd detail. The report indicated a noindex directive on a page that was perfectly indexed in Google.

That noindex tag didn’t belong to the page itself. It belonged to the block page returned by the firewall—pages that carry that directive specifically to avoid being indexed. And that block response was served with an HTTP 200 status code.

This isn’t unusual. Returning a block response with a 200 code instead of a 403 is a common configuration with a specific purpose: it prevents an attacker from deducing—based on the code—that they have encountered a defense mechanism. The side effect is that any automated check looking only at the status code yields a false result.

The current version reads the response body and recognizes block pages and verification challenges by their signatures. If it detects one, the verdict is “blocked”—even if the status code is 200—and a notification appears at the very top stating that the content received is not your actual page. In this scenario, meta robots tags and X-Robots-Tags are no longer displayed, as they belong to the firewall’s block page rather than your site.

If your website sits behind an application firewall, this is likely the most useful insight the tool provides.

What this tool cannot tell you

I prefer to be explicit about the limitations, as a false “all clear” is worse than having no data at all, and this tool is merely a test.

It spoofs the user agent, not the IP address. Legitimate bots publish their IP ranges, and many firewalls verify them via reverse DNS in addition to checking the user agent name. If your WAF performs this check, it might block this test while allowing the genuine bot through. A “blocked” result is a signal to investigate, not a definitive verdict.

It does not check whether your IP range is allowed by the firewall. That can only be determined by examining the configuration—information held by your security team.