Is my robots.txt blocking ChatGPT and other AI crawlers?

Updated 2026-10-17

Short answer: AI assistants find and quote sites through their own crawlers, and a robots.txt line (sometimes added by a plugin or a host's default) can block them without you knowing. Whether to allow them is your choice; the point is to know.

How to check it yourself

  1. Open /robots.txt on your domain.
  2. Look for User-agent lines naming AI crawlers such as GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot or Google-Extended, followed by Disallow: /.
  3. Also check User-agent: *, which applies to every crawler not named.
  4. If you use Cloudflare, check its AI crawler settings too: they can block crawlers separately from robots.txt.

Why it matters

AI assistants like ChatGPT, Claude and Perplexity answer more and more questions by reading websites. If your robots.txt blocks their crawlers, they can't read your pages, so they can't quote them or recommend you. Blocking can be the right choice, but it should be a decision, not an accident.

Common causes

  • A security or SEO plugin added AI-crawler blocks by default.
  • A hosting or CDN setting blocks known AI bots for every site on the account.
  • A broad rule like Disallow: / for unknown bots catches AI crawlers too.
  • Someone copied a robots.txt from another site without reading it.

What Preventionlabs shows

Every report's site files check lists which AI crawlers your robots.txt blocks, and checks for an llms.txt file, security.txt (present and not expired) and ads.txt when the site shows ads.

It reads robots.txt. Blocks set elsewhere, such as in Cloudflare's settings, need checking there.

Price

First report free, no card needed. Then A$50 a month, up to 10 domains.

Run your first report free