Free tool / AI crawler check

Can AI actually read your site?

ChatGPT, Claude, Perplexity and Gemini each send their own crawler. If your robots.txt blocks them, you will not be quoted in AI answers no matter how good your content is. This checks all 14 of them in a couple of seconds. Free, no signup.

robots.txt / 14 AI crawlers

The domain is not stored

01What it checks

Every crawler, and the answer it feeds.

Blocking is often accidental. A security plugin, a CDN bot rule, or a copied robots.txt from a template will quietly exclude GPTBot and Google-Extended without anyone deciding to.

  • GPTBotOpenAI / ChatGPT model training
  • OAI-SearchBotOpenAI / ChatGPT search results
  • ChatGPT-UserOpenAI / Live browsing in ChatGPT
  • ClaudeBotAnthropic / Claude model training
  • Claude-UserAnthropic / Live browsing in Claude
  • Claude-SearchBotAnthropic / Claude search results
  • PerplexityBotPerplexity / Perplexity answers
  • Perplexity-UserPerplexity / Live fetch for a user question
  • Google-ExtendedGoogle / Gemini training and grounding
  • Applebot-ExtendedApple / Apple Intelligence
  • meta-externalagentMeta / Meta AI
  • AmazonbotAmazon / Alexa and Rufus answers
  • BytespiderByteDance / Doubao and TikTok search
  • CCBotCommon Crawl / The dataset behind many open models

02Why this matters now

Only a handful of sites get named in an answer.

AI Overviews reach around two billion people a month. When someone asks ChatGPT or Perplexity for a recommendation, only a handful of sites get named. You cannot be one of them if the crawler that feeds the answer was turned away at the door.