Analyzes robots.txt directives for AI crawlers: GPTBot, Google-Extended, ClaudeBot, PerplexityBot, and others.
This check reads your robots.txt file and specifically looks for rules that affect AI company crawlers. It checks for GPTBot (used by OpenAI for ChatGPT), Google-Extended (used by Google for Gemini training), ClaudeBot (used by Anthropic for Claude), PerplexityBot, CCBot (used by Common Crawl), Bytespider (used by ByteDance/TikTok), and Amazonbot. It clearly tells you which AI crawlers are currently allowed and which are blocked on your site.
AI companies send web crawlers to collect content for training their AI models and powering AI-generated search answers. Some website owners want to allow this (for potential visibility when AI tools cite their content), while others prefer to block it (to protect their original content from being used without permission). This is a business and policy decision that depends on your goals, not an SEO best practice.
Blocking AI crawlers is NOT an SEO failure — this check is purely informational. It helps you understand your current policy and make an informed decision. Allowing AI crawlers may increase your chances of being cited in AI-generated answers, while blocking them protects your content from being used in AI training datasets. Either choice is valid.
Run a free audit to see how your site performs on AI Crawler Access and 159 other checks.
Start Free Audit