Validates the presence and accessibility of the robots.txt file.
A `robots.txt` file is a plain text file located at the root of your website (e.g., `yourdomain.com/robots.txt`) that tells search engine crawlers which pages or sections they are allowed or forbidden to crawl.
It's the first place search engine bots look. It allows you to prevent crawling of private areas, duplicate content, or unimportant pages, helping you guide 'crawl budget' to your most important pages.
An incorrectly configured `robots.txt` file is one of the most catastrophic SEO errors. You can accidentally block your entire website from being indexed by search engines.
# Allow all crawlers full access
User-agent: *
Allow: /
# Disallow crawlers from accessing a specific folder
User-agent: *
Disallow: /private/Run a free audit to see how your site performs on Robots.txt and 159 other checks.
Start Free Audit