Description
Parses /robots.txt and checks whether the major AI crawlers (GPTBot, ClaudeBot, PerplexityBot, CCBot, Google-Extended, and others) are allowed to crawl the site root. A blanket User-agent: * / Disallow: / fails the check outright.
Parses /robots.txt and checks whether the major AI crawlers (GPTBot, ClaudeBot, PerplexityBot, CCBot, Google-Extended, and others) are allowed to crawl the site root. A blanket User-agent: * / Disallow: / fails the check outright.