What the checker looks at
- Syntax: lines the file does not understand, misplaced directives and rules that do not belong to any group.
- Rules: what each group allows and blocks.
- Your homepage: whether the rules let crawlers reach it.
- Sitemap declarations: whether robots.txt points to your sitemap.
A site with no robots.txt is a valid result: the report says so and what that means.
What it cannot see
robots.txt only states your rules. A crawler you allow can still be blocked by your CDN, firewall or bot protection before it reaches your server. The AI crawler access check explains how to test that, and which AI crawlers to allow.
What to do next
Fix findings in order of severity. The AI Crawler Access and Technical Fix pack ($650 / ₹22,000) rewrites the file with one clean group per purpose and checks your server logs for real crawler visits.
Questions
Does robots.txt stop a page being indexed?
No. It stops crawling. Google can still index a blocked URL from links pointing to it. Use noindex to keep a page out of results.
Does blocking GPTBot remove me from ChatGPT?
No. ChatGPT search uses OAI-SearchBot. GPTBot relates to model training.