Free tool
Can AI assistants read your site?
Per-crawler verdicts from your real robots.txt — including the rule that allowed or blocked each one.
The scanner fetches your real robots.txt and evaluates all seven AI crawlers — about 30 to 60 seconds, 6 scans per hour per visitor.
How this works
- Source: a live scan of your domain by the WebsiteSpa engine, which fetches robots.txt and evaluates all seven primary AI crawlers against it.
- Each crawler row shows the vendor, what the crawler does, and the rule that decided the verdict — so you can see exactly which line to change.
- One blocked crawler can be enough to keep a whole assistant from quoting you.
Many companies block AI crawlers by accident — a copied robots.txt, a security plugin, or a blocked wildcard nobody meant to add. This check names the line.
Next step: run the free AI-readiness scan. It crawls your real pages and grades how well assistants can read and quote them — three domains free, no card.
Run the free scan