Firm Beacon

For website owners and SEO teams

Need to block AI training but allow AI search?

Check your robots.txt rules for OAI-SearchBot, GPTBot, Claude, Perplexity, Google and Bing. See search permissions separately from training permissions.

Free. No account or email needed.

See an example: allow AI search, block training

For https://example.com/product, these sample rules allow ChatGPT search and block OpenAI training:

User-agent: OAI-SearchBot
Allow: /
Disallow: /private/

User-agent: GPTBot
Disallow: /

OAI-SearchBot can crawl /product under these rules, but not /private/. GPTBot is blocked on both paths. These are example rules, not example.com's live settings. They do not prove firewall access, indexing or citations.

Use the exact page you want checked. Robots rules can differ between your homepage and other pages.

Leave this blank to fetch the public robots.txt file. Paste the file if your server blocks our checker.

Search access and training are different

OpenAI documents OAI-SearchBot for ChatGPT search and GPTBot for training. Their settings are independent. A site can allow one and block the other. Google-Extended is a content-use control and does not control Google Search indexing.

Does allowing a crawler make my site appear in AI answers?

No. Permission removes one possible barrier. Indexing, relevance, content quality and the search provider's systems still determine what gets shown. A firewall can also block a crawler that robots.txt allows.

If the rules allow a crawler but you suspect a block, see how to check CDN and firewall restrictions separately. A request with a copied bot user-agent is not proof of real crawler access.

Does this work outside the UK?

Yes. The checker works with public HTTP and HTTPS websites, including US and European businesses. It does not apply SRA rules or require a law firm website.

Why check a page instead of just a domain?

A site may allow its homepage but block product pages, articles or an entire directory. Check the URL you want search engines and AI search tools to find.

Sources: OpenAI crawler documentation, Anthropic crawler controls, Perplexity crawlers, Google robots.txt guide, Google-Extended.