Does blocking GPTBot block ChatGPT search?
No. GPTBot and OAI-SearchBot are separate OpenAI crawlers with different purposes. A robots.txt rule for one does not automatically decide what the other can access.
The short answer
OpenAI documents OAI-SearchBot for search in ChatGPT and GPTBot for training. If you block GPTBot, that does not by itself block OAI-SearchBot. If you want to control ChatGPT search access, check the rule for OAI-SearchBot as well.
Robots.txt controls are instructions for crawlers, not a guarantee of what a search service will show. A permitted page can still be left out because of relevance, indexing, content quality or other technical controls.
How the rules differ
| Crawler | Documented purpose | What its rule controls |
|---|---|---|
OAI-SearchBot | ChatGPT search | Whether OpenAI's search crawler may access the matching paths |
GPTBot | Training | Whether the training crawler may access the matching paths |
Google-Extended | Google extended content uses | A separate Google control. It does not control Google Search indexing |
Use separate user-agent groups when you want different policies. For example:
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /That example expresses different crawler rules. It does not promise that ChatGPT will cite the site or that a crawler will visit immediately.
Check the exact page, not only the domain
A broad rule can allow the homepage while a longer, more specific rule blocks a product page, article or directory. Our free AI crawler access checker reads the relevant robots.txt rules for a specific URL and shows the matching rule for each crawler. No account or email is required.
It checks robots.txt only. It does not test a firewall, meta noindex tags, indexing, citations or actual crawler visits.
Robots.txt allows the crawler. Why can it still be blocked?
A CDN or firewall can reject a request before it reaches your server. Robots.txt permission and a successful browser visit do not prove that a real crawler can fetch the page.
- Check the rules for the exact page and named crawler, rather than treating every AI bot as one group.
- Review your CDN's security events for that hostname and path. Look for the action and the rule responsible for a block or challenge. Check origin access logs separately: an edge-blocked request may never appear there.
- Use the crawler provider's published verification information when identifying real visits. Changing a curl user-agent string does not turn your request into a verified crawler request; IP-based rules can treat the two differently.
- If the block is unintended, adjust the specific policy and review subsequent verified requests. Keep protection for private pages and unrelated attacks in place.
Cloudflare's current AI bot policies distinguish Search, Agent and Training behavior. An Allow setting does not remove other firewall rules. Check your own zone's policies instead of assuming that a default applies to every website.
Keep four findings separate: what robots.txt permits, whether a verified crawler fetched the page, whether the provider indexed it, and whether an answer cited it. Our checker establishes only the robots.txt finding.
Sources
OpenAI crawler documentation describes OAI-SearchBot and GPTBot. The Google robots.txt guide explains how user-agent groups and path rules are applied.
This is technical information, not a promise of search visibility or legal advice. Crawler policies can change, so check the provider's current documentation before changing access rules.