About the AI Crawler Access Checker
AI companies use different bots for different jobs. Some crawl the web to collect training data, such as GPTBot, CCBot and Google-Extended (a control token Google reads rather than a separate crawler). Others fetch pages to answer searches or user requests, such as OAI-SearchBot, ChatGPT-User, Claude-SearchBot and PerplexityBot. This AI crawler checker fetches your public robots.txt and tests more than 20 of these user-agents against a URL, grouping training crawlers apart from AI search and retrieval bots.
That split matters because you may want to appear in AI search answers while opting out of training, or the reverse. The results show which bots your current rules allow or block and which rule matched. Remember robots.txt is a voluntary standard: reputable bots follow it, but it does not technically prevent access.
How to use it
- Enter your domain or the specific URL you want to test.
- Run the check to fetch robots.txt and test each AI user-agent.
- Compare the training crawler results with the AI search bot results.
- Update robots.txt to match your policy, using our robots.txt generator if needed.
Frequently asked questions
What is the difference between GPTBot and OAI-SearchBot?
GPTBot is OpenAI's crawler for collecting content that may be used to train its models. OAI-SearchBot is used to surface and link websites in ChatGPT search results. ChatGPT-User fetches pages when a user asks ChatGPT to visit them. OpenAI lets you control each separately in robots.txt, so you can block training while still allowing search.
What does Google-Extended control?
Google-Extended is a robots.txt token that lets you choose whether content Googlebot crawls may be used to train and ground Google's Gemini models. It is not a separate crawler and blocking it does not affect Google Search indexing or rankings. AI features inside Google Search are governed by regular Googlebot access and snippet controls.
Do all AI crawlers respect robots.txt?
Major AI companies state that their crawlers follow robots.txt, and you can verify their documented user-agent names. However, robots.txt is voluntary, some scrapers ignore it, and others may use generic user-agents. If you need stronger control, use firewall or CDN bot rules, which can block or rate-limit traffic regardless of robots.txt.