When someone asks ChatGPT, Claude or Perplexity for the best roofer, med spa or law firm in town, those tools do something simple before they answer. They send a crawler to read websites. If a crawler cannot read yours, the assistant has nothing of yours to quote, and it recommends someone it could read.
Most business owners have never looked at the file that controls this. It is called robots.txt, it sits at the root of your website, and it can quietly say “no” to AI tools without anyone at the business ever deciding that.
How a Website Says No to AI
There are two common ways a site ends up invisible to AI search.
The first is robots.txt. It is a short text file that lists which crawlers may read which parts of your site. A single line can block a specific AI crawler from the whole site. These lines can arrive by accident: a plugin default, a template copied from another site, or a well-meaning developer who blocked “AI bots” as a group without separating the ones that matter.
The second is your firewall. Many hosting and security services include bot protection, and some offer a one-click setting to block AI crawlers. That setting refuses the request before robots.txt is ever read, so the file can look perfectly open while the door is locked.
Search Bots and Training Bots Are Not the Same
This is the part most people miss. Each major AI company runs more than one crawler, and they do different jobs.
Search crawlers fetch pages so an assistant can answer a question and link to its sources. OpenAI uses OAI-SearchBot for ChatGPT search. Anthropic uses Claude-SearchBot. Perplexity uses PerplexityBot. Block these and that tool cannot cite your pages.
Training crawlers collect content to train future AI models. OpenAI uses GPTBot and Anthropic uses ClaudeBot. Google uses a token called Google-Extended, which controls whether your content can be used for Gemini training and for grounding answers in Gemini. Google says it does not affect your inclusion or ranking in Google Search.
Blocking a training bot is a legitimate business decision. Some owners would rather their content not train anyone’s model, and that choice does not stop search citations. Blocking a search bot is usually an accident, and it costs you visibility you never meant to give up.
There is also a third kind: agents that fetch a page because a user asked about it directly, such as ChatGPT-User and Perplexity-User. OpenAI says robots.txt rules may not apply to these user-initiated requests, and Perplexity says its user agent generally ignores them. So robots.txt is not a reliable way to control that traffic in either direction.
What Is llms.txt?
You may also hear about llms.txt. It is a proposed standard: a short plain-text file at the root of your site that summarizes who you are and links to your most important pages, written so AI tools can read it easily.
It is optional and it is early. It takes minutes to add, and it gives AI tools an accurate summary of your business if they choose to read it. Do not expect it to change rankings or citations on its own. It is a small, cheap step, not a strategy.
What to Check on Your Own Site
- Your robots.txt file. Open yourdomain.com/robots.txt in a browser. Look for “Disallow: /” under OAI-SearchBot, Claude-SearchBot, PerplexityBot, or under “User-agent: *”.
- Your firewall or bot protection. Ask whoever manages your hosting whether an AI bot blocking setting is turned on, and whether that was a deliberate choice.
- Your llms.txt file. Open yourdomain.com/llms.txt. A 404 means you do not have one, which is common and easy to fix.
- The rest of your AI visibility. Crawler access only means AI tools can read your site. Whether they recommend you depends on what they find: how AI decides which businesses to trust, your reviews, and your Google Business Profile.
If you would rather not read robots.txt line by line, our free AI Crawler Checker does it for you. Enter your web address and it shows which AI search and training crawlers your site allows, flags any it could not reach, and tells you whether you have an llms.txt file.
What We Found on Our Own Clients
When we checked the 42 client websites we manage in September 2026, none of them blocked an AI search crawler in robots.txt. That is the good news, and it is what you would expect from sites built with search in mind. The gaps we did find were elsewhere, which is why crawler access is the first question, not the only one.
If you want to go further than access, we have written about how to make AI recommend your business and why a strong FAQ page helps in AI search.
Frequently Asked Questions
Can robots.txt stop ChatGPT from mentioning my business?
Blocking OAI-SearchBot stops ChatGPT search from reading your pages, so it has nothing of yours to cite. ChatGPT can still mention your business from other sources, like reviews, directories and news coverage.
Should I block AI training bots?
It is your call. Blocking training bots such as GPTBot and ClaudeBot keeps your content out of future model training and generally does not affect AI search citations. Google-Extended is the one to think about separately, because it also controls grounding in Gemini apps.
Does llms.txt improve my rankings?
There is no evidence that it does. It is an optional, early proposal that gives AI tools a clean summary of your site. It is worth the few minutes it takes, but it is not a ranking factor.
Why would a checker say it could not read my site?
Usually your host or firewall refused the request, or the address has a typo. Security settings that refuse automated visitors can refuse AI crawlers too, even when robots.txt allows them.
Is being readable enough to get recommended by AI?
No. Access is the entry ticket. AI tools still weigh what they find: clear service pages, consistent business details, reviews and your Google Business Profile.
Check Your Site in a Minute
Start with the free AI Crawler Checker to see which AI crawlers can reach your site. When you want the full picture of how your business shows up on Google Maps and in AI answers, get your free audit.