Not sure why your competitors rank higher? Find out for free.
Not sure why your competitors rank higher? Find out for free.
See which AI search engines and AI training bots your robots.txt lets in, and whether your site publishes an llms.txt file. Free and instant.
Search bots such as OAI-SearchBot, PerplexityBot and Claude-SearchBot fetch pages so an assistant can answer a question and link to its sources. Training bots such as GPTBot, ClaudeBot and Google-Extended collect content to train future models. Blocking a training bot generally does not remove you from AI search answers, with one exception: Google-Extended also controls grounding in Gemini apps. Blocking a search bot keeps that engine from citing your pages.
It is a business decision. Some owners block training bots to keep their content out of model training. That choice does not affect whether AI search tools can cite you, as long as the search bots stay allowed, except Google-Extended, which also affects Gemini grounding.
llms.txt is a proposed standard: a short plain-text file at the root of your site that summarizes who you are and links to your most important pages. It is optional and still early. Few AI tools read it today, but it takes minutes to add.
Usually your host or firewall refused our request, or the address has a typo. Firewalls that block automated requests often block AI crawlers too, which can keep your site out of AI answers even when robots.txt allows them.
No. It reads robots.txt and llms.txt only. Whether AI tools actually recommend your business also depends on your content, your reviews and your Google Business Profile. Our free audit covers those.