Read your robots.txt for AI crawler rules and decide, with reasons, which crawlers to allow or block.
Small business owners hear conflicting claims about AI crawlers, often in the same forum thread. One claim is that every site should block AI bots because AI companies take content for free. Another is that blocking AI bots makes you invisible in ChatGPT. A third is that search bots and training bots are the same bots. These claims circulate without sources, and they leave owners unsure what their own robots.txt file should say.
The confusion comes from how AI companies work. Each company runs several crawlers for different purposes, under different names. Once you know which name does which job, you can decide about each crawler separately.
robots.txt gives instructions to crawlers by name, using a line that starts "User-agent:". Lesson 4.1, "Robots.txt, noindex and canonicals do different jobs", explained how those lines work. AI companies publish the names of their crawlers so that site owners can address them by name in the file. One company may run several crawlers for different jobs.
OpenAI is a good example. OAI-SearchBot is used to find and surface websites in ChatGPT's search features. GPTBot is used to collect content that may be used to train OpenAI's models. Each is controlled separately. A site can allow OAI-SearchBot, so its pages can appear and be cited in ChatGPT search answers, while blocking GPTBot, so its content is not used for training. It can also do the reverse, allow both, or block both.
Other providers, including Perplexity and Anthropic, publish their own crawler names and purposes. A provider may have one crawler name for search, another for training, and another for fetching a page when a user asks for it. Read each provider's current documentation to get the names and what each one does.
Google handles this differently from OpenAI. Googlebot crawls for Google Search, and Search includes AI Overviews and AI Mode. Lesson 8.1, "How AI Overviews and AI Mode choose their links", explained that these are part of Search.
Google also offers a separate product token called Google-Extended. A robots.txt rule for Google-Extended controls whether content Google crawls from your site may be used for training and grounding its Gemini models. Google says Google-Extended does not affect a site's inclusion in Google Search and is not used as a ranking signal. Blocking it does not remove your site from Google Search or from AI Overviews.
If you want to limit how your content appears in AI Overviews, Google points to the same controls used for ordinary results, such as nosnippet and noindex. They have the same effects on how your pages appear in Search.
It is tempting to block everything with "AI" in its name. That treats two kinds of crawler as one, and they do different things for you.
A search crawler fetches pages so an assistant can cite them in answers, usually with a link. If you block it, that assistant is less able to retrieve and cite your pages. For a small business that wants customers to find it, this is often a loss.
A training crawler collects content that may be used to train future models. Blocking it is a choice about how your content is used. According to the providers that separate the two, blocking a training crawler does not stop your pages being cited by their search features.
Many small businesses allow search crawlers so they can be cited. They then decide separately about training crawlers, based on how they feel about their content being used for training. Reasonable owners reach different decisions on training crawlers. What matters is that you choose deliberately and do not inherit a rule from a plugin or a forum thread.
Never use a blanket rule to deal with AI. A blanket rule is "User-agent: *" followed by "Disallow: /", and it blocks Googlebot and every other search engine along with the AI crawlers.
Crawlers can also be blocked outside robots.txt. Some hosting, security and content delivery services have settings that block AI crawlers, and some of these are on by default. If robots.txt allows a crawler but your hosting blocks it, the hosting setting wins.
Priya, a small business owner, read a post in a small business group saying every website should block AI bots immediately because AI companies take content for free. A reply said blocking AI bots would make you invisible in ChatGPT. A third person said the two were the same bots anyway. Nobody linked a source, and Priya ended the thread more confused than before. Later Priya checked their own robots.txt. A plugin had generated it, and it blocked GPTBot and nothing else. Nobody remembered choosing that rule. Priya decided to keep the GPTBot block, explicitly allow the search crawlers, and write down why.
AI crawlers are new, and companies add, rename and redefine them. The names and purposes above reflect providers' documentation at the time of writing. A typo in a crawler name means the rule does nothing, so copy names exactly from the current documentation. To go through your own file:
Open your domain followed by /robots.txt. Look for lines naming AI crawlers and for any blanket rules, such as "User-agent: *" with "Disallow: /". Check each provider's current documentation for the exact user-agent names and what each crawler controls: search, training or user-requested fetching. Decide crawler by crawler. Allow search crawlers if you want to be cited, and make a separate decision on training crawlers, including Google-Extended for Gemini training and grounding. Do not use a blanket all-crawler block to handle AI. Check your hosting, security and content delivery settings for AI-crawler blocks, because they override robots.txt. Write down the reason for each decision. After editing, confirm Googlebot can still crawl the pages you want found. Google Search Console has a robots.txt report that shows the file Google last fetched and any problems it found.
Now open your own robots.txt, list every rule that affects an AI crawler, and record your decision for each one next to the reason you made it.
Open your robots.txt and list every rule that affects AI crawlers. For each crawler, write whether you will allow or block it and why.
Junxiong-WFG Organisation is an authorised representative of AIA Financial Advisers Private Limited (Reg. No. 201715016G).