Which AI crawlers to allow in robots.txt: search, training and fetchers

Khalid Mohamed
Founder, CapitalSEO · Last reviewed · 8 min read

Published .

The short answer

Allow the search crawlers: Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot, PerplexityBot and Applebot. They decide whether AI answers can find and cite you. Training crawlers like GPTBot, ClaudeBot and Google-Extended are a separate choice, and blocking them doesn't remove you from search. A third group fetches pages when a person asks a question, and some of those ignore robots.txt.

Most "block the AI bots" advice treats every AI crawler as the same thing, but most AI companies now run several crawlers, and each one does a different job. Block the wrong one and you drop out of ChatGPT or Claude answers without meaning to. Block the right one and nothing changes in search at all.

This guide sorts them by job, shows which robots.txt line controls what, and gives you two files you can copy. Every claim comes from the AI company's own documentation, checked on October 18, 2026.

Who this is for: business owners and marketers who want their site cited in AI answers, and want to decide on purpose what AI models can train on.
Who it isn't for: sites that need to keep content private. robots.txt is a request, not a lock.

Three jobs, three kinds of crawler

Every AI crawler does one of three jobs:

  1. Search. It builds the index that an AI answer searches when someone asks a question. If it can't reach your pages, that AI can't cite you. This is the one that matters for visibility.
  2. Training. It collects pages to train future AI models. Blocking it keeps your content out of training data. Mostly, it doesn't change whether you're cited today. Google-Extended is the exception, covered below.
  3. Fetching for a person. When someone asks ChatGPT or Perplexity to read a page, the AI visits it on their behalf. Some companies treat these visits as human requests, so robots.txt may not apply. Anthropic says its fetcher, Claude-User, does follow it.

Here is who runs what.

CompanySearch: lets AI answers cite youTraining controlFetches for a person
GoogleGooglebot
also AI Overviews and AI Mode		Google-Extended
Gemini training and grounding; no effect on Search		—
MicrosoftBingbot
no separate AI token published		NOCACHE / NOARCHIVE meta tags
no robots.txt token		—
OpenAIOAI-SearchBot
ChatGPT search		GPTBotChatGPT-User
robots.txt “may not apply”
AnthropicClaude-SearchBotClaudeBotClaude-User
honours robots.txt
PerplexityPerplexityBotNone
PerplexityBot isn't used for training		Perplexity-User
“generally ignores robots.txt”
AppleApplebot
Apple search features		Applebot-Extended
doesn't crawl; controls training use		—
MetaMeta-WebIndexer
Meta AI search		Meta-ExternalAgent
training and other product uses		Meta-ExternalFetcher
“may bypass robots.txt”
Allow the first column if you want to be cited. The second column is your choice. The third may not read robots.txt at all. Sources: each company's own crawler documentation, checked October 18, 2026.

A few things in that table are easy to miss:

  • OpenAI says its settings are independent. "Each setting is independent of the others." Blocking GPTBot keeps you out of training; it does not keep you out of ChatGPT search. Blocking OAI-SearchBot does: opted-out sites "will not be shown in ChatGPT search answers," though OpenAI says they can still appear as navigational links. Confirmed–Industry
  • Anthropic follows the same split. Blocking ClaudeBot signals that your future content "should be excluded from our AI model training datasets." Blocking Claude-SearchBot "may reduce your site's visibility and accuracy in user search results." Anthropic also honours the Crawl-delay line, if you need to slow it down. Confirmed–Industry
  • Perplexity says PerplexityBot isn't used to train models. It's for showing and linking sites in Perplexity's answers. Perplexity-User, its fetcher, "generally ignores robots.txt rules." Confirmed–Industry
  • Apple's training control doesn't crawl at all. "Applebot-Extended does not crawl webpages." It only tells Apple whether pages Applebot already collected can be used for training. Blocking it doesn't affect Apple's search results. Confirmed–Industry
  • Meta runs five crawlers. Meta-WebIndexer "navigates the web to improve Meta AI search result quality for users," so it's the one to allow if you want Meta AI to find you. Meta-ExternalAgent covers training AI models and other product uses, so blocking it is a broader call. Meta-ExternalFetcher "may bypass robots.txt rules." Confirmed–Industry

Google is the exception

Google doesn't have a separate AI search crawler. AI Overviews and AI Mode use the same index as regular search, built by Googlebot. Confirmed–Google So there is no robots.txt line that keeps you in Google Search but out of AI Overviews.

Google-Extended is not that line. It controls whether your content is used to train Gemini models and to ground them, and Google is explicit: "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search." Confirmed–Google

The word to notice is grounding: Google says Google-Extended also covers "providing content from the Google Search index to the model at prompt time." Confirmed–Google So blocking it may keep your pages out of Gemini answers outside Google Search, even though your Search listing and AI Overviews stay the same. Our view

If you want to limit what Google's AI features show from a page, Google points to the same controls as regular snippets: nosnippet, data-nosnippet, max-snippet and noindex. Confirmed–Google Those limit your regular search listing too. There's no AI-only switch.

When this fails: when someone blocks Googlebot to "stop Google's AI." That removes the site from Google Search entirely.

Microsoft: no separate Copilot crawler

Microsoft hasn't published a separate robots.txt token for its AI answers. Our view In 2023, Bing gave site owners two meta tags aimed at Bingbot instead: with NOCACHE, Bing Chat can show only your URL, title and snippet; with NOARCHIVE, your content isn't used in Bing Chat answers or linked from them. The tags also limit training: with NOCACHE, only the URL, title and snippet may be used to train Microsoft's models; with NOARCHIVE, none of the content is. Pages with either tag still appear in regular Bing results. Confirmed–Industry

Bing Chat has since become Copilot, and that post hasn't been updated. Treat these tags as the last thing Microsoft confirmed, not a current guarantee. Our view

What we recommend for a local business

Search crawlers: allow all of them. If you want customers to find you through AI answers, every search crawler in the table needs access. There's no upside to blocking OAI-SearchBot, Claude-SearchBot, PerplexityBot, Meta-WebIndexer or Applebot for a business that wants to be found. Our view

Training crawlers: a business decision, not an SEO one. Blocking them doesn't help or hurt your search visibility. The question is whether you want your content in future AI models.

  • If your content is your product (paid courses, a publication, research you sell), blocking training crawlers protects it. Our view
  • If you're a plumber, a clinic or a consultant, your pages exist to bring in customers. Having AI models know what you do and where you work is more likely to help than hurt. We'd leave training crawlers allowed. Our view

Fetchers: don't count on robots.txt. Several companies say their user-triggered fetchers may ignore it. If a page must stay private, put it behind a login.

On capitalseo.ca, we allow them. This site exists to teach, and nothing on it is paid or private, so if an AI learns from our guides and passes the advice on, that's the job working. That choice doesn't affect whether we're cited; the search crawlers decide that. Our view

We're not alone. On October 20, 2026, we read the robots.txt files of six SEO agencies that show up when you search for SEO in Ottawa. All six let training crawlers in, and some label those lines "for citations" or "GEO visibility," which mixes up the two jobs. Their sites still work, because a general "allow everyone" rule lets the search crawlers in too. Tested by us

Two robots.txt files you can copy

File A: allow everything. This is what capitalseo.ca uses.

User-agent: *
Allow: /

Sitemap: https://yourdomain.ca/sitemap.xml
File A. Every crawler, search and training, may read every page.

File B: allow search, block training. Each training crawler gets its own group. A crawler follows the group that names it, so these rules don't touch the search crawlers, which fall back to the User-agent: * group. The order of the groups doesn't matter.

# Training crawlers: blocked
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: meta-externalagent
Disallow: /

# Everyone else, including every search crawler: allowed
User-agent: *
Allow: /

Sitemap: https://yourdomain.ca/sitemap.xml
File B. Search crawlers keep full access; the five named training controls are refused. Meta-ExternalAgent also covers some other Meta product uses.

Replace the sitemap line with your own. Add a group for any other training crawler you want to block, using the name from its company's documentation.

When this fails: when you copy a long "block all AI bots" list from a forum. Many of those lists include OAI-SearchBot, Claude-SearchBot and PerplexityBot, which takes you out of those answers. Read each line before you paste it.

Check what's actually blocking AI crawlers

robots.txt isn't the only thing that can block a crawler. Your hosting or security service can block it before it ever reads the file.

  1. Open your robots.txt. Go to yourdomain.ca/robots.txt and compare each User-agent line with the table above.
  2. Check your CDN or firewall. Since July 2025, Cloudflare asks every new domain whether to allow AI crawlers, and every new domain "starts with the default of control." Confirmed–Industry If your site uses Cloudflare, check its AI crawler settings, not just robots.txt. On capitalseo.ca, hosted on Vercel, we checked on October 18, 2026: no firewall rules are set, so nothing blocks crawlers before robots.txt. Tested by us
  3. Look for a Crawl-delay line. It asks crawlers to wait between requests. Google ignores it, but Anthropic says its crawlers follow it, and Bing has supported it for years. Confirmed–Google Confirmed–Industry A delay of 10 seconds, a common leftover on WordPress sites, slows down the crawlers behind Claude and Copilot answers. One of the six agencies we checked has exactly that. Tested by us
  4. Check Googlebot in Search Console. Use the URL Inspection tool to confirm Googlebot can reach a page. Google suggests the same check when a page's snippet controls don't seem to work. Confirmed–Google
  5. Give it time. Perplexity and Meta say robots.txt changes can take up to 24 hours to apply. Google says recrawling can take days to months. Confirmed–Industry Confirmed–Google

When this fails: when you change robots.txt and check AI answers the next morning. Citations follow crawling, and crawling takes time.

Myths to skip

  • "Block Google-Extended to stay out of AI Overviews." It doesn't affect Google Search. AI Overviews come from Googlebot's index. Confirmed–Google
  • "Blocking GPTBot hides you from ChatGPT." ChatGPT search uses OAI-SearchBot. The two settings are independent. Confirmed–Industry
  • "robots.txt keeps AI away from your content." Several fetchers that act for a person say they may ignore it. Confirmed–Industry
  • "Add an llms.txt file so AI can read your site." Google Search doesn't use it. See AI search and GEO. Confirmed–Google

Next step

Open your robots.txt today and look for the search crawlers in the first column of the table. If any of them is blocked, decide whether you meant to. Then check your CDN or firewall settings.

To see whether AI answers already cite you, use the monthly AI check in the analytics guide. If you'd like someone to check this with you, it's part of the free visibility audit.

About this page

Who: Written by Khalid Mohamed, founder of CapitalSEO, an Ottawa SEO consultant.
How: Written with AI assistance. Every crawler name and quote was checked against the company's own documentation on October 18, 2026, and labelled. Crawler names change; we'll update this page when a company changes its documentation. Opinions are labelled "Our view."
Why: Most AI crawler lists mix search, training and fetching into one "block or allow" decision. Getting that wrong can cost you AI citations.
Last substantive update: October 20, 2026: first published.

Sources

↑