flareOps
Guide

Allow AI crawlers: robots.txt for OAI-SearchBot, PerplexityBot and Claude

Last checked: · Updated · By flareOps (how we write)

In short: Allow the search bots: OAI-SearchBot (ChatGPT), PerplexityBot, Claude-SearchBot, Googlebot and Bingbot. Blocking them hides your store from those assistants' answers. Training bots like GPTBot, ClaudeBot and Google-Extended are separate: you can block them without leaving search. Then check your security plugin and CDN, which can block bots even when robots.txt allows them.

AI assistants can only recommend pages they can read. The first gate is your robots.txt file, and the second is your security setup. This guide lists the bots that matter for an online store, what each one does, and how to check that nothing blocks them by accident. It works for WooCommerce, Wix and custom sites. Bot names and rules were checked on each company's own page on 4 Nov 2026.

Search bots, user bots and training bots

There are three kinds of AI bots. They need different decisions.

  • Search bots build the index an assistant searches. Block one and you leave that assistant's answers.
  • User bots open a page live because a person asked. Most do not follow robots.txt reliably.
  • Training bots collect pages to train models. Blocking them does not remove you from search.

The bot table

Bot (user-agent token)CompanyKindWhat the company says
OAI-SearchBotOpenAISearchOpted-out sites "will not be shown in ChatGPT search answers"
ChatGPT-UserOpenAIUserrobots.txt rules "may not apply" to user actions
GPTBotOpenAITrainingDisallowing it means content should not be used to train models
PerplexityBotPerplexitySearchSurfaces and links sites in results; "not used to crawl content for AI foundation models"
Perplexity-UserPerplexityUser"Generally ignores robots.txt rules"
Claude-SearchBotAnthropicSearchBlocking it "prevents our system from indexing your content for search"
Claude-UserAnthropicUserBlocking it prevents Claude fetching your pages for a user's question
ClaudeBotAnthropicTrainingBlocking it excludes future content from training
GooglebotGoogleSearchControls Google Search, including AI Overviews and AI Mode
Google-ExtendedGoogleTraining / Gemini groundingGoogle says it does not affect Search inclusion or ranking
BingbotMicrosoftSearchFeeds Bing and Copilot; ChatGPT search also leans on Bing results

OpenAI says a robots.txt change takes about 24 hours to reach its search systems.

A safe robots.txt for a store

This allows the search bots and leaves the training choice to you:

# Search and live-fetch bots: allow
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: Googlebot
User-agent: Bingbot
Allow: /
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/

# Training bots: your choice. Delete this group to allow them.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

# Everyone else
User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/

Sitemap: https://example.com/sitemap.xml

Two rules from the robots.txt standard (RFC 9309) catch people out:

  1. A bot uses only the most specific group that names it. If you have a User-agent: OAI-SearchBot group, the * group no longer applies to it. Repeat any disallows you need inside each group.
  2. The longest matching rule wins. Allow: /products/ beats Disallow: / for product URLs.

Change the cart and account paths to your platform's. On WordPress, /wp-admin/ is usually blocked already.

Platform notes

  • WooCommerce / WordPress: WordPress makes a virtual robots.txt. SEO plugins (Yoast, Rank Math) let you edit it. Check Settings → Reading → Search engine visibility is not ticked ("Discourage search engines"). Security plugins and host firewalls (Wordfence, host-level bot protection, ModSecurity rules) can block or rate-limit bots regardless of robots.txt. WooCommerce's "Coming soon" mode also hides products from bots.
  • Wix: Wix serves robots.txt for you and lets you edit it in the SEO settings. Open it and check that no one added blocks for the bots above.
  • Custom sites: serve a real /robots.txt file. If you use a framework, make sure a staging-only "disallow all" rule did not ship to production.

The CDN trap

Your robots.txt can say "welcome" while your CDN says "go away". Cloudflare started blocking AI crawlers by default for new customers on 1 July 2025, and any site can switch blocking on with one click. Other CDNs and hosts have similar bot settings.

If you use Cloudflare, check AI Crawl Control (and Security → Bots) in the dashboard. Allow the search bots you want. Also check whether Cloudflare manages your robots.txt: managed files add Content-Signal: lines.

How to check it works

  1. Open yoursite.com/robots.txt and read it with the table above.
  2. Run one product page through the free AI readiness checker. It reads robots.txt for each bot and fetches the page as a plain client, so a challenge page or 403 shows up.
  3. In Search Console, use URL Inspection on a product page to see what Googlebot gets.
  4. Look at your server or plugin logs after a few days for OAI-SearchBot, PerplexityBot and ChatGPT-User visits. A ChatGPT-User visit means a real person's chat opened your page.

How flareOps helps

flareOps' store check parses your robots.txt the way the standard says (most specific group, longest rule wins) for 25 AI and search bots, and separates search, user and training bots in plain English. It also flags a robots.txt that answers with a server error (Google treats that as "disallow all"), a long Crawl-delay, a missing Sitemap line, Cloudflare's managed Content-Signal lines, and homepages or product pages that return a challenge or 403 to an honest client. With the WooCommerce plugin or the @flareops.ai/next package, it also logs real AI bot visits. The store check is free; Pro is $29 per month per store.

Questions people ask

If I block GPTBot, will my store disappear from ChatGPT?

No. OpenAI says GPTBot is for training. ChatGPT search uses OAI-SearchBot. You can block GPTBot and still allow OAI-SearchBot.

What happens if I block OAI-SearchBot?

OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. It takes about 24 hours for a robots.txt change to reach OpenAI's search systems.

Does blocking Google-Extended remove me from Google AI Overviews?

Google says Google-Extended does not affect a site's inclusion in Google Search and is not a ranking signal. It controls Gemini training and grounding outside Search. AI Overviews and AI Mode are controlled by Googlebot.

Do ChatGPT-User and Perplexity-User follow robots.txt?

Not reliably. OpenAI says robots.txt rules may not apply to ChatGPT-User because a user started the action. Perplexity says Perplexity-User generally ignores robots.txt for the same reason.

My robots.txt allows everything. Can bots still be blocked?

Yes. Security plugins, hosting firewalls and CDNs such as Cloudflare can block or challenge bots before robots.txt matters. Cloudflare has blocked AI crawlers by default for new customers since 1 July 2025.

Should I add a Crawl-delay?

Usually not. Google ignores Crawl-delay; Bing honours it. A long delay slows how fast Bing, and the AI answers built on Bing, see your new prices and stock.

How do I test whether a bot can reach my product page?

Use the free AI readiness checker, or Google's URL Inspection tool in Search Console for Googlebot. Your server logs show which bots actually visited.

Sources

  1. OpenAI: Overview of OpenAI crawlers
  2. Perplexity: Perplexity crawlers
  3. Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
  4. Google Search Central: Google's common crawlers (Google-Extended)
  5. Search Engine Land: Cloudflare to block AI crawlers by default (Jul 2025)
  6. RFC 9309: Robots Exclusion Protocol

Sources checked on 4 November 2026.