← All articles

Is Your Website Blocking ChatGPT? How to Check robots.txt, Your CDN and Bot Protection

By Rei · CiteMonthly · Published 12 October 2026

Short answer: Yes, a site can block ChatGPT by accident. OpenAI says that to be eligible for ChatGPT search answers you must "allow OAI-Searchbot to crawl the site and confirm that the website host or content delivery network allows traffic from OpenAI's published searchbot IP addresses" [1]. The same pattern applies to Perplexity (allow PerplexityBot and its published IPs) [3] and to Google (Googlebot obeys robots.txt for automatic crawling) [4] [5]. Two separate problems show up in practice: a Disallow rule in robots.txt, and a host or CDN "bot protection" product that challenges or blocks the crawler even when robots.txt allows it. This article is a checklist for both, based on pages we opened on 12 Oct 2026.

Which bots matter for ChatGPT, Perplexity and Google?

They are not the same crawler, and the settings are independent.

OpenAI documents four user agents that matter for most sites [2]:

  • OAI-SearchBot — used "to surface websites in search results in ChatGPT's search features." Sites opted out of it "will not be shown in ChatGPT search answers, though can still appear as navigational links." OpenAI recommends allowing it in robots.txt and allowing its published IP ranges.
  • GPTBot — used to crawl content that "may be used in training" OpenAI's generative AI foundation models. Disallowing it is how you opt out of that training use. It is independent of OAI-SearchBot: you can allow search and block training.
  • ChatGPT-User — used when a person asks ChatGPT a question and ChatGPT visits a page. It "is not used for crawling the web in an automatic fashion," robots.txt "may not apply," and it "is not used to determine whether content may appear in Search."
  • OAI-AdsBot — only visits pages submitted as ads on ChatGPT; not used for foundation-model training.

OpenAI also notes that robots.txt changes for search can take "~24 hours" to take effect [2].

Perplexity documents two agents [3]:

  • PerplexityBot — "designed to surface and link websites in search results on Perplexity," and "not used to crawl content for AI foundation models." Allow it in robots.txt and permit its published IPs; changes may take up to 24 hours.
  • Perplexity-User — used when a person asks a question and Perplexity fetches a page. "Since a user requested the fetch, this fetcher generally ignores robots.txt rules."

Google lists Googlebot as the crawler whose robots.txt preferences affect Google Search (including Discover and related Search features) [5]. It "always obey[s] robots.txt rules when crawling automatically" [5]. Google-Extended is a separate token for Gemini training and grounding; Google says it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal" [5].

Our inference (not any vendor's wording): for customer questions in ChatGPT or Perplexity, keep OAI-SearchBot and PerplexityBot open. GPTBot and Google-Extended are training/grounding controls, not the ChatGPT-search or Google Search inclusion switches.

How do I check my robots.txt file?

A robots.txt file tells crawlers which URLs they may access. Google says it lives at the site root — for www.example.com, at www.example.com/robots.txt — and that unless you specify otherwise, "all files are implicitly allowed for crawling" [4] [6].

Step 1. Open https://yourdomain.com/robots.txt in a private browser window (Google's suggested check that the file is readable) [6].

Step 2. Look for User-agent: groups with Disallow: or Allow: lines. Google's simple example [6]:

User-agent: Googlebot
Disallow: /nogooglebot/

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

Step 3. Watch for these common blocks:

  • User-agent: * then Disallow: / — Google's docs show this as "Block all crawlers except AdsBot" [6]. Honouring crawlers will stay out of the whole site.
  • A group for OAI-SearchBot, PerplexityBot or Googlebot with Disallow: / — that opts that crawler out.
  • A CDN-managed block listing GPTBot or other training crawlers. That may be intentional for training; check it does not also disallow the search crawlers you want.

Google warns robots.txt is not a security lock: the instructions "cannot enforce crawler behavior," and a disallowed URL can still appear in Google results without a snippet if other sites link to it [4]. For ChatGPT search eligibility, though, you must allow OAI-SearchBot [1] [2]. On Wix or Blogger, Google notes you may edit crawl settings in the CMS instead of the file [4] [6].

Can bot protection block crawlers even when robots.txt allows them?

Yes. OpenAI's eligibility rule adds the host or CDN: confirm it "allows traffic from OpenAI's published searchbot IP addresses" [1]. Perplexity's docs include a "WAF Configuration" section with Cloudflare and AWS examples that combine User-Agent matching with IP allowlists [3].

Cloudflare products that can challenge or block bots even when robots.txt is open:

  • Bot Fight Mode matches known bot patterns and "issues computationally expensive challenges" [7]. False positives can challenge "legitimate human or automated traffic" [8]. You "cannot bypass or skip Bot Fight Mode using WAF custom rules or Page Rules"; for exceptions Cloudflare points to Super Bot Fight Mode, which supports Skip rules [7] [8].
  • AI bot policies separate Search, Agent and Training. Search covers crawlers that "collect or index your content to answer questions about it later"; Training covers crawlers used to "train or fine-tune a model" [9]. You can Allow Search while Disallowing Training — the same split as OAI-SearchBot vs GPTBot.
  • Managed robots.txt (Bot Preference Sync) can prepend directives to your file (or create one). Cloudflare's example prepends Disallow: / for GPTBot, ClaudeBot, Bytespider and Google-Extended, with a content-signal of search=yes, ai-train=no [10]. It also states that "robots.txt compliance is voluntary" and that edge enforcement is a separate product [10].

Our inference (not Cloudflare's wording): a one-click "block AI bots" or Bot Fight Mode toggle can challenge ChatGPT or Perplexity at the edge even when robots.txt looks fine. That is why OpenAI and Perplexity ask you to check IPs and WAF rules, not only the text file.

What should I ask my host or developer to check?

  1. robots.txt — No Disallow: / for OAI-SearchBot, PerplexityBot, Googlebot, or under User-agent: * unless that blanket block is intentional [2] [3] [6].
  2. Published IP ranges — Allow OpenAI's OAI-SearchBot ranges (openai.com/searchbot.json) and PerplexityBot ranges (perplexity.com/perplexitybot.json) [2] [3]. Google publishes common-crawler ranges separately [5].
  3. Bot Fight Mode / similar — If search crawlers are challenged, Cloudflare's guidance is to disable Bot Fight Mode, use IP Access rules that match first, or move to Super Bot Fight Mode so you can Skip approved traffic [7] [8].
  4. AI bot policies — Keep Search on Allow for ChatGPT/Perplexity-style crawlers; use Training Disallow only to opt out of training [9] [10].
  5. WAF allow rule — Perplexity's Cloudflare pattern: User-Agent contains PerplexityBot or Perplexity-User and IP in published ranges, action Allow [3]. OpenAI implies the same IP-plus-UA approach by publishing searchbot IPs [1] [2].
  6. Wait ~24 hours after a robots.txt change before retesting [2] [3].

None of this guarantees a mention. OpenAI says ChatGPT ranks with "multiple factors" and "Placement is not guaranteed" [1]. The goal is not to block the door.

How do I test whether the fix worked?

  1. Re-open /robots.txt and confirm the allow rules.
  2. Ask your host for a log or Security Analytics event showing OAI-SearchBot / PerplexityBot / Googlebot returning HTTP 200 (Cloudflare labels Bot Fight Mode challenges in Analytics [7]).
  3. After about a day, ask ChatGPT (with Search) a question your site should answer. Citations can still be incomplete or wrong even when crawling works [1].

Frequently asked questions

If I block GPTBot, will I disappear from ChatGPT search?

Not according to OpenAI. GPTBot (training) and OAI-SearchBot (search) are controlled separately. Search appearance is managed with OAI-SearchBot [2].

Does a blank or missing robots.txt block ChatGPT?

No. Google's create guide says that unless you specify otherwise, files are "implicitly allowed for crawling" [6]. The risk is an explicit Disallow, or a CDN that injects one [10].

Is robots.txt enough on its own?

No. OpenAI requires the host/CDN to allow its searchbot IPs as well [1]. Perplexity documents WAF allowlisting for the same reason [3]. Cloudflare notes that robots.txt compliance is voluntary and that edge blocking is a separate control [10].

Will Bot Fight Mode always block Googlebot or OAI-SearchBot?

Cloudflare does not publish a per-crawler block list for Bot Fight Mode. It does say the product challenges known bot patterns, false positives happen, and you cannot Skip it with WAF custom rules [7] [8]. Super Bot Fight Mode adds a Verified bots category that includes Googlebot and Bingbot [11]. Treat a challenge to a search crawler as something to check in Analytics, not a permanent ban.

Does Google-Extended control Google Search?

No. Google says it manages Gemini training and grounding, and "does not impact a site's inclusion in Google Search" [5].

Want to know what ChatGPT says about you?

CiteMonthly runs a free "Does ChatGPT mention you?" check. We ask ChatGPT and Perplexity the questions your customers ask, then email you their answers: who gets named, which sources they cite, and whether you're one of them. No card, no account. Email citemonthly@gmail.com to request your free check.

Sources

  1. OpenAI Help Center, "Searching the web with ChatGPT". https://help.openai.com/en/articles/9237897-chatgpt-search (opened 12 Oct 2026)
  2. OpenAI Platform docs, "Overview of OpenAI Crawlers". https://platform.openai.com/docs/bots (opened 12 Oct 2026)
  3. Perplexity docs, "Perplexity Crawlers". https://docs.perplexity.ai/guides/bots (opened 12 Oct 2026)
  4. Google Search Central, "Introduction to robots.txt". https://developers.google.com/search/docs/crawling-indexing/robots/intro (opened 12 Oct 2026)
  5. Google Search Central, "Google's common crawlers". https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers (opened 12 Oct 2026)
  6. Google Search Central, "Create and submit a robots.txt file". https://developers.google.com/search/docs/crawling-indexing/robots/create-robots-txt (opened 12 Oct 2026)
  7. Cloudflare Docs, "Get started with Bot Fight Mode". https://developers.cloudflare.com/bots/get-started/bot-fight-mode/ (opened 12 Oct 2026)
  8. Cloudflare Docs, "Handle False Positives from Bot Fight Mode or Super Bot Fight Mode". https://developers.cloudflare.com/bots/troubleshooting/false-positives/ (opened 12 Oct 2026)
  9. Cloudflare Docs, "Block AI Bots". https://developers.cloudflare.com/bots/additional-configurations/block-ai-bots/ (opened 12 Oct 2026)
  10. Cloudflare Docs, "robots.txt setting" (managed robots.txt / Bot Preference Sync). https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/ (opened 12 Oct 2026)
  11. Cloudflare Docs, "Custom rules" (bot settings table: Verified bots includes Googlebot, Bingbot). https://developers.cloudflare.com/bots/additional-configurations/custom-rules/ (opened 12 Oct 2026)