Marketing paid for advertising on ChatGPT. Security’s WAF was blocking ChatGPT’s crawlers. Both teams were doing exactly what they should be doing — and nobody realized the two decisions were in direct conflict.
This wasn’t a mistake. It was an inevitable collision. Marketing’s job is to chase visibility on every emerging channel. Security’s job is to block unauthorized data collection. When the same vendor is on both sides of that equation, the collision is structural.
If you’re a security leader at a large enterprise, this is coming for you. Here’s how to handle it without being the person who just says no.
The Paradox Every Enterprise Will Hit
ChatGPT launched advertising in February 2026. Brands are already running sponsored placements inside chat responses. OpenAI is projecting up to $25 billion in ad-related revenue by 2029.
Your marketing team sees a new channel to reach 800 million weekly active users. Your security team sees exposure — every page an AI crawler touches becomes potential training data. Proprietary content, pricing strategies, technical documentation, all of it.
Cloudflare’s data puts a number on the imbalance: OpenAI’s crawl-to-referral ratio is roughly 1,700 to 1. For every 1,700 pages they crawl, they send about one visit back. They consume your content at scale and return almost nothing.
Block their crawlers entirely and your brand disappears from AI-powered search. Your marketing team just paid for ads on a platform your firewall won’t let it index.
What Goes Wrong If You Just Unblock Everything
The pressure will be to “just open it up” so the ads work. Here’s what that looks like in practice.
Your entire web estate — product specs, pricing pages, technical documentation, support articles, competitive positioning — becomes training data for a model that serves 800 million users. Your proprietary content shows up paraphrased in ChatGPT answers, attributed to no one. A competitor asks ChatGPT about your pricing strategy and gets a surprisingly detailed answer sourced from pages you never intended to be public in that context.
You didn’t get breached. You just left the front door open and labeled it “please crawl.”
That’s the risk your business stakeholders need to understand before anyone touches a firewall rule.
What You’re Actually Dealing With
The crawler landscape shifted hard in late 2025. Most enterprises haven’t caught up.
OpenAI now operates three separate crawlers, and most security teams are treating them as one:
GPTBot — Collects data to train foundation models. Traffic grew 305% year-over-year. This is the one to be most cautious about.
OAI-SearchBot — Powers search results and shopping features in ChatGPT. Block this and your content won’t surface when users search.
ChatGPT-User — Handles user-initiated browsing, Custom GPTs, and GPT Actions. In December 2025, OpenAI quietly removed robots.txt compliance language for this crawler. It no longer promises to respect your no-crawl directives.
Three crawlers. Three different purposes. Three different compliance behaviors. One binary WAF rule doesn’t cut it.
And some AI companies aren’t even pretending. In August 2025, Cloudflare published a forensic report showing Perplexity AI deploying stealth crawlers — spoofed browser user agents, rotating IP addresses across different networks, ignoring robots.txt entirely. Millions of requests per day across tens of thousands of domains. Cloudflare delisted Perplexity as a verified bot. Perplexity called it a “publicity stunt.”
Think of robots.txt like a “No Soliciting” sign on your front door. You still need that door to open — for family, for friends, for the people you actually invited. Some salesmen see the sign and respect it. Others knock anyway. And a few put on a disguise and pretend to be your neighbor.
That’s what’s happening on the web right now. Your website has to be open to customers, partners, and legitimate search engines. AI crawlers know that — and not all of them care about the sign. Over 5.6 million websites now block GPTBot — up 70% in just a few months. The signs are going up everywhere. The knocking hasn’t stopped.
A Framework for Handling This
When security gets asked to unblock the WAF for ChatGPT's crawlers, use a phased approach.
1. Solve the Legal Question First
Before touching a single firewall rule, confirm your Master Service Agreement with OpenAI covers the advertising relationship. What does it say about data usage? Does the advertising agreement address how crawled content is handled?
If your legal team hasn’t reviewed the MSA in the context of crawler access, stop here. Don’t let marketing’s urgency bypass legal due diligence.
2. Advertise Without Full Crawler Access (The Move Nobody’s Talking About)
This is the key insight: you don’t have to unblock everything to run ads.
Serve static, curated content on the specific URLs tied to the advertising campaign. Marketing gets their brand presence. The crawlers see only what you deliberately expose — not your full web estate.
OpenAI’s own guidance suggests that restricting crawler access can impact ad ranking and performance. They want full access. Of course they do. But “it might impact rankings” is not “it won’t work.” This buys time to build a proper strategy while the ads are live.
Most enterprises are treating this as a binary — block everything or allow everything. The static content approach is the middle path that lets you say yes to marketing without handing over the keys.
3. Build Granular Crawler Policies
Stop making one decision for three crawlers:
GPTBot (training): Block unless your organization has deliberately decided to contribute training data. For most enterprises, there’s no upside.
OAI-SearchBot (search visibility): Consider allowing. If marketing is investing in ChatGPT ads, blocking this undermines their spend.
ChatGPT-User (user browsing): Requires infrastructure-level controls — WAF rules, bot management, rate limiting. A robots.txt entry isn’t enough since this crawler no longer promises to respect it.
4. Classify Your Content by Risk
Not all pages carry the same exposure. Audit your web properties:
Low risk, high visibility value: Marketing pages, blog content, product overviews. Candidates for crawler access.
High risk, low visibility value: Internal docs, technical specs, pricing, competitive intelligence. Stay locked down.
Gray zone: Product details, support docs, research content. Case-by-case with business stakeholders.
Frame this for leadership as a data governance decision, not a security decision. “What intellectual property are we willing to expose for brand visibility?” That’s a question security informs — not one we answer alone.
5. Enforce at the Infrastructure Level
Don’t trust voluntary compliance. Invest in enforcement.
Here's something most security teams don't realize: if your site runs behind AWS WAF and you've enabled Bot Control with the AI category rule, AI crawlers are being blocked by default. AWS WAF's managed Bot Control rule group includes a CategoryAI rule that blocks all AI bots — and unlike every other bot category, it blocks both verified and unverified AI bots. Search engine bots, monitoring bots, SEO bots — those all get a pass if they're verified. AI bots don't. When that rule is enabled, AWS treats them all as hostile.
That’s likely how many enterprises ended up in the paradox in the first place. Security enabled Bot Control as a best practice. Marketing bought ChatGPT ads. Neither team connected the dots.
This is also why “just unblock AI bots” is the wrong response. Instead, use scope-down statements to selectively allow specific AI crawlers while keeping the default block in place for everything else. For example, you could create a scope-down statement that excludes OAI-SearchBot from the CategoryAI block — allowing ChatGPT search indexing for your ad campaign — while GPTBot and every other AI crawler stays blocked. That gives you the granularity from Step 3 at the infrastructure level.
Beyond AWS, similar enforcement options exist:
Cloudflare’s Robotcop enforces robots.txt at the network edge rather than relying on crawlers to obey
Bot management platforms that fingerprint crawlers beyond user-agent strings — critical for catching stealth crawlers like the ones Perplexity deployed
Active monitoring — stealth crawlers mean set-and-forget doesn’t work. Review your bot traffic logs regularly.
The One Thing to Do Monday Morning
The old web bargain — crawlers index your content, send you traffic, you monetize that traffic — is broken. AI platforms now crawl your content, train on it, and sell ads against it. Security leaders need to be at the table for this conversation, not waiting for a ticket from marketing.
Your job isn’t to block every bot or allow every bot. It’s to make sure your organization has a deliberate strategy — not a default one.
Start with one question: Does anyone at your company know which AI crawlers are currently hitting your web properties, and has anyone made an intentional decision about each one?
If the answer is no, you’ve found your next project.
Further Reading
Overview of OpenAI Crawlers — OpenAI’s official documentation on GPTBot, OAI-SearchBot, and ChatGPT-User
Perplexity Is Using Stealth, Undeclared Crawlers to Evade Website No-Crawl Directives — Cloudflare’s forensic report on Perplexity’s crawling behavior
From Googlebot to GPTBot: Who’s Crawling Your Site in 2025 — Cloudflare’s data on AI crawler traffic growth and crawl-to-refer ratios
AWS WAF Bot Control Rule Group — AWS documentation including the CategoryAI default block rule
Our Approach to Advertising and Expanding Access to ChatGPT — OpenAI’s advertising announcement
I’m a Director of Security Engineering at a Fortune 500 company, navigating these decisions in real time. I write about securing AI adoption at enterprise scale — real lessons, not vendor marketing.
How is your organization navigating the AI crawler decision? Hit reply. I want to hear it.