Controlled Openings: How Enterprises Should Actually Let AI Crawlers In

A three-tier playbook for Enterprise AI, app owners, and security teams — closed by default, opened deliberately, governed on a cadence.

Apr 11, 2026

The Controlled Openings Model

Tier 1 — The Declaration. Owned by Enterprise AI, Marketing, and Security. Legal reviews. The robots.txt file is the meeting minutes — every AI crawler named, every posture stated, closed by default.

Tier 2 — Active Enforcement. Owned by app teams. Security audits. The WAF or CDN baseline that turns the declaration into a control on every property.

Tier 3 — Governance. Owned by the AI governance organization. The allowed list register, named owners, and the quarterly review that keeps every opening on a clock.

Closed by default. Every opening is scoped to one application, owned by a named human, and expires on a date. Everything else is a wish.

A marketing director buys an ad placement inside ChatGPT. The placement requires OpenAI to crawl the landing page as part of ad onboarding — and the crawl fails. OAI-SearchBot can't reach the page. The ad can't go live.

Here's the conflict. Marketing needs the crawler in. Every enterprise with a public web estate has a legitimate reason to block AI crawlers by default — scraping pressure, referral asymmetry, brand misrepresentation risk, contested compliance from bots like PerplexityBot. Both positions are correct. Neither can unilaterally win. And right now, most enterprises don't have a shared place where those two positions can see each other before one of them collides with the other.

This is the state most enterprises are actually in. Not a policy. Not even a wish. Two legitimate positions with no shared place to meet.

The fix is a model I call controlled openings — closed to AI crawlers by default, opened deliberately per application, with a named owner and an expiry date on every opening. Not one enterprise-wide yes or no. A portfolio of small, deliberate yeses — each one owned, scoped, and on a clock.

This post is the playbook for getting there — written for the three groups who have to operate it together: Enterprise AI leadership, app owners, and security teams. No one of them can make the call alone. Enterprise AI leadership owns the vendor relationships every allow touches, and co-authors the declaration with Marketing's referral data and Legal's MSA review. App owners operate the enforcement on the properties they're accountable for. Security owns the enforcement standard that keeps the whole thing consistent across hundreds of properties, and audits against it. The AI governance organization owns the allowed list register and runs the quarterly review that keeps every opening on a clock. The controlled openings model is what lets every owner act in the same direction without stepping on each other — gives Marketing and Legal a named seat at the table when the declaration is written, and most importantly, actually addresses Marketing's needs instead of leaving them to discover the block and the ad won’t go live.

The call has to land somewhere every stakeholder can read — not buried in a WAF rule group that only the app team operating that property can see. That somewhere is robots.txt.

robots.txt is the declaration. It names every AI crawler explicitly and records whether the company allows it, disallows it, or has granted a time-bound allow for a specific application. The declaration is jointly authored by Enterprise AI leadership, business and marketing, and security. Legal reviews every change before it merges. App teams implement the declaration on their own properties. Security reviews allow requests against the standard. Governance keeps all of it aligned over time.

Tier 1: The Declaration

The failure mode most enterprises are in right now isn’t having the wrong AI bot policy. It’s having a different one on every property. Some app teams are genuinely sophisticated — their robots.txt names every major AI crawler and their WAF is tuned. Other teams copied a robots.txt from a template years ago and haven’t touched it since. Marketing landing pages spin up without any declaration at all.

A standardized AI bot robots.txt fixes the consistency problem by giving every app team the same starting point. The classification in the template is made jointly by the three co-authors. Enterprise AI brings the vendor relationships. Marketing brings the referral data and the campaign roadmap. Security brings the threat picture. Legal reviews every draft against current MSA terms before it merges, flagging any Allow that conflicts with data usage clauses. The file is the meeting minutes of that conversation, written in a format bots can parse.

The Decision Matrix

Every known AI crawler is Block by default. An allow is never enterprise-wide — it’s scoped to a specific application. OAI-SearchBot allowed on the corporate marketing site for an ad campaign is not OAI-SearchBot allowed on the developer docs or the support portal. Each app team runs its own allow list against its own property, because the business case for any given crawler almost never applies uniformly across the enterprise. All per-application allows are recorded in a central register so the governance layer can see across them, but the decisions themselves are made where the property is owned.

Training Crawlers — Default: Block

None of these return referral traffic. All of them consume at scale. The crawl-to-referral ratio on training crawlers is effectively infinite — you give content, you get nothing back. The only reason to allow any of them is a deliberate strategic decision by Enterprise AI leadership to contribute training data to a specific partner, and that decision runs through Tier 3 as an allow request, not a default.

Search, Answer, and User-Initiated Crawlers — Default: Block

These crawlers drive real referral traffic and matter to marketing and Enterprise AI. That’s exactly why they go through the allowing process — because the business case must be made and owned by a named stakeholder, not inherited as a default. Block is the posture. Every allow has an owner and an expiry.

Two rows deserve specific notes. ChatGPT-User had its robots.txt compliance language revised by OpenAI in December 2025 — OpenAI’s current documentation states that because these actions are initiated by a user, “robots.txt rules may not apply,” which means enforcement for this crawler has to happen entirely at Tier 2. PerplexityBot has documented stealth behavior per Cloudflare’s August 2025 forensic report, including spoofed user-agents and rotating IPs. A Disallow in Tier 1 is still worth recording for audit purposes, but enforcement rests on the WAF’s signal rules, not the user-agent string.

Anthropic’s crawler roster expanded in early 2026 to include three distinct agents: ClaudeBot (training, in the table above), Claude-User (user-initiated), and Claude-SearchBot (search indexing). Each is independently controllable via robots.txt. Enterprises that blocked only ClaudeBot and assumed full coverage need to revisit their declarations.

The Catch-All

Every crawler not named above is an unknown. Unknowns do not get an implicit pass. When a new bot shows up in an app team’s AI Activity Dashboard, it triggers the Tier 3 intake process. The default state while the ticket is open is Block. The default is always Block for anything undeclared.

Tier 2: Active Enforcement

App owners operate this tier. Security audits it. Declaration without enforcement is a wish.

Security’s job at Tier 2 is to publish an enforcement standard — the set of controls every public-facing property must implement, regardless of which WAF or CDN sits in front of it. The standard is platform-agnostic by design: block AI bot categories by default, enforce every robots.txt Disallow at the edge, apply targeted inspection to sensitive endpoints.

AWS WAF Bot Control is the reference implementation walked through below. The label namespace and CategoryAI rule are well-designed for exactly this problem, and if your property lives in AWS, this is the shortest path from standard to deployed control. If it lives somewhere else — Cloudflare, Akamai, Azure Front Door, F5 — read this section as the pattern, not the product. Map CategoryAI to your platform's AI bot category. Map the signal rules to your platform's stealth detection. Map the label namespace to whatever your platform calls its tagging layer. The names differ. The controls don't.

Three building blocks matter for AI bot enforcement: category rules, signal rules, and the label namespace.

CategoryAI is unique. Every other category rule respects the verified/unverified distinction — verified search bots pass, unverified scrapers get blocked. CategoryAI blocks both by default. Per AWS documentation, this is the one category where AWS treats all AI bots as hostile until proven otherwise. That’s the right posture and the foundation of the enforcement layer.

Signal rules catch stealth crawlers. SignalAutomatedBrowser, SignalNonBrowserUserAgent, and SignalKnownBotDataCenter detect crawlers using spoofed user-agents, rotating IPs, or datacenter egress — the exact techniques Cloudflare documented Perplexity using in 2025.

The label namespace is what you write policy against. bot:name:<n>, bot:verified, bot:unverified, and the bot:web_bot_auth:<status> labels added in Bot Control v4.0 with Web Bot Authentication support (launched November 2025). Scope-down statements match on labels, not user-agent strings, because user-agents can be spoofed and labels are applied by AWS after verification. Web Bot Authentication is where this is headed long-term — cryptographic identity for AI agents — and any allowlist logic written today should leave room for bot:web_bot_auth:verified as the preferred match condition once crawler support catches up.

The Enforcement Standard

Security publishes three rules as the standard. App teams deploy them.

CategoryAI = Block. Every AI bot hits the wall unless explicitly exempted. No Count mode. No grace period. The default is Block because the declaration in Tier 1 is Block for everything until a human says otherwise.
Path-based rules enforce every Disallow directive. Admin, internal, private paths get blanket bot blocks regardless of user-agent.
Targeted rules on sensitive endpoints. Login, checkout, cart, and API surfaces get targeted inspection level regardless of declared identity, because the cost of a false negative on those endpoints is too high to trust identification alone.

App teams deploy this baseline however they already deploy WAF configurations. Security will do the audit: every public-facing property is checked against the baseline on a recurring cadence. Missing CategoryAI Block rule, missing path-based enforcement, missing targeted inspection on sensitive endpoints — each one is a finding routed to the app team for remediation. The audit is the control. The deployment is an implementation detail.

The Audit Template

Here’s the template for the audit output. Every row is one internet-facing application, sourced from the AWS WAF AI Activity Dashboard over a 30-day window. The last column is the only one that matters — if reality isn’t matching the declaration, the row becomes a finding.

Every ✗ becomes a ticket. The ticket either fixes the enforcement (bring Tier 2 in line with Tier 1) or fixes the declaration (bring Tier 1 in line with reality — which means filing an allow request or updating the template). Both paths are legitimate. Silence is not.

Sensitive Path Inventory

The audit template tells you whether the baseline is deployed. The sensitive path inventory tells you which paths on each application need enforcement beyond the baseline. These are the paths that were already identified as sensitive in robots.txt — WAF Bot Control makes the restriction actually enforceable.

Four generic categories cover most enterprise web estates:

The pattern across all four is the same: every path that appears as Disallow in Tier 1’s declaration gets a corresponding block rule in Tier 2. The four categories just organize the rules by business purpose so reviewers can apply different enforcement postures without hand-building every rule individually.

The last row deserves one additional note. If an AI Activity Dashboard audit shows bot traffic hitting /admin or any internal path on a public URL, the finding isn’t just “the WAF needs a rule” — it’s “this application is exposing an internal surface at a public URL, and the real fix is upstream of the WAF.” The sensitive path inventory is where Tier 2 enforcement intersects with application security review, and the audit catches both.

Tier 3: Governance

The AI governance organization owns this tier. App owners and business stakeholders file their requests through it.

Every allow runs through the same loop: Default State → Monitor → Allow Request → Review → Implement → Validate → Ongoing. Not a sequence — a loop. Allows get granted, validated, and re-justified on a cadence, or they accumulate into the mess this post exists to prevent.

Default State. CategoryAI blocks all AI bots. Every internet-facing application starts here and returns here if an allow lapses.

Monitor. The AI Activity Dashboard, CloudWatch, and WAF logs. App teams and security watch for blocked requests from named bots that might represent undeclared business dependencies, and for allowed bots behaving outside their declared scope.

Allow Request. A business unit submits a written request naming the bot, the business justification, the specific paths the bot needs, and expected request volume.

Review. Security validates identity via Bot Control labels and determines access level (full, path-restricted, or rate-capped). The reviewer’s first question isn’t how much risk the allow carries — it’s which risk dimensions are material for this specific application. A static sustainability microsite and a dynamic pricing page both pass through the same review process, but they’re scored against different dimensions because they have different things to lose.

Three risk dimensions inform every allow review:

Customer journey interception and brand misrepresentation are universal — every enterprise with a public web estate carries some exposure on both. Dynamic data exposure is concentrated in specific business types: e-commerce, travel, financial services, and any application with logged-in or personalized experiences visible at public URLs. A brochure-ware application with mostly static marketing pages scores low on this dimension and the review moves on. The reviewer’s job is to decide which dimensions are live for this application before scoring any of them.

Against those three risks, one mitigation lens shifts the math on whether an allow is acceptable: static content as sunk cost. When the content a crawler wants to index has already been paid to produce, has no ongoing revenue tied to gatekeeping it, and captured its business value at publication rather than through ongoing access control, the incremental risk of letting a crawler consume it is close to zero. A completed blog post, a published press release, a closed-campaign landing page, an archived product overview — the money was already spent, the content was always intended for broad distribution, and blocking it forfeits visibility without protecting any live business value. Sunk cost is the reviewer’s defense argument when one of the three dimensions scores high but the content has no ongoing gating value. It’s what makes an allow defensible in cases where a pure risk read would lean toward block.

The decision to allow still requires the named business owner to sign.

Implement. Three mechanical options in order of preference: label-based allow (override CategoryAI to Count, add a custom allow rule matching on bot:name plus a verification label), scope-down exclusion (exempt a specific path plus a shared secret header from CategoryAI evaluation), or WBA-based allow (match on bot:web_bot_auth:verified — future state, limited crawler support today).

Validate. Seven-day watch on the newly allowed bot. Revocation triggers: volume anomalies, path access outside declared scope, signal rule hits like SignalKnownBotDataCenter on a bot you’d verified, or any behavior inconsistent with the stated business purpose.

Ongoing. Quarterly review of every active allow against the allowed list register. Every allow expires unless the business owner re-justifies it in writing.

How This Scales Across Hundreds of Applications

At a large enterprise, “the company’s website” isn’t one website. It’s hundreds of internet-facing applications — product sites, regional domains, microsites, acquired brands, support portals, developer docs, campaign landing pages — each operated by a different app team, each with its own WAF configuration, each with its own robots.txt or no robots.txt at all. That's the scale the three-tier model is built for.

robots.txt scales through templates. The three co-authors publish a standardized robots.txt template, Legal reviews it, and every app team forks it for their own properties. Deviations require written reasons reviewed by the same co-authors and re-approved by Legal. Nobody owns every property’s file — the co-authors own the template and the diff review.

The enforcement standard scales through the audit. Security publishes the Bot Control baseline and checks every public-facing property against it on a recurring cadence. App teams deploy the rules through whatever IaC or console workflow they already use. Security finds the gaps and routes them back as findings. The audit is what makes the standard real across hundreds of properties without central deployment tooling touching anyone’s account.

The allowed list register scales through one cadence. Every allow across every property lives in one register with one review cadence. App teams and business owners make the decisions. The AI governance organization owns the process, the cadence, and the audit trail. Decentralized execution, centralized governance.

This is the only model that answers the ChatGPT advertising collision from my previous post at enterprise scale. Marketing reads the template, sees OAI-SearchBot is Block by default, files an allow request through the allowing process, and gets a label-based allow deployed against the specific properties tied to the campaign — with an expiry date, a named owner, and a quarterly review on the calendar. The collision becomes a governance event and the ad goes live on schedule.

Your First Controlled Opening

The controlled openings model is a program, not a one-week project. But every program starts with a first move, and the first move is different depending on which seat you’re in.

If you’re in Enterprise AI leadership: Find the first three business stakeholders who have already asked, or are about to ask, "why is this AI tool blocked?" Marketing wanting to run an ad campaign inside ChatGPT. Sales wanting Perplexity to surface the company in answer results. Comms wanting press releases indexed by AI search. Those are your first three controlled openings — not because the risk is low, but because the demand is already there and the conversation has to happen anyway. Get ahead of it by walking them through the allowing process before they hit the wall.

If you're in the AI governance organization: Stand up the allowed list register before any allows exist. Empty is the right starting state. The register is the artifact every other decision in the model points back to, and standing it up takes a spreadsheet and a recurring review — not a tooling project. The day you have a register, the model has a memory. Without one, every allow is a snowflake and every audit is a search.

If you own a public-facing application: Pull 30 days of bot traffic from your WAF logs or the AI Activity Dashboard. You cannot declare a posture until you know what’s already hitting you. Half the app teams who think they’re blocking everything are quietly allowing a dozen crawlers they’ve never named.

If you’re in security: Publish the Bot Control baseline as a written standard. Not “when we have time” — the standard is the prerequisite, not the follow-up. You cannot find gaps against a baseline that doesn’t exist, and the whole audit function collapses without one. Enforcement across the estate can lag the standard — that’s what the audit is for — but the standard itself is required. The standard is what makes the audit possible, and the audit is what makes the standard real.

None of these moves wait for permission. If you're reading this and your honest answer is "I'd need to charter a program to do any of that," the framework isn't the blocker — your operating model is.

When the User Is the Bot

The three-tier model handles bots that announce themselves and the ones that don’t. It does not handle the case where the bot is acting on behalf of an authenticated, authorized customer.

Agentic browsers are shipping now, not coming. ChatGPT Atlas, Perplexity’s Comet, Claude in Chrome, Gemini wired into Google’s stack — your customers are already using them to interact with your applications, and they will be using them more next quarter than this one. When a customer tells their agent to log in and complete a purchase, they have, with full authorization, violated your “no automated access” Terms of Service (ToS) and your “I am a human” login attestation. Your security stack was built to stop unauthorized automation. This is authorized automation. Nothing in the three tiers catches it.

That's the next post.

Wish, Collision, or Policy

A robots.txt that matches the WAF enforcement across every app team’s properties, governed by an allowing process with named owners and expiry dates, is a policy.

A WAF without a matching robots.txt is a collision waiting to happen. Silent rule groups enforcing decisions no business stakeholder ever saw, until the day a marketing campaign hits the wall and the incident review has to reconstruct who decided what, when, and why.

A robots.txt without matching WAF enforcement is a wish. The honest crawlers respect it. The ones who don’t treat the declaration as decoration.

Only the first state is a policy.

Declare in robots.txt. Enforce in AWS WAF Bot Control. Govern through the allowed list register. Closed by default. Every opening is scoped to a single application, owned by a named human, and expires on a date. That’s a controlled opening. Everything else is a wish.

Next week: whether you can actually detect agentic browser traffic. Subscribe so you don't miss it.

StepToCyber

Discussion about this post

Ready for more?