Agent 8 · page guide Want me to connect this article’s main idea to your website? Ask Agent 8
ConversationAgent 8 Sales Guide
AYSA Agent 8

The useful point here is that visibility in AI answers is becoming a real client requirement, but it does not replace the Google SEO foundations that help every search system understand and trust a website. GEO extends good SEO; it is not a separate shortcut.

What I can execute: For your website, I can check both layers: crawlability and search performance first, then answer-ready content, entity clarity, citations, and brand mentions. I will turn the gaps into specific actions and ask for approval before anything is changed.

Ask me anything about this page or what I could do for your website.

Do not share passwords, access keys, payment details, or other sensitive information.
SEO Strategy Nov 16, 2026 18 min read

Indexing Is Still Allowed. Agents Need Permission. What United’s New Rules Signal For AI Search, SEO, And “Buy-For-Me” Automation

United drew a bright line between traditional search indexing and autonomous AI agents that shop, book, scrape, or ground answers on its data. That line is about to matter for every brand—because AI discovery is becoming action, not just information.

Featured image for Indexing Is Still Allowed. Agents Need Permission. What United’s New Rules Signal For AI Search, SEO, And “Buy-For-Me” Automation

United Airlines just published one of the clearest “lines in the sand” we’ve seen so far in the AI era: search engines can index public pages, but AI agents that search, book, purchase, or collect information on the site need written permission—and United bars the use of its site/app data to train, fine-tune, ground, or evaluate AI models.

This isn’t only a travel story. It’s an early signal of how the web is being re-partitioned into (1) content that remains discoverable via traditional search Indexing and (2) data and workflows that companies will increasingly treat like protected infrastructure once AI agents can act on a user’s behalf.

As the web shifts from “search to click” toward “ask to action,” businesses need to update their mental model—and their technical + legal posture. If you’re a founder, marketer, or agency leader, this is the moment to stop treating AI as a single bucket (“AI crawlers”) and start treating it as multiple behaviors: indexing, summarizing, grounding, scraping, and transacting.

This editorial uses United’s move as a practical case study and turns it into an action plan for SMEs and mid-market companies. I’ll explain what changed, why it matters, what can go wrong, and how to respond without torching your organic growth. I’ll also share how we think about this at AYSA—where Monitoring and Approved Execution become the difference between a smart policy and a broken website.

Concise summary

Whiteboard comparing search indexing with autonomous AI agents taking actions like booking or purchasing.
Indexing discovers and organizes content. Agents can transact—so the risk and rules change.
  • United allows traditional search engines to index public pages, but requires written permission for AI agents that search, book, buy, manage, or extract/copy data from united.com or its app.
  • United’s terms also bar using its data to train, fine-tune, ground, evaluate, or otherwise develop AI models—a specific callout that targets how modern AI products are built and improved.
  • The restriction is not primarily enforced via Robots.txt; it’s enforced by terms + contracts + technical blocking (and, for travel, distribution relationships).
  • For businesses, the big takeaway is simple: indexing (visibility) is being separated from agents (action). You need a strategy for both.

Key takeaways for business owners and marketers

Small business team reviewing availability and pricing while discussing AI access rules.
If your business has inventory, pricing, or compliance steps, AI agents raise new operational risk.
  • Don’t confuse indexing with agents. Indexing helps customers find you. Agents can log in, transact, and trigger liability/compliance issues.
  • Robots.txt isn’t enough. Many personal agents don’t present a reliable user agent string, and policy enforcement often happens at the application/security layer.
  • Policy without execution is theater. If your terms say “no agents” but your site is wide open to automated flows, you have a gap. If you block too aggressively, you can destroy revenue channels.
  • SEO is becoming AEO/GEO. Being “indexable” isn’t the finish line anymore; you also need to be choosable in AI answers and recommendation systems.

Table of contents

Three cards labeled robots.txt, Terms, and Bot Protection representing a layered approach to AI access control.
Robots.txt is only one layer. Legal terms and enforcement are the other two.

What changed at United (and what didn’t)

According to reporting by Search Engine Journal, United updated its website terms (Nov. 11) to explicitly allow search engines to index public pages while requiring written permission for “Automated Agents” that interact with united.com or the United app in ways that go far beyond reading content—searching, booking, purchasing, managing, extracting, or copying data. The terms also bar using United’s data to train, fine-tune, ground, evaluate, or otherwise develop AI models.

Two details matter here:

  1. This is not framed as “anti-SEO.” The terms make a carve-out for search indexing of public pages.
  2. This is framed as “anti-unauthorized automation.” The airline is targeting a category of tools that behave like assistants but can also behave like bots—especially when they can log in, transact, or extract structured commercial data.

Search Engine Journal also notes that United’s robots.txt (as of the reporting date) did not list AI crawlers. That’s a big clue: this is less about traditional crawler management and more about behavior management through terms and enforcement. (Source: Search Engine Journal.)

What didn’t change? Public web pages are still meant to be found. That’s the subtlety most brands will miss if they only hear “United blocks AI.” It’s more accurate to say United is differentiating indexing from agents.

The shift: from “crawling pages” to “agents taking actions”

For 20+ years, most businesses thought about automated access in a single bucket: crawlers. You either let them in, rate-limit them, or block them.

Now we have at least four different behaviors that get lumped into “AI,” but carry very different business implications:

  • Indexing: discovering and storing information so it can be retrieved later (classic search engine behavior).
  • Answering/summarizing: generating responses based on indexed content, which may or may not send Clicks back.
  • Grounding: pulling in external content at query-time to improve accuracy and specificity of an answer.
  • Agents/transacting: logging in, checking availability, booking, purchasing, modifying orders, or interacting with accounts.

United’s policy move is important because it treats “agents” as fundamentally different from indexing—different risk, different economics, and different control requirements.

If you run a business with any of the following, you should pay attention:

  • customer accounts (PII, saved payment methods, loyalty points)
  • inventory/availability (rooms, seats, appointments, limited stock)
  • dynamic pricing (fares, surge pricing, promotions)
  • regulated steps (age verification, attestations, waivers)
  • mission-critical workflows (cancellations, returns, rescheduling)

In other words: almost every business beyond a simple brochure site.

Why United did it: user safety, regulatory exposure, and disintermediation

United’s public reasoning (as summarized by Search Engine Journal) centers on customer safety and security: users sharing login credentials with third-party assistants can expose sensitive information, and automated actions could lead to unintended purchases or account changes. United also points to the risk that automated actions could violate FAA and DOT regulations—using check-in as an example where a person must acknowledge that their bags contain no prohibited items.

Even if you’re not in a regulated industry, the logic generalizes:

  • Authentication is fragile. When third-party tools “act as the user,” businesses lose visibility into who is actually behind the keyboard.
  • Consent and confirmations are legal artifacts. Many flows are designed around human acknowledgement. Agents can break the assumption that a human reviewed the statement.
  • Support burden increases. If an agent changes a reservation, buys the wrong item, or triggers fraud rules, your support team deals with the fallout.

There’s also a competitive angle. Search Engine Journal cites a Wall Street Journal report in which United’s CEO said he didn’t want to create another middleman. That sentence matters because it describes the next distribution fight: if AI agents become the default “interface layer” for commerce, brands will be pressured to pay for access, yield margin, or accept loss of customer relationship.

So this is not just “security policy.” It’s channel strategy.

The technical line in the sand: robots.txt vs terms vs enforcement

Many marketers instinctively ask: “Did they block AI crawlers in robots.txt?”

United’s approach (again, per Search Engine Journal’s reporting) is the opposite: it separates indexing from agent behavior in its legal terms and related contracts, without relying on robots.txt for AI-specific agents.

That’s aligned with a hard reality: robots.txt was built for cooperative crawlers. It’s not a security mechanism. It’s a set of instructions that compliant bots might respect—and non-compliant systems can ignore.

For modern AI agents, there’s an additional complication: many don’t present a stable user agent string you can reliably match. As Search Engine Journal notes (via an example about personal agents), this makes “just add a rule to robots.txt” an incomplete solution.

The emerging best practice is a three-layer model:

  1. Robots directives (discovery control): manage cooperative indexing and reduce unnecessary crawl load.
  2. Terms and policies (legal control): define what is permitted, who is bound, and what uses are prohibited.
  3. Technical enforcement (behavior control): bot management, rate limiting, anomaly detection, auth hardening, and transaction safeguards.

If those three layers disagree, you’re either:

  • overexposed (policy says “no agents,” but nothing enforces it), or
  • overblocked (security team blocks patterns that also block search visibility and legitimate users).

The “no grounding / no training” clause: why it’s a big deal

One of the most consequential parts of the United language described by Search Engine Journal is the explicit prohibition on using United data to “train, fine-tune, ground, evaluate, or otherwise develop” AI models.

Why that matters:

  • It targets the AI product lifecycle. Training and fine-tuning are how models learn; evaluation is how they’re tested; grounding is how they generate more accurate, source-linked answers in real time.
  • It creates a clean permission boundary. “Indexing” is allowed, but “using our data to power your AI system” is not—unless approved.
  • It changes vendor conversations. If you’re a platform, agency, or tool provider, you now have to separate “we can read public pages” from “we can use your data inside our AI output.”

There’s an unresolved gray area that United’s move highlights for the entire market: What about AI answers generated from search engine indexes? Search Engine Journal points out that the terms do not address how answers created by AI systems from search engines’ indexes are treated. This ambiguity will likely persist until courts, regulators, and industry agreements catch up.

For business owners, the takeaway is practical: you should assume your content can influence AI answers through multiple pathways—some you control (your site), some you don’t (third-party indexes, aggregators, and platform summaries).

This doesn’t stop at your site: agencies, affiliates, platforms, and contracts

United didn’t only update website terms. Search Engine Journal reports that United also updated its contract of carriage and extended restrictions in ways that impact travel agencies and booking platforms—specifically restricting the use of third-party AI agents to search, book, or manage United travel on behalf of customers without permission.

Even if you’re not in travel, the bigger pattern is critical: AI agent governance will move “downstream” into partner agreements. That includes:

  • affiliate and reseller contracts
  • marketplace and channel partner policies
  • API terms of use
  • vendor security requirements

If you run a brand with partners, you’ll need to answer: Are partners allowed to use AI agents on your customers’ behalf? And if yes, under what conditions (authentication, logging, rate limits, liability, audits)?

If you run an agency, you’ll need to answer: Are you introducing tools that could breach a client’s platform terms? Not maliciously—just operationally. That risk is new, and it’s easy to stumble into it.

What this means for AI search, AEO, and GEO

We’re entering a split era:

  • SEO remains necessary for discovery, indexing, and brand demand capture.
  • AEO/GEO becomes decisive for being selected in AI-generated answers and recommendations (where the click may not happen).

United’s move reinforces that businesses may want to keep the top-of-funnel wide open (indexing) while putting gates around the bottom-of-funnel (agents that transact or scrape commercial data).

That’s also how most SMEs should think—because blocking discovery harms growth, but unmanaged automation can harm operations.

In practice, this means your “AI search strategy” needs three components:

  1. Visibility: ensure public content is indexable and easy to understand (technical SEO, content clarity, structured data).
  2. Eligibility: make your content “citable” and recommendation-friendly (clear policies, accurate entities, schema where appropriate, strong about/contact signals).
  3. Control: define what automation is allowed (terms), and enforce it (security and product constraints).

At AYSA, this is exactly why we treat AI visibility as an execution discipline, not a one-time audit. If you want the framework, see AI Search Visibility and our AI SEO tools overview.

A concrete SME scenario: the boutique hotel caught between AI discovery and AI booking bots

Let’s make this real with a scenario that mirrors the United situation, but at SME scale.

Business: a 40-room boutique hotel in a mid-sized U.S. city.
Stack: WordPress site, booking engine embedded, Google Business Profile, some OTAs, basic analytics.
Goal: keep direct bookings growing, reduce OTA commissions, protect staff time.

Here’s what changes as AI agents become mainstream:

  • Guests ask an AI assistant: “Find me a quiet boutique hotel near downtown with parking and late checkout.” The assistant summarizes options without always clicking through.
  • Guests try “book it for me” features that may use browser-based agent capabilities to navigate the hotel site and booking flow.
  • Third parties may deploy scrapers/agents to watch room rates and availability for resale or arbitrage.

If the hotel overreacts and blocks anything “AI,” it may reduce discovery in search and AI answers. If it does nothing, it can end up with:

  • fake bookings or reservation churn from automation
  • PCI or account-security issues
  • increased customer support load (“I didn’t book that”)
  • rate parity or inventory leakage issues

The smart middle path looks a lot like United’s logic:

  • Keep public pages open for indexing (rooms, amenities, policies, location pages, FAQs).
  • Put stricter controls around transaction endpoints (checkout, booking confirmation, account management).
  • Update terms so “unauthorized agents” that transact or extract data are not permitted without written permission.
  • Monitor for suspicious automated patterns and apply rate limits or step-up verification where needed.

This is exactly the kind of balance SMEs need—because your website must still market and sell, but you can’t ignore the new automation layer.

What can go wrong (realistic failure modes)

When companies respond to AI agents, I see a handful of predictable mistakes. None require bad intent—just the wrong mental model.

1) Blocking discovery while trying to stop automation

The fastest way to harm revenue is to apply broad blocking at the wrong layer (e.g., overzealous bot rules that catch search engine crawlers or legitimate users behind privacy browsers/VPNs). If you lose indexation or reduce crawlability, you don’t just lose rankings—you lose the pipeline that feeds every other channel.

2) Writing policy language that can’t be enforced

Terms that say “no bots” without defining behaviors (“search,” “copy,” “purchase,” “manage accounts”) are too vague to be operational. Conversely, extremely strict clauses without practical exceptions can be impossible to maintain—especially if your own vendors use automation for monitoring, QA, or accessibility.

3) Leaving a loophole through partners

United extended controls through travel agencies and booking platforms. Many brands won’t. If your partners can deploy agents against you, your site-level controls won’t be enough. You need contractual alignment, not just technical controls.

4) Treating AI answers as “someone else’s problem”

Even if you disallow agents, your content can still shape AI-generated answers through indexing and summarization. If your policies, prices, or product details are unclear, AI systems can misrepresent you. That’s an SEO problem and a customer experience problem.

5) No monitoring, so you only learn after damage is done

In the AI era, change happens quietly: new referrers, new user behaviors, new spikes in automated traffic, new ranking volatility across AI surfaces. If you don’t monitor, you will diagnose problems too late—after conversions drop or support tickets spike.

This is why we built AYSA with monitoring at the center: AYSA Monitoring. You can’t manage what you don’t continuously observe.

What to monitor now (even if attribution is imperfect)

Most SMEs don’t have perfect visibility into “agent traffic.” That’s normal. But you can still monitor patterns that strongly correlate with agent activity or AI-driven discovery changes.

Operational and security signals

  • Login anomalies: unusual login attempts, rapid account switching, repeated failed logins.
  • Checkout friction spikes: sudden increases in abandoned carts at specific steps.
  • Rate-limit triggers: repeated requests to pricing, availability, or search endpoints.
  • Unusual user paths: navigation that looks like automation (very fast, repetitive sequences).

Search and discovery signals

  • Indexation health: are key pages still indexed and crawlable?
  • Query shifts: more long-form, conversational queries landing on FAQs/policy pages.
  • SERP/AI surface volatility: brand queries producing different answer boxes, summaries, or fewer clicks (harder to attribute, but changes show up in traffic patterns).

Content integrity signals

  • Policy consistency: does your cancellation/returns/shipping info match across pages?
  • Entity clarity: are your brand, products/services, and locations described consistently?
  • Structured data coverage: where it’s appropriate, does schema align with page content and business reality?

Monitoring isn’t glamorous, but it’s the only way to avoid reacting blindly. If you want a practical system that flags issues and prepares changes you can approve, explore the AYSA approach starting with AI Search Visibility.

A practical action plan (policy, product, and SEO working together)

If you’re a business owner or marketing leader, here’s the playbook I recommend. It’s written for SMEs, but it scales up.

Step 1: Map “public discovery” vs “protected actions”

Create a simple inventory of your site/app surfaces:

  • Public content: marketing pages, knowledge base, FAQs, location pages, public product catalogs (maybe).
  • Semi-public data: pricing, availability, promotions, feeds, search endpoints.
  • Protected actions: login, checkout, booking, account changes, cancellations/refunds, loyalty points, saved payment methods.

Your goal is not to “block AI.” Your goal is to decide what can be indexed and what requires explicit permission or stricter controls when automation is involved.

Step 2: Align the three layers (robots, terms, enforcement)

Use a structured checklist:

  • Robots.txt: confirm key public pages are crawlable; prevent waste on low-value endpoints if needed.
  • Terms/policies: define “automated agent” behaviors in plain language; set permission requirements; clarify prohibited uses (scraping, grounding, model development) if that’s your choice.
  • Enforcement: rate-limit sensitive endpoints, add bot protection where appropriate, protect logins, and consider step-up verification for high-risk actions.

United is a strong example of separating “indexing allowed” from “agents require permission.” Read the original reporting here: Search Engine Journal.

Step 3: Decide your stance on AI “grounding” and “training”

You may not care about model training, but you should still decide what you want. Options include:

  • Open posture: allow indexing; accept that content may be summarized or grounded; focus on being accurately represented.
  • Controlled posture: allow indexing, but restrict automated extraction and explicit grounding/model development from your transactional data.
  • Licensed posture: offer an API or paid access for partners/agents under contract terms.

SMEs usually start with “open” for marketing content and “controlled” for transactional surfaces. That’s the pragmatic default.

Step 4: Build content that AI systems can’t easily get wrong

This is the AEO/GEO side. If AI answers are going to summarize you, make summarization easy and accurate:

  • Put key details in one canonical place (shipping, returns, cancellation, pricing rules).
  • Use consistent language across pages.
  • Make your “about” and “contact” information unambiguous.
  • Use structured data where appropriate (carefully—accuracy matters more than volume).

If you want a practical starting point for AI-era visibility work, see AI SEO Tools and browse our latest thinking at AYSA Blog.

Step 5: Prepare for the “agent economy” without surrendering your margin

United’s CEO comment (as cited by Search Engine Journal) is a warning: AI agents can become new middlemen. Even if you’re not United, the same dynamic can show up in your niche.

Questions to ask now:

  • If an AI assistant becomes the default interface for shopping in your category, what happens to your brand demand?
  • Do you have a differentiated story (quality, expertise, trust, guarantees) that an agent can relay?
  • Do you have direct relationships (email/SMS, memberships, loyalty, community) that reduce platform dependence?

Where AYSA fits: monitor → prepare → approve → execute

Most teams fail in the gap between knowing and doing. They run an audit, identify issues, then nothing ships—because the work crosses SEO, engineering, content, legal, and operations.

AYSA is built to close that gap with an approved execution model:

  1. Monitor: track the signals that matter for AI visibility and technical health (Monitoring).
  2. Prepare: generate prioritized recommendations and change sets (technical fixes, content updates, structured data, internal linking, policy-page improvements) designed for AI-era discovery.
  3. Ask for approval: you stay in control; nothing silently changes on your site.
  4. Execute accepted changes: ship improvements consistently, not just in quarterly bursts.

That matters specifically for this “indexing vs agents” moment because it demands cross-functional precision:

  • You can’t let legal write terms that contradict your technical setup.
  • You can’t let security block patterns that tank your crawlability and organic traffic.
  • You can’t let content drift create AI answer inaccuracies that support then has to clean up.

If you want to operationalize this without building a giant internal program, start here:

What to do next

If you do nothing else after reading this, do these seven actions in order:

  1. Write down your “public vs protected” map (public marketing content vs transactional endpoints).
  2. Confirm indexation basics: your top pages should be crawlable and internally linked.
  3. Review robots.txt for accidental self-sabotage (blocking key sections, over-broad disallows).
  4. Review your terms (or ask counsel) to define automated agent behavior and permission rules.
  5. Harden your high-risk flows: login, checkout, account changes, refunds/cancellations.
  6. Improve AI-readability: canonical policy pages, clear FAQs, consistent product/service descriptions.
  7. Set up ongoing monitoring and execution so this becomes a system, not a one-time reaction.

If you want to implement this with an approval-first execution workflow, explore AYSA’s monitoring and AI visibility solutions: Monitoring and AI Search Visibility.

Sources and further reading

Note: This editorial references United’s terms and related details as described in the Search Engine Journal source linked above. For legal interpretation or implementation, consult your counsel and review the most current official documents directly.

Related AI SEO resources

Continue the AI search topic inside AYSA.

Use these pages to connect the article with AI SEO tools, AI visibility monitoring, AI Overviews and approved website execution.

Execution hubs

Turn this topic into a website action plan.

Use these AYSA hubs to move from reading to technical fixes, AI visibility monitoring, research, glossary context and approval-first SEO execution.

Marius Dosinescu, author at AYSA.ai

Written by

Marius Dosinescu

Marius Dosinescu is the founder of AYSA.ai, an entrepreneur focused on SEO automation, ecommerce growth, authority building and approved website execution for businesses that want organic growth without specialist overhead.

SEO execution, not more busywork

Turn SEO reading into approved website action.

AYSA monitors your website, prepares the work, asks for approval, and executes approved changes inside your website.

Start now View pricing

Only €29 to €99 per month, depending on the size of your business.

AYSA SEO Magazine

Latest search intelligence.

View all articles