AI's impact on SEO: how ChatGPT, Perplexity, Copilot and Gemini find and cite pages

What changes when people search through AI assistants

An assistant that searches the web works in three steps. It turns the question into one or more search queries, retrieves pages from a search index, and writes an answer from what it read, linking some of the pages it used. Google's guide to generative AI search, last updated in July 2026, calls the retrieval step retrieval-augmented generation, or grounding, and the extra searches query fan-out. Microsoft reports the same step from the site owner's side: Bing Webmaster Tools lists grounding queries, the key phrases the AI used to retrieve content it went on to cite.

Three things follow for strategy.

• Passages compete, not only pages. Microsoft describes assistants such as Copilot breaking pages into smaller pieces, judging each piece for authority and relevance, and assembling one answer from several sources. A section that answers one question cleanly can be used even when the rest of the page covers something broader.
• Visibility does not always produce a click. The Bing Webmaster Guidelines say content can appear as impressions, citations or grounding references in Copilot responses, and that a fall in clicks does not always mean a loss of visibility. Measurement has to count citations as well as visits.
• The foundations do not change. Google says that optimizing for generative AI search is optimizing for search, and therefore still SEO. Bing says the practices that support discovery, accurate indexing and clear content also support eligibility for grounding and citations. A page that is blocked, fails to render or is missing from the index cannot be cited by anyone.

Two names for this work are now common. Answer engine optimization (AEO) and generative engine optimization (GEO) both describe work aimed at visibility in AI answers; Google's guide lists them as such and says it treats the work as SEO. Bing's guidelines define GEO as work on a page's eligibility for grounding and reference in AI responses, and state that GEO no more guarantees a citation than SEO guarantees a ranking. The term itself comes from research first posted in November 2023 and accepted to KDD 2024, whose findings appear below.

How each AI engine finds and cites pages

Each operator documents which bot feeds its answers and which settings limit other uses of your content. As of September 2026:

ChatGPT search

OpenAI launched ChatGPT search on October 31, 2024, and since February 5, 2025 it has been open to everyone in regions where ChatGPT is available, without signing up. OpenAI says it draws on third-party search providers and on content supplied by partners, and its own crawler documentation separates three user agents:

• OAI-SearchBot surfaces websites in ChatGPT's search features. Sites that opt out are not shown in ChatGPT search answers, although they can still appear as navigational links. OpenAI recommends allowing it in robots.txt and allowing requests from its published IP ranges.
• GPTBot crawls content that may be used to train OpenAI's foundation models. It is an independent setting: blocking it does not remove you from search.
• ChatGPT-User loads a page when a user's question calls for it. It does not crawl automatically and is not used to decide what appears in search, and because a person started the request, OpenAI says robots.txt rules may not apply.

OpenAI adds that robots.txt changes take about 24 hours to reach search. Its publisher FAQ also notes that its Atlas browser may show just the link and title of a disallowed page if OpenAI learns the URL from a third-party search provider or from other pages. A noindex meta tag prevents that, provided the crawler is allowed to fetch the page and read it.

Perplexity

Perplexity's crawler documentation lists two agents. PerplexityBot surfaces and links websites in Perplexity's search results and, Perplexity says, is not used to crawl content for AI foundation models, so blocking it buys no training protection. Perplexity-User visits a page when a user's question needs it and may link the page in the answer; because the user asked, it generally ignores robots.txt. Perplexity publishes IP ranges for both, gives example allow rules for Cloudflare and AWS WAF, and says robots.txt changes can take up to 24 hours to take effect.

Microsoft Copilot

Copilot answers from Bing. Microsoft says Copilot and its other search experiences are powered by Bing's search index, and the Bing Webmaster Guidelines cover Bing search, Copilot and grounding API results together. SEO for Copilot is therefore SEO for Bing:

• Do not block Bingbot, and do not hide critical content behind client-side rendering. The guidelines list both as things to avoid.
• Verify the site in Bing Webmaster Tools and submit an XML sitemap that lists only canonical URLs, with accurate lastmod dates.
• Use IndexNow to tell Bing when URLs are added, updated or removed. Bing says timely notifications reduce outdated or incorrect URL references in Copilot answers.
• Know the directives Bing applies to AI use. noarchive keeps a page's content out of Copilot responses and grounding results, and nocache limits Copilot to the URL, title and snippet. nosnippet and data-nosnippet stop Bing showing captions and may lower the quality of citations.

IndexNow is not Bing-only. As of September 2026 the IndexNow FAQ lists endpoints for Amazon, Bing, Naver, Seznam.cz, Yandex and Yep. Google is not among them, so IndexNow does nothing for Gemini or AI Overviews. For businesses that serve an area, Microsoft also points to Bing Places to keep the address, hours and contact details current and eligible for AI answers; local SEO covers those profiles across engines.

Gemini and Google's AI features

AI Overviews and AI Mode are part of Google Search and use its index. Google's page for site owners says that to appear as a supporting link, a page must be indexed and eligible to show with a snippet, with no additional technical requirements and no special markup. Robots.txt rules for Googlebot are the control, and nosnippet, data-nosnippet, max-snippet and noindex limit what Search shows from a page, AI features included. Since August 31, 2026, every site also has a Search generative AI control in Search Console settings, which excludes the site from AI Overviews, AI Mode and generative AI features in Discover while leaving the rest of Search alone; AI Overviews in Google Search compares these controls. How those features pick their sources is part of how the Google search engine works.

The Gemini app has a separate control. Google-Extended is a robots.txt token, not a crawler: it tells Google whether content it crawls from your site may be used to train future Gemini models and for grounding in the Gemini app and in Grounding with Google Search on Vertex AI. Google states that it does not affect inclusion or ranking in Google Search. Blocking it keeps you in Search, AI features included, but tells Google not to use your pages to ground Gemini app answers. For background on the models themselves, see Google Gemini explained.

Claude

Anthropic's crawler article, dated April 7, 2026, describes three bots:

• ClaudeBot collects content that could contribute to model training. Blocking it signals that the site's future content should be excluded from training datasets.
• Claude-SearchBot indexes content to improve search results. Blocking it may reduce the site's visibility in Claude's search results.
• Claude-User fetches pages when a person asks Claude something. Unlike the equivalent fetchers at OpenAI and Perplexity, it follows robots.txt, and blocking it may reduce visibility for user-directed search.

All three honour robots.txt and the non-standard Crawl-delay rule. Anthropic asks site owners to repeat their rules on every subdomain, and warns that blocking its IP addresses may not work as an opt-out, because it stops the bots from reading robots.txt.

Search access or training access: the robots.txt decision

Because search crawlers and training controls use different tokens, one robots.txt can let an assistant cite you while declining to have your content used for model training. The effects are not symmetric, so decide token by token:

One reasonable policy for a business site is to stay visible in every AI search product and opt out of training:

# Search and AI answers: allowed
User-agent: Googlebot
User-agent: Bingbot
User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: Claude-SearchBot
User-agent: Claude-User
Allow: /

# Model training: opted out
User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /

# Add this group only if you accept losing Gemini app grounding
# User-agent: Google-Extended
# Disallow: /

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

Before you rely on the file, check four things:

• User-triggered fetchers play by their own rules. OpenAI says robots.txt may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores it. Claude-User follows it.
• Changes are not instant, and every host needs its own file. OpenAI mentions about 24 hours, Perplexity up to 24 hours, and a file on www.example.com says nothing about docs.example.com.
• Your CDN may contradict your robots.txt. Google lists crawling allowed in robots.txt and by any CDN or hosting infrastructure among the SEO fundamentals for its AI features. Cloudflare's AI bot policies sort bots into Search, Agent (which includes chat fetch bots) and Training, and its documentation set new defaults for new domains from September 15, 2026: Training and Agent bots blocked on pages that display ads, Search allowed. A bot that serves both search and training is blocked by any setting that blocks training. Make sure the edge admits the bots your robots.txt invites.
• robots.txt is a request, not enforcement. Anyone can fake a user agent. The crawling guide explains how to verify bots against their operators' published IP ranges.

What llms.txt is, and who says they use it

llms.txt is a proposal by Jeremy Howard, first published on September 3, 2024; the current version 2 was last modified on August 10, 2026. It asks sites to publish a Markdown file at /llms.txt with a title, a short summary and links to clean Markdown versions of important pages, so that an AI agent can find what it needs without parsing navigation, ads and scripts. The proposal says the files are used most heavily for software documentation, where coding agents follow them to API references and tutorials. A minimal file looks like this:

# Example Co

> Example Co builds booking software for dental clinics.

## Docs

- [API reference](https://www.example.com/docs/api.md): endpoints, authentication and rate limits
- [Integration guide](https://www.example.com/docs/integrate.md): connecting a practice management system

Who reads it is the question that matters:

• Google Search does not. Google's guide to generative AI search says you do not need llms.txt or other AI text files to appear in Search, including its AI features, because Google Search does not use them. Creating one for other systems neither helps nor harms your Google visibility.
• Chrome's Lighthouse checks for one. Its agentic browsing audits flag a server error when fetching the file and mark a missing file as not applicable, because providing it is optional.
• The AI search crawlers' documentation does not mention it. As of September 2026, the crawler pages from OpenAI, Anthropic and Perplexity and Bing's webmaster guidelines say nothing about llms.txt. The proposal notes that OpenAI, Anthropic and Google publish llms.txt files for their own developer documentation, which shows the format is useful for docs; it is not a statement that their search bots read yours.

The practical answer: publish one if AI agents and coding assistants are an audience for your documentation or API, keep it accurate, and do not expect it to change search visibility or citations. It never replaces crawlable HTML.

What makes a page citable across AI engines

No engine publishes its citation formula, but Google, Microsoft and published research point to the same features.

1. Answer near the top. Bing's guidelines ask for essential information near the top of the page and warn against long introductions, and Microsoft notes that assistants can often lift a clear question-and-answer pair word for word. Put the direct answer in the first paragraph and the detail below it.
2. One topic per URL, with headings that say what follows. Bing says pages focused on one primary topic are more likely to be selected for grounding, and asks for titles, headings and content that agree. A heading written as the question it answers is easier to match than "Learn more".
3. Entities named clearly and consistently. Bing links clear, consistent names for people, organizations, products and places to grounding visibility and citation accuracy. Use the full name of a product or standard, say what it is, and keep the same name across the site; semantic SEO walks through mapping a topic's entities.
4. Statements that stand on their own. A cited passage is read without its surroundings. Bing asks for explicit facts and definitions, key statements that do not depend on implied context, and important information visible on the page itself. Microsoft adds that AI systems may not render content hidden in tabs or expandable menus, and that PDFs and images are weaker carriers of key facts than HTML.
5. Original data, evidence and sources. Google's guide favours non-commodity content, such as a first-hand perspective, over summaries of what others have already said. Bing's AI Performance announcement lists examples, data and cited sources as ways to build trust when content is reused.
6. Structured data that matches the visible page. Schema.org markup in JSON-LD helps machines label what a page is, but Google's structured data guidelines forbid marking up content readers cannot see, and Bing's say markup must accurately reflect visible content. Neither requires special markup for AI answers: Google says there is no special schema.org type to add, and Bing says structured data may support clearer grounding without guaranteeing it.
7. Fast, crawlable pages that carry their content in HTML. Bing warns against hiding critical content behind client-side rendering, and Google lists page experience and content available as text among the fundamentals for its AI features. The OpenAI, Perplexity and Anthropic crawler pages do not say whether their bots run JavaScript, so serve the main content in the initial HTML.
8. Fresh, accurate content. Bing asks for updates when facts change, accurate lastmod values and IndexNow notifications, so that answers do not cite an outdated version.
9. Mentions and links on sites people trust. Bing treats links from relevant external sites as a discovery and authority signal, and treats bought links and artificial promotion as abuse. Google says its AI features can reflect what is said about a business in blogs, videos and forum discussions, and that chasing inauthentic mentions is not as helpful as it might seem. Earn them with original research others cite and accurate listings; how Google judges authority explains why vendor authority scores are not the target.

The GEO study tested the content side of this on a benchmark of 10,000 queries. Rewriting a source to cite sources, add quotations from credible sources or add statistics raised its visibility in generated answers by 30 to 40 percent on one of the study's two visibility metrics and 15 to 30 percent on the other, while keyword stuffing, the classic SEO tactic, did not perform well. The gains were largest for pages ranked lower in classic search: citing sources raised visibility by 115.1 percent for sources in fifth place, while the top-ranked source lost 30.3 percent on average. These are research results on systems from 2023 and 2024, not a forecast for today's assistants, but they point the same way as the engines' own guidance.

Some popular tactics point the other way. Bing lists keyword stuffing and artificially engineered language meant to trigger citations or AI responses as abuse. Google says there is no need to break content into tiny chunks for AI, and that creating separate pages for every variation of a query, to manipulate rankings or AI responses, violates its scaled content abuse policy. Clear sections help; artificial fragments do not.

How to measure AI search visibility

Rankings alone do not show what happens in AI answers. Use five signals, each from a source you can check.

AI referral traffic in analytics

As of September 2026, GA4's default channel group includes an AI Assistant channel for visits from sources such as ChatGPT, Gemini, DeepSeek, Copilot and Grok, with the medium set to ai-assistant. Clicks from Google's AI Overviews and AI Mode are excluded and count as Organic Search. Open the Traffic acquisition report under Acquisition and read Session source within that channel.

Two gaps are worth closing:

• The default channel follows Google's list, which you cannot edit. To define your own, create a custom channel group with an "AI assistants" channel that matches a regex on source, and move it above Referral, as Google's help page advises, so those visits are not classified as referrals first. Check the source values in your own reports and adjust the list:

^(.*\.)?(chatgpt\.com|openai\.com|perplexity\.ai|claude\.ai|copilot\.microsoft\.com|gemini\.google\.com)$

• Not every AI visit carries a referrer. Visits from apps and copied links can arrive as Direct. ChatGPT narrows this gap: OpenAI's publisher FAQ says it adds utm_source=chatgpt.com to referral URLs, so those visits can be attributed even when no referrer is sent.

Citations in Bing Webmaster Tools

Microsoft reports citations directly. AI Performance in Bing Webmaster Tools, in public preview since February 10, 2026, shows how often your pages are cited as sources in Microsoft Copilot, AI-generated summaries in Bing and select partner integrations: total citations, the average number of cited pages per day, a sample of grounding queries, citation counts per URL and the trend over time. On June 16, 2026 Microsoft added four preview features: Intents, which classify grounding queries (informational, commercial, navigational and others); Topics, which group them into themes; Citation Share, your percentage of all citations shown for a grounding query; and Compare, which overlays an earlier period. Microsoft describes Citation Share as an observational metric, not a ranking or a competitive scoreboard.

Bing's classic Search performance report also counts impressions and clicks from chat responses as a traffic source of its own, next to web, news and images, with up to six months of history. On the Google side, clicks from AI Overviews and AI Mode count in Search Console's Performance report under Web search, and since August 31, 2026 a separate generative AI performance report shows impressions from those features by page; AI Overviews in Google Search explains how to read both.

Crawls and fetches in your server logs

Logs show AI engines at work before analytics does. Visits from a search bot (OAI-SearchBot, PerplexityBot, Claude-SearchBot, Bingbot) mean your pages can be indexed. A user-triggered fetch (ChatGPT-User, Perplexity-User, Claude-User) means an assistant read the page to answer someone's question at that moment, whether or not that person then clicked.

grep -oE "OAI-SearchBot|ChatGPT-User|GPTBot|PerplexityBot|Perplexity-User|Claude-SearchBot|Claude-User|ClaudeBot|bingbot" access.log \
  | sort | uniq -c | sort -rn

User agents can be faked, so confirm important patterns against the operators' published IP ranges. Google-Extended never appears in logs, because it has no user agent of its own.

Brand search trends

An answer can name your company without sending a click, and one place that shows up is searches for your name. Search Console's branded queries filter, available to all eligible sites since March 11, 2026, splits Performance data into branded and non-branded queries with an AI-assisted classifier, so expect a few misclassified queries. For a wider view, compare your brand with competitors in Google Trends Explore. Google Trends is a random sample of aggregated, anonymized Google and YouTube searches, so read it as relative interest, not volume.

A fixed set of test questions

Finally, check the answers themselves. Keep a list of 20 to 30 questions your customers actually ask, run them in ChatGPT, Perplexity, Copilot, Gemini and Claude on a fixed schedule, and record which sources each answer cites. Expect variation between runs: Microsoft notes that citation patterns shift with models, freshness signals and user behaviour, and Google says AI Overviews and AI Mode may use different models and show different links. The trend across months is the signal, not any single answer.

Our SEO service covers this measurement work: Search Console and GA4 set up properly, log-file analysis of crawler behaviour, and citations in AI answers tracked alongside classic rankings.

Where to start: an AI search checklist

1. Audit access. Read robots.txt and your CDN or firewall bot settings together, and confirm that Googlebot, Bingbot, OAI-SearchBot, PerplexityBot and Claude-SearchBot can fetch your key pages.
2. Decide on training. Choose GPTBot, ClaudeBot and Google-Extended separately, knowing that Google-Extended also covers Gemini app grounding.
3. Get into Bing. Verify the site in Bing Webmaster Tools, submit the sitemap and turn on IndexNow; Copilot answers from Bing's index.
4. Check rendering. Make sure the main content of each key page is in the initial HTML, not only after JavaScript runs.
5. Rewrite the top of key pages. Put the direct answer in the first paragraph, name the entities, and link the evidence.
6. Publish something original. Data, measurements or first-hand experience that a summary of other pages cannot supply.
7. Match markup to text. Validate structured data and remove anything that describes content not on the page.
8. Set a baseline. Save a GA4 view of the AI Assistant channel, open AI Performance in Bing Webmaster Tools, start a log report and record your test questions, then review them monthly.