How AI Decides Which Pages to Search & Cite

Retrieval augmented era is a framework that determines which content material AI instruments retrieve and cite earlier than producing a solution.

You could perceive RAG as a result of it’s one of many methods ChatGPT, AI Mode and different AI search engines like google and yahoo select which pages get included in its reply.

This information explains how RAG works (in plain English), what makes content material extra prone to be retrieved, and how one can measure your visibility in AI techniques that use RAG with Ahrefs Model Radar.

Retrieval augmented era (RAG) is a way the place an LLM queries an index—like a search engine, data base, or vector database—to seek out further, contextually related data for its response—somewhat than simply defaulting to what it realized throughout coaching.

Massive language fashions are skilled on large datasets, however that coaching has a cutoff date.

Ask an AI mannequin what occurred final week, or what’s in your stay manufacturing database, and also you’re asking it to work from reminiscence with no reference materials in entrance of it.

While you question an AI mannequin on data it doesn’t but have, that’s when it’s almost definitely to go rogue and begin telling you that poison is sweet for you…

That is in any other case generally known as an AI “hallucination”.

RAG offers AI fashions entry to the right, up-to-date materials to keep away from this destiny.

With RAG, LLMs are both supplementing or overriding their inner data—generally known as their “parametric reminiscence”—in an try to present a extra dependable reply.

This course of can also be typically generally known as “grounding”; anchoring the response to particular sources so the mannequin isn’t simply freestyling from its coaching knowledge.

The three phrases map to the three levels of the method:

• Retrieval: The AI mannequin runs a search question to seek out (or retrieve) related content material
• Augmented: It provides that retrieved content material to its enter (augmenting its data)
• Era: It makes use of the question and the retrieved content material to write down (or generate) a response

Most AI instruments use each RAG and skilled data in tandem.

Most AI instruments have not less than two issues working underneath the hood: the bottom mannequin generates language from patterns realized throughout coaching. The retrieval layer goes searching for sources to connect.

Stepping into the bottom mannequin’s data means being a part of its coaching knowledge, and that isn’t one thing you may simply management.

However entering into the retrieval outcomes is, in some ways, an extension of search engine optimization.

Each RAG-powered AI reply follows three steps: search, retrieve, generate.

To know the small print of what seemingly happens at every stage, right here’s what we learn about how ChatGPT retrieves its sources.

Step 1: The AI decides whether or not or to not run a search

Earlier than something will get retrieved, the AI will resolve whether or not it even wants to complement its data with exterior knowledge.

Easy fact-finding queries like “What’s a VPN?” can normally be fielded by the core mannequin primarily based on its current coaching data. No retrieval wanted.

In ChatGPT’s case, a smaller classifier mannequin (a part of the “sonicberry” system in accordance with David McSweeney, who put within the work to seek out out simply how ChatGPT retrieves sources) will run first, assigning likelihood scores to find out whether or not a question wants: no search, a easy search, or a complicated multi-step search.

Different AI instruments will deal with this step in another way, however all of them share the identical underlying logic: not each question triggers a search.

Step 2: The AI runs a search

Each time somebody asks ChatGPT a query that requires extra context, it expands that question into a number of associated queries, then sends them to an exterior search index like Bing or Google to gather outcomes.

That growth course of is named question fan-out (do not forget that for later).

As soon as a collection of pages are collected, on-page search engine optimization components just like the title, meta description/abstract, and URL decide which web page will get learn in full, in accordance with research by AI Knowledgeable Dan Petrovic.

From there, he discovered that sources are shortlisted for scraping primarily based on “relevance, authority, recency, and variety of perspective”.

Step 2: Content material will get damaged into chunks—and the closest match wins

Earlier than it may be absolutely retrieved and served within the response, the scraped internet content material will get damaged into smaller items referred to as chunks.

Consider chunking like tearing a e-book into particular person chapters. The system breaks the web page into items, then asks which piece greatest solutions the query.

ChatGPT converts the search question and every chunk right into a numerical illustration of which means, generally known as an embedding, then measures their cosine similarity—a rating of how semantically shut they are.

The only technique to image this: think about an enormous map the place related concepts sit shut collectively and unrelated concepts are far aside. On this map, “canine” and “pet” can be close to one another. “Canine” and “skateboard” can be on reverse ends.

Embeddings are like GPS coordinates on that map—each piece of textual content will get assigned coordinates primarily based on its which means.

Cosine similarity is the measure of how shut collectively two units of coordinates are.

The AI retrieves the chunks whose coordinates are closest to the fan-out question’s coordinates, and the closest match wins.

That is why particular, clear language helps retrieval—it’s simpler to map to the right vector “coordinate”.

Step 3: Retrieved content material masses into the AI’s working reminiscence—briefly

The matching chunks are loaded into the AI’s context window—its short-term working reminiscence—alongside the consumer’s authentic query.

It synthesizes a solution utilizing each, then it deletes the chunks.

Dan Petrovic tested this directly: he requested an AI mannequin to retrieve data on a widely known individual, then in a follow-up message requested it to recall a particular snippet from its sources. It couldn’t.

The uncooked content material is “purged” the second a response is generated.

RAG and coaching knowledge usually get conflated, however they do very various things.

Coaching knowledge is what builds an AI mannequin’s parametric reminiscence—the inner data I discussed earlier.

This occurs throughout pretraining, when the mannequin learns from an enormous, basic corpus of textual content scraped from the online and different sources (e.g. CommonCrawl), and may occur once more throughout fine-tuning, when a mannequin is additional skilled on a narrower dataset to shift its conduct or data.

Both approach, that data will get baked into the mannequin itself. No lookup wanted—it’s simply a part of what the mannequin “is aware of”.

However you don’t get a say in it. It occurs on the mannequin developer’s schedule, utilizing no matter knowledge they select to coach on. It’s not one thing you may request, goal, or confirm occurred to your content material.

RAG, however, is a course of you have got some management over. The standard, construction, and indexing of your content material straight impacts whether or not it will get retrieved.

Each time a consumer’s question triggers a retrieval step, the mannequin pulls in present data from exterior knowledge while not having to be retrained.

For many industrial AI instruments, that is the mechanism behind most modern solutions they provide you.

	RAG	Coaching knowledge
The way it works	Retrieves exterior content material at question time—by no means turns into a part of the mannequin	Content material is absorbed into the mannequin’s parameters throughout coaching, turning into a part of what it “is aware of” internally
Replace value	Low. Updates the data base and the mannequin’s subsequent reply displays it	Excessive. Solely modifications when the mannequin is retrained—on the developer’s schedule, not yours
One thing you may affect?	Sure. Content material high quality, indexing, and construction all have an effect on whether or not your content material will get retrieved to floor present data, cite you as a supply, or point out your particular model	In a roundabout way. You’ll be able to publish content material and hope it will get crawled however, not like RAG, you may’t optimize one web page and hope to see it cited. What you are able to do is construct a constant model narrative throughout sufficient content material over time that it shapes how future fashions describe you.

The whole lot we cowl on this article—main with definitions, together with entities, Q&A construction, freshness—straight influences the retrieval course of.

That’s the layer you may truly affect by way of content material.

Being cited in an AI response is a win, nevertheless it’s not the identical as being baked into what an AI basically is aware of about your model.

Search Person Optimization knowledgeable Dorron Shapow places it effectively:

Optimizing for retrieval isn’t unsuitable. In techniques that rely closely on stay seek for industrial queries, it could possibly completely affect what will get surfaced. However assuming retrieval visibility is identical as foundational mannequin weighting is the place the technique breaks. One takes weeks. The opposite is the sluggish work of entity coherence—how constantly and clearly your model is known throughout the broader internet—and it takes years.”

Question fan-out is the method that occurs behind the scenes whenever you submit a question to an AI system.

Somewhat than looking out your precise phrases, it breaks your query into a number of associated sub-queries, runs every one individually, and pulls sources from the mixed outcomes.

Say somebody searches “What's going to occur if I swap out common flour for wholemeal flour in a lemon drizzle” in Google, the underlying AI search mannequin wouldn’t simply search that phrase, it may also look up:

• Greatest flour for lemon drizzle
• Baking with wholemeal flour ideas
• How does wholemeal flour have an effect on cake density?

Earlier than synthesizing a solution.

AI does the identical factor robotically on most complicated queries.

Some SEOs have been capable of extract these inner sub-queries straight.

For example, Metehan Yeşilyurt has developed a technique to immediate Google AI Mode into outputting the search queries it used for grounding.

However if you happen to don’t have time to go digging, you can even see the fan-out queries generated by ChatGPT, Grok, and Perplexity within the AI Responses report in Ahrefs Model Radar.

Throughout question fan-out, the AI splits your query into smaller sub-queries, searches all of them directly, combines and re-ranks the results, then merges the pages that do effectively throughout a number of searches into one ultimate checklist.

That checklist is what the AI truly reads to write down your reply.

We’ve simplified the fan-out course of right here for ease of understanding, however for a deeper-dive learn our information: What’s Question Fan-Out? Understanding the Hidden Queries Driving AI Search.

For ChatGPT and different AI search engines like google and yahoo, Retrieval Augmented Era runs on search engine optimization.

In actual fact, many entrepreneurs and SEOs view AI search as a “wrapper” on prime of “conventional” search engines like google and yahoo like Google, since some AI assistants draw so closely from them.

When ChatGPT, Perplexity, or Google AI Overviews have to reply a query, they run precise internet searches ¹ ² ³

Google Gemini and AI Overviews use Google Search. Microsoft Copilot makes use of Bing. ChatGPT pulls from each Google and Bing. Claude makes use of Courageous Search.

Which means the retrieval layer of each main AI device is powered by a conventional search engine.

• Listed content material is the beginning pool. You want your content material to point out up in Google earlier than it exhibits up in AI.
• Search optimized content material will get you cited: Even when search and AI outcomes don’t at all times neatly overlap, each prioritize authoritative, well-structured, well-optimized content material.
• Model mentions in search correlate strongly with AI visibility: AI techniques decide up on how usually and the place your model is referenced throughout the online—search-optimized content material and digital PR straight feeds this ¹

Regardless of some variations, search engine optimization and GEO are intrinsically linked.

In case your content material doesn’t present up in a search index, an AI bot goes to have a tough time discovering it, and if it could possibly’t discover it, it could possibly’t retrieve it.

Comply with these seven greatest observe ideas if you wish to get your content material cited in RAG search.

Ensure that content material is accessible to AI crawlers

Once they exit to fetch content material, many AI crawlers are unable to learn and cite sure pages.

JavaScript content material (like tabs or accordions) or textual content in pictures is usually inaccessible to AI bots.

As a substitute, AI techniques retrieve static HTML content material.

Right here’s what occurs when a web page accommodates JavaScript.

Suganthan Mohanadasan lately tapped into the network files of dozens of ChatGPT conversations, and studied the mannequin’s chain-of-thought course of, the place it describes the way it sources data in layman’s phrases.

For a related B2B SaaS question, ChatGPT positioned official pricing for Ahrefs however struggled to seek out costs for Profound and Peec, reasoning that this data was hidden inside JavaScript.

ChatGPT deferred to third-party sources like G2 since “the official web page is tough to parse and doesn’t present costs”.

The ethical of the story: if you would like your most essential data—like your pricing— to be precisely portrayed in AI search, your content material ought to ideally be served by way of HTML, not JavaScript.

JavaScript isn’t the one technique to lock a crawler out—you additionally have to keep away from blocking AI crawlers (like OAI_SearchBot) in your robots.txt and firewall guidelines if you wish to be cited by way of retrieval ¹ ².

When you use Cloudflare, you may monitor how AI bots are crawling your web site—together with which pages they go to most frequently and which of them they miss—by way of Ahrefs Bot Analytics.

Lead together with your greatest data

AI pays probably the most consideration to the start of your web page, however its consideration drops steadily from there.

In line with Kevin Indig’s study of 1.2 million ChatGPT citations, the primary 30% of a web page’s content material generates 44.2% of all citations.

The center third generates 31.1%, and the underside third: simply 24.7%.

Your most essential data—definitions, key claims, distinctive knowledge—must be on the very prime of your content material.

That is the alternative of the standard “save one of the best for final” method. In content material optimized for AI citations, the punchline goes first.

This is named serving the Backside Line Up Entrance (BLUF).

Reply the question instantly within the first sentence under the subheading—don’t bury the reply two paragraphs in.

This straight mirrors how RAG techniques match content material to queries—but in addition, how customers learn, so that you’re satisfying each beings and bots alike!

This eye-tracking knowledge exhibits readers focus probably the most consideration on the very prime of a web page and scan much less and fewer as they transfer down, so in case your key takeaway is buried in paragraph three, most readers by no means truly see it—therefore, “backside line up entrance”.

Optimize for fan-out subjects

To indicate up within the fan-out outcomes that AI techniques draw on, it’s useful to create subject clusters—the associated questions, definitions, comparisons, and subtopics that AI may seek for whereas making ready a solution.

When you’re searching for hints as to what these sub-topics is likely to be, faucet into “Individuals additionally ask” containers and “Individuals additionally seek for” queries on the backside of Google.

They mirror the most-asked questions and angles round your subject, which are typically just like the queries AI generates in a fan-out.

Optimize your page speed

Slow pages are bad news in any search engine, but in AI search the cost is even steeper.

In his breakdown of how ChatGPT works, search engine optimization Guide David McSweeney notes that ChatGPT seems to fetch grounding pages on a tough timeout of round two seconds: in case your server is sluggish, your web page will get minimize, and even when it responds in time, a excessive time-to-first-byte (TTFB) means your content material will get truncated.

Underneath 1 second TTFB: you’re in all probability advantageous. Your full web page has time to load, get chunked up, and fed to the mannequin.

Over 1 second: you’re playing. The connection may get minimize mid-download—typically so early that solely your <head> tag made it by way of, which means the mannequin by no means even noticed your precise content material.

Pace decides whether or not you make it into the mannequin’s context window at all.

Examine your time-to-first-byte in Website Audit.

1. Head to the Efficiency report
2. Discover the “Time to first byte distribution” chart
3. Click on “Medium: 200–300 ms” to your quick-win optimization alternatives

Then kind by natural site visitors to seek out your most essential content material that will have to be optimized

In case your server is just too sluggish, your web page might by no means make it into an AI reply—however in some circumstances you’ll by no means know, as a result of the customer (on this case, a bot) merely gave up and left.

Jan-Willem Bobbink appears for cases of this by figuring out the HTTP standing code 499 in his server logs.

A 499 standing code means the shopper closed the connection earlier than the server completed responding.

That is one other clear sign that your web site is just too sluggish for AI retrieval.

Create deep, entity-led content material

The content material that will get cited most frequently by way of RAG search accommodates roughly 20.6% entity density.

Which means, 20.6% of its phrases are correct nouns—named instruments, manufacturers, individuals, corporations, research—in comparison with 5-8% in “common” content material.

An entity is any particular named factor. For instance, “An search engine optimization device” shouldn’t be an entity— however “Ahrefs” is.

The extra named entities you embody, the extra anchor factors your content material has on the which means map—making it retrievable for a broader vary of associated queries.

However you’re not going to win citations by randomly “entity stuffing”. Your content material, and its entities, have to be related to the consumer’s question.

Right here’s another excuse entities matter.

Fan-out queries usually use a “synonym cloud” approach to steer retrieval in the direction of particular angles and entities, and finally higher match the intent of the consumer’s authentic question.

For instance, ChatGPT’s frontier mannequin might remodel a question like “What are the ten greatest trainers?” into synonym-rich fan-out queries like:

• greatest trainers 2026
• evaluations trainers
• prime picks
• awards

To nudge the embedding towards “better of” intent, as seen under by way of Model Radar.

So what does this imply to your content material?

Effectively, to paraphrase David McSweeney: Generic pages that point out the whole lot rating okay throughout the board.

However specialised pages that go deep on one angle win that angle outright.

Getting cited is subsequently about anchoring your content material to particular entities.

Right here’s a sensible technique to enrich your content material with entities: undergo your again catalog and substitute generics with specifics.

Change:

• “A search engine” → “Google”
• “Analysis suggests” → “A 2024 examine from Waseda College discovered”
• “An AI assistant” → “ChatGPT” or “Perplexity”

You’ll be able to confirm your work utilizing Google’s Natural Language API.

The free demo model exhibits you each entity Google detected in your web page, and the class it assigned your content material to.

When you pay for full entry, you’ll additionally get the salience rating—a worth for a way outstanding and essential Google thinks an entity is to your web page.

Run the API in your web page, then run it on the top-ranking web page to your goal key phrase.

The hole between these two outputs offers you your entity optimization guidelines:

• Entity crossover
• Entity gaps
• Salience scores (increased when the subject is called earlier and extra prominently)
• Class crossover
• Class gaps

Alternatively, run your draft by way of Ahrefs’ AI Content material Helper.

It grades your content material in opposition to your prime opponents to your goal key phrase and highlights the subjects they cowl that you just’re lacking—helpful for catching subject gaps which may make you invisible in fan-out outcomes.

Add data achieve—say one thing the mannequin doesn’t already know

Entity protection will get you retrieved, however there’s one thing that comes earlier than that: does your content material even qualify for retrieval within the first place?

A leaked Claude system prompt revealed that AI techniques like Claude have a never_search command for queries about “timeless or steady” data.

Claude solutions never_search questions from coaching knowledge alone, and doesn’t go searching for exterior URLs to cite.

Progress Advisor Gaetano DiNardi thinks different LLMs are seemingly following the identical logic. In his phrases:

“the worth of publishing pages on generalized data is zero.”

That is the data achieve drawback.

Consider the whole lot a mannequin already is aware of because the overculture—the averaged-out, consensus model of a subject that’s been listed 1000’s of instances.

In case your content material solely restates it, you’re redundant from the RAG framework—an AI mannequin has nothing to realize from citing you.

What it does cite is content material that provides one thing new: proprietary knowledge, a named concept, a particular discovering from a examine, a conclusion the mannequin can’t synthesize from its current data base.

OpenAI researcher Karthik Narasimhan revealed a paper on Generative Engine Optimization that gives additional proof of this.

Together with friends at Princeton College, he studied which strategies are almost definitely to spice up visibility in RAG AI techniques like Perplexity.

Their findings revealed that web sites that includes distinctive data like quotes and statistics had been mostly referenced; seeing 30-40% visibility uplift in AI responses.

LLMO methodology examined	Place-adjusted phrase depend (visibility) 👇	Subjective impression (relevance, click on potential)
Quotes	27.2	24.7
Statistics	25.2	23.7
Fluency	24.7	21.9
Citing sources	24.6	21.9
Technical phrases	22.7	21.4
Straightforward-to-understand	22	20.5
Authoritative	21.3	22.9
Distinctive phrases	20.5	20.4
No optimization	19.3	19.3
Key phrase stuffing	17.7	20.2

Kevin Indig additionally discovered that date and quantity are the entity varieties that predict ChatGPT citations most.

And Eric Lancheres studied 150 rating pages and located the most important rating predictor was their variety of unique data points.

Having your content material retrieved is a matter of surfacing recent data and distinctive knowledge, not chorusing what different pages have already coated.

Embrace a question-and-answer construction

Content material structured as query → quick reply is cited twice as often as content material that doesn’t observe this conference (18% vs. 8.9%), in accordance with Kevin Indig’s knowledge.

That is one more instance of BLUF in play.

AI fashions attempt to match consumer queries (nearly at all times a query) to a bit that solutions it.

Within the phrases of Suganthan Mohanadansan:

“Citations bind to a particular sentence, not the entire reply, so being topically related isn’t sufficient, it’s a must to be one of the best help for a exact declare.”

Formatting your content material as a Q&A will help AI fashions like ChatGPT make a direct, unambiguous match.

Mohanadasan additionally discovered that ChatGPT deduplicates outcomes by area, so 20 skinny pages in your web site don’t add as much as 20 possibilities at quotation.

ChatGPT selects the one web page that greatest matches the consumer’s preliminary question and fan-out subqueries.

Put your strongest reply on that web page, not unfold throughout all 20.

Maintain content material recent

RAG search techniques have a desire for present content material.

We ran a examine of 17 million citations, and located that AI assistants constantly want to quote more energizing content material than search engines like google and yahoo.

URLs cited by AI assistants are 25.7% more energizing on common than URLs in commonplace natural SERPs—and ChatGPT and Perplexity truly order their citations from latest to oldest.

However don’t simply take our phrase for it. Freshness is a confirmed, documented sign in AI retrieval.

Metehan Yeşilyurt’s analysis confirmed this. He found that ChatGPT has a configuration setting called use_freshness_scoring_profile: true, which bakes in a scientific recency bias.

So, your content material has a significantly better likelihood of being retrieved and ultimately cited if you happen to replace your key pages often.

Even minor updates can reset the freshness sign. Refresh statistics and examples yearly and add a visual “final up to date” date.

Optimizing your content material for RAG is important, however that you must know if it’s working.

Ahrefs Model Radar was constructed to assist manufacturers monitor their visibility in retrieval augmented AI outcomes.

Right here’s how I counsel utilizing it to enhance your visibility in RAG.

Monitor your baseline visibility

Earlier than altering something, discover out the place you truly stand.

Search your model in Model Radar to see how usually you’re showing in AI solutions to your goal subjects, and which platforms are citing you.

If mentions are low or absent, see who’s being cited as a substitute.

Discover out which AI platforms are citing you (and which aren’t)

Completely different AI platforms have completely different retrieval architectures with completely different biases towards freshness, authority, and construction.

Model Radar’s platform breakdown can reveal gaps like “AI Mode cites us often, however we lack visibility in Perplexity.”

In case your web site performs badly on just one platform, the problem is probably going with how that platform evaluates it—not the content material itself.

For instance, if a web page ranked effectively on Google however not on Bing, we’d see that as a Bing-specific sign (like hyperlinks, entities, or indexing) somewhat than the web page being low high quality general—the identical is true of AI visibility.

Uncover which queries are triggering your citations

Seeing the precise queries that result in quotation tells you what’s working, and flags associated queries the place you’re not showing but.

Due to question fan-out, you might already be getting cited for queries you’d by no means have thought to focus on.

Model Radar’s database accommodates tens of millions of current queries, which means you may come across new content material alternatives you wouldn’t in any other case know existed.

Monitor whether or not content material updates change your quotation price

When you’ve made modifications to optimize your content material for retrieval—making use of BLUF, focusing on fan-out queries, incorporating statistics—monitor Model Radar to see whether or not your citations develop within the following weeks.

This allows you to construct a suggestions loop: optimize → publish → measure → iterate.

The identical type of methodology that works for monitoring natural rankings additionally applies to AI quotation monitoring.

Benchmark in opposition to opponents

Discover out which of your opponents is being constantly cited by AI for queries you care about, then analyze the construction and content material of their most-cited pages.

Simply add a Your model: Not talked about and Your model: Discovered however not cited filter to an AI Responses or Cited Pages report in Model Radar.

This can present you the subjects and third-party discussions your model tends to be unnoticed of.

Then it’s only a case of reverse-engineering your opponents’ strikes to shut the hole.

RAG is the bridge between search and AI. It follows predictable guidelines, selling pages it could possibly entry, fetch shortly, and topic-match straight to present the very best reply.

Monitor your AI visibility with Ahrefs Brand Radar to see whether your content is showing up across ChatGPT, Perplexity, Google AI Overviews, and the other tools your audience actually uses.

Got questions? Ping me on LinkedIn.