How AI Answer Engines Choose Which Sources to Cite

A system model, not a secret formula

A visible citation is the final interface step of a longer system. Separate crawling and indexing, retrieval, reranking, answer synthesis, and citation selection. This is a diagnostic model, not a published blueprint for every product. It helps a team ask whether a page was inaccessible, hard to match to a question, less useful than another source, or simply not selected for display. Without controlled evidence, it cannot prove why one commercial engine chose a specific source.

Crawling and indexing

Crawling is a bot fetching a page; indexing is processing material so it can later be found. The stages are distinct. Google says indexing and serving are not guaranteed even when standard requirements are met. OpenAI documents that OAI-SearchBot can be stopped by robots rules, WAFs, CAPTCHAs, authentication, and rate limits. A public successful page is a prerequisite for some systems, not evidence that it entered an answer index or will be visible to a particular user.

Retrieval

Retrieval-augmented generation retrieves material relevant to a query and provides it to a model as it produces an answer. Published research describes several retrievers and augmentation designs. A production answer product may combine web search, proprietary indexes, tool calls, freshness thresholds, metadata filters, and safety systems. A strong source can therefore be absent for a particular wording, language, market, date, or user context. Retrieval is candidate selection; it is not endorsement of the source or a guarantee of a displayed citation.

Reranking

Reranking can reorder candidate passages or documents using a later, stronger relevance estimate. Research evaluates reranking methods, but it does not publish the weights of consumer answer engines. Inference: if a direct, current primary page repeatedly appears for a narrow query, answer fit and evidence quality are sensible things to inspect. It is not justified to declare that an engine has awarded that page a permanent authority score. Repeated observations can describe a pattern; they do not expose the full pipeline.

Answer synthesis

A generator can summarize, compare, qualify, or decline based on retrieved context and instructions. Source choice and answer wording are related but not identical. A model may use several sources for one statement, omit relevant detail, or frame a source differently from its original text. Perplexity’s crawler documentation distinguishes PerplexityBot, which it says is designed to surface and link websites in Perplexity search results, from Perplexity-User, which may visit a page for a user request and include a link in its response. That documents stated crawler roles, not a commitment to cite a named publisher, page, or sentence in every answer.

Citation selection

A citation may point to a source used in generation, a retrieved result, a claim-supporting passage, or an interface-selected set of links. This matters because a citation is not a complete audit trail. The ALCE citation-evaluation paper separates answer correctness from citation quality and evaluates whether cited passages support generated statements. Its benchmark is not an audit of any named commercial answer engine, but it supports the narrower practice of inspecting the linked page beside the answer claim. If the page does not support the sentence, record a citation-quality issue rather than treating the link as proof that the claim is accurate.

Engine-specific opacity

Google offers webmaster guidance, not a public formula for citations. OpenAI offers crawler access guidance, not a public recipe for ChatGPT source selection. Perplexity documents crawler access and user-request fetching, not a universal publisher-selection rule. Commercial systems can change indexes, policies, interfaces, and models. A report that claims certainty about source selection should present controlled evidence sufficient for that claim. Otherwise the responsible labels are observation, hypothesis, or inference—not explanation of a hidden algorithm.

Responsible publisher actions

Publish a direct answer, state scope and limitations, attach first-party or original evidence, date volatile facts, and remove avoidable access barriers. Save defined query runs with conditions, answer text, citations, and limitations. Compare source pages, not just domains. If an answer cites a page that does not support the nearby claim, clarify your own page if warranted and preserve the mismatch in the audit. The goal is not reverse engineering from a handful of outputs; it is a record that distinguishes access issues, content ambiguity, changing prompts, and uncertainty.

Absence is not a diagnosis

When a page is absent from one answer, several explanations remain open. The product may not have fetched it, may not have indexed it, may have retrieved other material for that wording, may have preferred another passage during reranking, may have generated an answer without displaying the source, or may have changed behavior between runs. The user’s language, location, account state, date, freshness needs, safety rules, and interface can also matter. The observable outcome is real; the causal story is usually not.

This is why a simple domain-level scoreboard is a weak debugging tool. A domain can be visible because one dated press page is linked, while the product guide that would help a reader remains absent. Conversely, a strong page can be used as background context without appearing in a visible source list. Research on retrieval-augmented generation establishes that external retrieval can support generated output; it does not show which commercial product retrieved, used, or displayed a particular page on a particular day. Treat any claim that bridges those gaps as an inference and state the observations that motivated it.

Do not turn repeated retries into evidence. If a team keeps changing a prompt until a desired source appears, it has selected an outcome rather than sampled a system. Save the original wording and conditions, define in advance how many runs or engines are in scope, and retain both positive and negative observations. A later change may justify a follow-up investigation, but not a retrospective assertion that the first result revealed a permanent source-selection rule.

A diagnostic sequence that preserves uncertainty

Start with the page itself. Confirm that a non-authenticated request receives the intended content, that robots and preview controls say what the organization intends, and that the canonical URL, visible text, and structured data agree. Google explains that eligibility for its AI features depends on being indexed and eligible for a Search snippet, while also warning that crawling, indexing, and serving are not guaranteed. That is a useful documented boundary: access is necessary for the described feature but is not proof of appearance.

Next, evaluate answer fit without pretending to know a ranking weight. Put the user’s question beside the page. Does the page answer it early, name conditions, date fast-changing facts, and link the primary evidence? Compare it with any source actually displayed: which claim does each support, and where do their scopes differ? Then record a hypothesis in operational language, such as “the page may be hard to use for this narrow implementation question because its requirement is buried,” rather than “the engine penalized our domain.” The former suggests a reader-benefiting edit and remains falsifiable.

Finally, re-observe a fixed core sample after the edit and report the full context. A changed output can be noted; it cannot by itself establish that the edit caused the change. Perplexity’s documentation can establish that a product offers citations in relevant configurations, and platform crawler documentation can establish access controls, but neither replaces a transparent experimental design. The responsible goal is a clearer page and a better evidence record, not a fabricated map of an opaque pipeline.

Keep a distinction between evidence and explanation in every report. Evidence is a saved result, a fetch test, a source page, or a documented control. Explanation is the limited hypothesis that connects those facts. When the explanation cannot be tested, record it as an open question instead of converting it into an algorithm claim.

Related reading: The Small-Team AI Visibility Audit.