Validating the RAG Reply Earlier than the Consumer Sees It: Spans, Quotes, and the Suggestions Loop

A typed answer is not a checked answer. Structured output is the start of validation, not the end. The model still cites lines outside the input range, paraphrases quotes it swore were verbatim, sets complete_answer_found=True on a partial answer, and returns shapes the brief didn’t ask for.

The fix is post-generation validation. The validator takes the parsed question alongside the answer so it can flag shape mismatches against what was requested. Three checks combine:

• Shape: the returned answer must be an instance of the schema the registry picked for the brief, with items populated when answer_found=True.
• Evidence: every Span must reference a real line range, every quote must be a substring of the cited lines after a tolerant whitespace + bibliographic-refs normalization.
• Format: ISO 8601 for dates, ISO 4217 for currencies, etc.

Below is the implementation. Notice the per-item, per-span loop: each problem is reported individually so the failure mode is visible at a glance (a wrong span on item 2 doesn’t hide a wrong currency on item 3).

def validate_answer(answer: AnswerBase, line_df: pd.DataFrame,
                    parsed_q: ParsedQuestion | None = None) -> list[str]:
    errors: list[str] = []
    valid_lines = set(line_df["overall_line_num"].values)

    if parsed_q is not None:
        ExpectedSchema = ANSWER_REGISTRY[parsed_q.expected_answer_shape]
        if not isinstance(answer, ExpectedSchema):
            errors.append(f"shape mismatch: {ExpectedSchema.__name__} expected")

    if bool(answer.items) != answer.answer_found:
        errors.append(f"answer_found mismatch len(items)={len(answer.items)}")

    for i, item in enumerate(answer.items):
        if not item.spans:
            errors.append(f"item[{i}] has no spans")
        for j, sp in enumerate(item.spans):
            if sp.line_start not in valid_lines:
                errors.append(f"line_start {sp.line_start} not in input")
            if sp.line_end < sp.line_start:
                errors.append(f"line_end before line_start")
            if sp.quote:
                cited = _join_cited_lines(line_df, sp)
                if _normalize(sp.quote) not in _normalize(cited):
                    errors.append(f"quote not verbatim in cited lines")

        if (date := getattr(item, "date", None)) is not None:
            if not re.fullmatch(r"d{4}-d{2}-d{2}", date.iso):
                errors.append(f"date.iso not YYYY-MM-DD")
        if (amt := getattr(item, "amount", None)) is not None:
            if not re.fullmatch(r"[A-Z]{3}", amt.currency):
                errors.append(f"currency not ISO 4217")

        if answer.extraction_method == "verbatim" and not any(sp.quote for sp in item.spans):
            errors.append(f"verbatim but no quote")
    return errors

One axis the loop doesn’t cover: fields against each other. Every check above reads one field at a time. The next category is cross-field: constraints where two fields have to agree. A contract’s start_date must precede its end_date; the line items of an invoice must sum to its stated total. The schema hints at where this bites: a value with extraction_method="computed" (a total the model added up rather than read off the page) is exactly what should be reconciled against its parts. Declare the constraints on the schema and run them in the same pass, appending to the same errors list: end_date 2024-12-01 precedes start_date 2025-01-01, line items sum to 350, stated total is 400. A mismatch is not a parsing artifact, it is a number the user must not trust. (Consistency across different documents is a corpus-scale problem, left to a later article.)

When validation fails, the pipeline has options:

• Retry with a stricter prompt or a different model.
• Flag for review: return the answer with a warning.
• Reject: refuse to return the answer to the user.

Which option you pick depends on the context. For low-stakes interactive use, retry. For audit-critical paths (legal, compliance, financial), reject and require human review. The retry path matters beyond the individual call: rejected outputs never reach the user, so the delivered paraphrase rate drops to whatever survives the substring check, even if the model’s raw rate is unchanged.

To see all three options at work, the demo below builds a deliberately bad AmountAnswer that exercises four defects at once: shape mismatch (the running brief asked for list = TextAnswer/ListAnswer), span outside input range, currency that isn’t ISO 4217, verbatim claim with no quote on any span. The validator reports each. In a production pipeline this answer would be rejected; in a lower-stakes context it would be retried with a stricter prompt.

In enterprise RAG, returning the wrong answer is worse than returning no answer. A wrong answer in an insurance contract can mislead a claims handler. A wrong answer in a regulatory filing can result in a compliance violation. A wrong answer about a non-compete clause can cost a deal. A “not found” forces the user to look further, which is the right behavior when the system doesn’t have the answer.

Three things make “not found” work properly:

1. The schema permits it cleanly: items=[], answer_found=False, no fake citations and no fake values. An empty items list is the structured way to say “I don’t know”: no text to misread, no spans to chase, no value to render.
2. The system prompt requires it: Explicit instruction in BASE: “If the passages do not contain the requested answer, return items=[], answer_found=False, and explain in caveats what was or wasn’t found.” This isn’t optional. Without it, models default to producing something even when nothing supports it.
3. Downstream code distinguishes NA from a real answer. Checking len(answer.items) == 0 is enough. The application doesn’t render an empty answer as a real one. It says “the document doesn’t appear to contain this information” clearly, without dressing it up.

The hardest part isn’t the technical implementation. It’s the cultural willingness to accept that “not found” is a correct outcome. Teams under pressure to produce “smart” answers often optimize for low NA rates, which is exactly the wrong incentive. A high NA rate on questions whose answers aren’t in the corpus is a sign the system is honest. A low NA rate on the same questions is a sign it’s hallucinating.

# NA path: off-topic question on the Attention paper. The BASE rule triggers cleanly:
# items=[], answer_found=False, caveats explain what was not found. No fabricated number.
missing_q = ParsedQuestion(
    original_question="What is the company's annual revenue in 2024?",
    keywords=[Keyword(text="revenue"), Keyword(text="2024")],
    expected_answer_shape="amount",
    retrieval=RetrievalQuery(main_query="revenue 2024"),
    generation=GenerationBrief(
        original_question="What is the company's annual revenue in 2024?",
        format_constraint={"currency": "USD"},
    ),
)
missing_result = generate(missing_q, filtered_line_df, client)
a = missing_result.answer
print("schema used :", missing_result.meta["schema_used"])
print("items       :", a.items)
print("answer_found:", a.answer_found)
print("caveats     :", a.caveats)
print("validation  :", validate_answer(a, line_df_overall, missing_q))

A specific case sits between “answer found” and “not found”: the question expects a typed shape, but the document only carries a softer form. “What’s the premium?” expects an Amount(value, currency). The document says “pricing is negotiated case-by-case”. There’s information, just not in the shape that was asked for. Three options to handle this, each with a real cost:

Option 1, silent downgrade to text: Set the schema to TextAnswer, put the prose answer in items[0].text, drop the original expected_answer_shape. The user gets something. Downstream code that expects an Amount breaks (the application was rendering a price tag and now gets a sentence). The mismatch is invisible: nothing in the output flags that the shape was downgraded. The bug shows up in production when a chart shows a string where a number should be.

Option 2, explicit NA with caveats: Keep the requested schema (e.g. AmountAnswer), set items=[], answer_found=False, complete_answer_found=False. Put the prose explanation in caveats: “Document mentions pricing is negotiated case-by-case but does not give a specific amount.” Downstream code sees answer_found=False and renders the “not found” path, showing the caveat to the user. The user knows the system tried, knows what was found, and knows what’s missing. No silent breakage, no fake value.

Option 3, force the shape with a default value. Set items[0].amount = Amount(value=0, currency="EUR"). Convenient for downstream code (always typed), catastrophic for everything else. The validator can’t tell a real 0 EUR (a free service) from a “we couldn’t extract a value” 0 EUR. Audit trails get polluted with phantom zeros. Don’t.

The series picks Option 2: The cost is that downstream code must handle answer_found=False explicitly. The benefit is that no information is silently lost: the validator can flag the mismatch (section 1.1), the audit log carries the caveat, and the user never sees a fabricated number. The schema’s answer_found and complete_answer_found fields exist precisely so the pipeline can branch cleanly on these cases without inferring intent from missing data.

The dispatcher’s shape fragments (Article 8B, prompt assembly) reinforce this rule in the system prompt: “If the document gives the value without a currency, set answer_found=False and add a caveat. Do NOT guess.” The model is told what to do with the mismatch; the schema gives it a way to say so; the validator catches the cases where it didn’t.

# Shape mismatch: ask for a date the paper does NOT carry. Expect items=[],
# answer_found=False, explanatory caveats. Option 2 from the shape-mismatch section:
# no silent downgrade, no phantom date. The DateAnswer schema is still returned.
date_q = ParsedQuestion(
    original_question="On what calendar date was multi-head attention with 8 heads first deployed in production at Google?",
    keywords=[Keyword(text="multi-head attention"), Keyword(text="deployment")],
    expected_answer_shape="date",
    retrieval=RetrievalQuery(main_query="multi-head attention deployment date"),
    generation=GenerationBrief(
        original_question="On what calendar date was multi-head attention with 8 heads first deployed in production at Google?",
        format_constraint={"date_format": "YYYY-MM-DD"},
    ),
)
date_result = generate(date_q, filtered_line_df, client)
a = date_result.answer
print("schema used          :", date_result.meta["schema_used"])
print("items                :", a.items)
print("answer_found         :", a.answer_found)
print("complete_answer_found:", a.complete_answer_found)
print("caveats              :", a.caveats)
print("validation           :", validate_answer(a, line_df_overall, date_q))

Line numbers scroll the document. A rectangle lands on the answer. The bbox-join is what turns a citation into something the viewer can paint on the PDF. It is the split the whole brick runs on: the model’s output is pure structured data (line numbers here), and every displayed element, the quoted text and the box, is recovered afterward from the source tables. The schema work of Article 8A (the answer contract) stops at line numbers, and those come in two flavours across this brick: the rich contract’s Span uses the global line_start / line_end, while the minimal AnswerWithEvidence carries the page-scoped start_page_num / start_line_num / end_page_num / end_line_num. Either form tells the pipeline which lines the model leaned on. Neither is enough for the UI to show them. The join below maps whichever it gets back to line_df rows. A viewer next to a PDF wants a Box of (page, x0, y0, x1, y1) it can paint as a yellow overlay.

The bridge is a post-generation join with line_df. The parser already emitted (x0, y0, x1, y1) per line at parsing time; the LLM picks line numbers here; the join is one DataFrame filter:

In: line_df (cached from parsing) + a citation (page, line_start, line_end). Out: a list of Box(page, x0, y0, x1, y1) ready for the viewer overlay.

def bboxes_for_citation(line_df, *, page, line_start, line_end,
                        mode="union", image_df=None):
    matched = line_df[
        (line_df["page_num"] == page)
        & (line_df["line_num"] >= line_start)
        & (line_df["line_num"] <= line_end)
    ]
    if matched.empty:
        return []
    if mode == "union":
        return [{
            "page": page,
            "x0": float(matched["x0"].min()),
            "y0": float(matched["y0"].min()),
            "x1": float(matched["x1"].max()),
            "y1": float(matched["y1"].max()),
        }]
    return [
        {"page": page, "x0": float(r["x0"]), "y0": float(r["y0"]),
         "x1": float(r["x1"]), "y1": float(r["y1"])}
        for _, r in matched.sort_values("line_num").iterrows()
    ]

Two modes worth shipping:

• union (default) gives the envelope of all cited lines on the page: one box, fits 95% of citations where the answer span is contiguous and single-column. Cheap on the wire, easy on the viewer.
• per_line gives one box per line, useful when the citation crosses a column break or wraps around an embedded image. The envelope would cover the gap, the per-line list does not.

The optional image_df argument is what makes figure citations work. The parser produces an image_df alongside the line_df, one row per embedded image with the same (page, x0, y0, x1, y1) shape. When the citation’s line range overlaps an image vertically on the same page, that image’s bbox is merged into the result: in union mode the envelope stretches to cover the figure plus its caption; in per_line mode the image is appended as an extra box. The reader who clicks on a citation that points at “see Figure 3” lands on the figure, not just on the caption line.

The function lives in src/docintel/generation/citation_bbox.py. The cross-document variant (corpus_pdf_qa, a follow-up pipeline) does the same join, but on the original document’s line_df, not on the aggregated context the LLM saw. Synthetic page numbers from aggregation are remapped to (document_id, original_page) before the lookup, so the bbox points at the source PDF, not the prompt’s reading order.

Why this earns a section and not a footnote: the whole chain the generation articles built (schema, dispatcher, validation, feedback) is invisible to the user unless the viewer can land a rectangle on the page. The join is twenty lines, and those twenty lines are what turn a structured answer into something the reader can verify with their eye.

Most RAG diagrams end at generate → respond. The richer schema makes generation talk back: the answer carries fields that tell the pipeline to broaden, re-parse, ask, or ship. The line becomes a loop.

Each self-assessment field on AnswerBase triggers a specific pipeline path. Same-run paths react immediately to fix this question’s answer; long-term paths accumulate knowledge for future questions on the same concept.

Same-run signals (act on THIS question before returning):

• complete_answer_found = False → expand retrieval scope and retry. The canonical same-run trigger. The answer we got is partial (1 of 5 expected exclusions; one half of a comparison missing). Broaden the keyword set (using llm_discovered_keywords if available, deduped against the original keywords: see code below) and call the generator again. Cost: one extra round-trip. Benefit: full coverage on multi-section questions.
• context_structured = False → re-parse the source pages with a different method (Camelot, Docling, vision-language model), then re-retrieve. The model has detected an upstream parsing failure that the parser didn’t notice.
• conflicting_evidence = True → don’t return the answer; show the conflict to the user “two passages disagree on this date”.
• suggested_clarification set → don’t answer; ask the user one targeted question back. Cheaper than answering wrong.

Long-term signal (accumulate, don’t react now):

• llm_discovered_keywords → enrich the concept’s keyword table (the expert dictionary built at question parsing time, Source B). The model spotted terms like “declaration page” or “schedule of benefits” that the original brief didn’t carry. For THIS run, re-retrieving with the same model rarely helps. For the next question on the same concept, those terms should already be in the dictionary so retrieval finds the right passages from the start. The pipeline persists discovered keywords to a concept-keyed table; the next parse_question call reads them back into the brief. Dedup against the existing entry is mandatory (the model often re-suggests known terms). The table grows with the project, not the question count.

Same-run retry and long-term enrichment can interact: if complete_answer_found=False and discovered_keywords are present and at least one of them is new, the retry can use the deduped union of original + discovered keywords for broader retrieval in this run. But the trigger for retry is the completeness signal, not the keyword discovery: discovered keywords on a complete answer go to the long-term table and nowhere else.

The model isn’t just producing an answer; it’s diagnosing the pipeline and proposing fixes. The cost is one extra round-trip. The benefit is a system that recovers from upstream limits: retrieval that missed on a vocabulary mismatch, a parser that missed on a table, a context too narrow. People loosely call this “agentic”; it’s just feedback control.

The feedback loop that closes back to the parsing brick is the one most pipelines skip. Generation becomes a quality detector for parsing. The result: silent quality loss when documents have edge cases.

CONCEPT_KEYWORD_TABLE: dict[str, set[str]] = {}

def persist_discovered_keywords(parsed_q, result, concept_key=None) -> set[str]:
    key = concept_key or (parsed_q.keywords[0].text if parsed_q.keywords else "_default")
    known = CONCEPT_KEYWORD_TABLE.setdefault(key, set())
    discovered = {t.lower() for t in result.answer.llm_discovered_keywords}
    new_terms = discovered - known
    known.update(new_terms)
    return new_terms

def maybe_retry_when_incomplete(result, parsed_q, page_df, line_df, generate_fn):
    if result.answer.complete_answer_found and result.answer.confidence > 0.8:
        return None
    existing = {k.text.lower() for k in parsed_q.keywords}
    new_terms = [t for t in result.answer.llm_discovered_keywords
                 if t.lower() not in existing]
    if not new_terms:
        return None
    enriched = [k.text for k in parsed_q.keywords] + new_terms
    new_parsed = parsed_q.model_copy(update={"keywords": [Keyword(text=t) for t in enriched]})
    _, new_candidates = retrieve_pages(page_df, line_df, enriched, top_k=3)
    return generate_fn(new_parsed, new_candidates, client)

A real RAG system rarely talks to one model. You might use OpenAI in production, Anthropic for evaluation, Mistral or a self-hosted model for internal sensitive data, and Ollama locally for development. Each has its own SDK, its own quirks, its own failure modes. Spreading provider-specific calls across the codebase is how you get into trouble.

The pattern is to wrap all providers behind a single function :

from typing import TypeVar
from pydantic import BaseModel
T = TypeVar("T", bound=BaseModel)
def get_completion(
    system: str, user: str, schema: type[T],
    provider: str = "openai", model: str | None = None, temperature: float = 0.0,
) -> T:
    """Single entry point for any LLM call. Returns a parsed Pydantic instance."""
    if provider == "openai":    return _openai_call(system, user, schema, model, temperature)
    if provider == "anthropic": return _anthropic_call(system, user, schema, model, temperature)
    if provider == "ollama":    return _ollama_call(system, user, schema, model, temperature)
    if provider == "mistral":   return _mistral_call(system, user, schema, model, temperature)
    raise ValueError(f"Unknown provider: {provider}")

Every other module in the codebase calls get_completion. Nobody imports openai directly. This pays in three ways:

• Provider swap is one variable: Changing from OpenAI to Mistral is changing provider="openai" to provider="mistral". Nothing else moves. A/B testing between providers is trivial.
• Local development matches production. Development uses Ollama locally with Phi-4 or Mistral-Nemo; production uses Azure OpenAI. Same code path, different config. The bugs you find locally are the bugs you’d have found in prod.
• Fallback logic lives in one place: When OpenAI rate-limits you, the wrapper transparently retries with Anthropic. When a self-hosted model is down, fall back to a hosted one. The application code doesn’t know.

The wrapper also normalizes quirks. Some providers handle structured output natively (OpenAI’s responses.parse), some need JSON schema in the prompt (most), some need grammar files (llama.cpp). The wrapper hides these differences behind a uniform schema: type[T] parameter. Inside the wrapper, the right machinery is invoked for the right provider. In this notebook, generate() talks to OpenAI directly to keep the dispatcher demo focused; in production, swap client.responses.parse for get_completion(...).

A short list of generation anti-patterns I see repeatedly. None of them is fatal in isolation. Together they ruin reliability:

• The schema is too loose: A bare answer: str with no convention for the NA value. The model invents its own (“not available”, “n/a”, “unknown”) and downstream code can’t reliably detect them. Pin the NA representation: empty items list, answer_found=False.
• Validation is skipped in production: Validation is treated as a development-only concern. In production, raw model output flows directly to the user. The day a model paraphrases a quote, you find out from the user, not from your logs.
• Confidence scores taken at face value. The model says 0.95, the application shows it as “high confidence”. But the model is poorly calibrated. Treat confidence as a triage signal, not a guarantee.
• The system prompt is treated as boilerplate. “Answer based on the context” is what most teams write, and it’s barely a prompt. The system prompt is where you encode every constraint: NA behavior, citation requirement, paraphrase prohibition, format requirements. It deserves the same care as the schema.
• Temperature greater than 0: Reproducibility lost. The same question on the same documents gives different answers each call.
• No token budgeting: Prompts grow as the candidate set grows, and approximations like “about 4 characters per token” mask it. One day, a request hits the limit and silently truncates. Quality drops without anyone noticing.
• No feedback fields: The pipeline never knows when retrieval drifted, when parsing was bad, when the context was incomplete. Open loop. Quality plateaus and stays there.
• Open-source models tested with thinking modes enabled. A reasoning model is great for math problems and bad for JSON. If your output is structured, disable reasoning or use a non-thinking model.

A team runs RAG over insurance contracts with a basic schema: answer, line_start, line_end. It passes the test questions. In production, the answers are “almost right but not quite”: paraphrasing drift, missing conditions, the occasional confident-but-wrong value. The team switches models, fine-tunes, swaps embedding providers. None of it helps.

The fix turns out to be in the schema. They add quotes (verbatim snippets), caveats (limitations), and complete_answer_found (whether the passages held the whole answer). The first time they validate quotes against source lines, they find that 12% of “high-confidence” answers contain quotes that don’t appear in the cited passages. The model was paraphrasing while pretending to quote. The issue lives in the architecture, not in the model: the schema didn’t ask for verbatim, so the model returned a paraphrase.

They add one rule: every quote must be a substring of the cited lines, or the answer is rejected. The paraphrase rate drops from 12% to 0.3%. Nothing about the model changed. It just knew it would be checked.

A few weeks later, a different problem shows up. About 8% of answers come back with complete_answer_found=False, but the system was returning them anyway. They add a feedback loop: on complete_answer_found=False, the system re-retrieves with broader scope and tries again. Recall on multi-section answers (exclusions, conditions, lists) jumps noticeably.

The interesting observation, six months in: most of the gains came from the schema, not from the model. They never fine-tuned. They never switched providers. They added structure, validated it, and looped feedback into the pipeline. The model had been capable all along; it just hadn’t been asked properly.

In this brick, generation is controlled execution: a typed function that consumes a ParsedQuestion and produces a structured object whose every field has a job. The user sees the answer, the citations, the caveats; the pipeline sees the feedback fields (discovered keywords, context completeness, parsing quality, conflicting evidence) and decides whether to retry, expand, re-parse, or return “not found”. Four choices keep it auditable: the schema is the contract, the prompt is composed from fragments instead of picked as a whole template, the answer is validated before anyone reads it, and the model provider is a switchable resource.

This closes Part II. The next article opens Part III with the four upgraded bricks wired together into one pipeline.

Constrained decoding guarantees the shape of the output, never its truth: the launch post’s “100% schema adherence” is exactly where this article starts, not where it ends. The published ideas closest to the feedback half are Self-RAG’s reflection tokens; the contrast with chain-of-thought shows what changes when the model emits evidence instead of reasoning prose.

Same direction as the article: