Coding Brokers Do not Want Longer Historical past — They Want Intent Continuity

TL;DR

• I constructed an entire, working implementation in pure Python and shared precise benchmark numbers from actual runs (no simulated knowledge).

• The core lesson: simply pulling up previous historical past is not the identical as figuring out what’s truly nonetheless correct.

• A primary search setup solely grabbed 57% of the necessities a coding agent wanted. Including a verification layer pushed that to 100%.

• Out of 8 duties, the baseline received zero proper, primary search received 4, and intent-aware search nailed all 8.

• I did all of this with zero embeddings, zero vector databases, and completely no LLM calls within the pipeline.

• I additionally come clean with a bug in my unique experiment design that nearly made my outcomes look method higher than they really have been.

Why Extra Historical past Is not Sufficient

I arrange a coding agent workflow that labored completely at first. However as soon as a challenge received lengthy sufficient, it began inflicting issues.

When a challenge handed just a few dozen steps, core guidelines started vanishing. Nobody deleted them. The context window was not full. These guidelines have been nonetheless technically sitting within the chat logs. They simply dropped off the radar as a result of new requests didn’t set off the agent to verify if an older resolution nonetheless mattered.

As an illustration, you may inform the agent on day one to by no means expose inside database IDs in API responses. Sixty messages later, you ask it to construct a brand new authentication stream. That new request says nothing about IDs. For the reason that agent lacks a transparent motive to look again, it skips that step and ships an endpoint leaking the precise knowledge you tried to guard.

This isn’t a made up situation. It’s the precise check case I used for this text. Under, I’ll present you ways three completely different strategies deal with this precise downside.

Each outcome proven right here comes from precise check runs utilizing Python 3.12 with no outdoors dependencies. You may clone the repo and run run_experiment.py to breed the numbers your self, until I particularly name out an remoted check.

Full Code: https://github.com/Emmimal/intent-continuity/

What Intent Continuity Truly Means

Phrases get blended up right here fairly quick, so allow us to clear up the definitions.

Customary RAG, launched by Lewis et al. (2020) [1], connects a language mannequin to a retrieval system that finds related info from an exterior data supply. The essential query is easy: what info is related to this question?

Larger context home windows let fashions maintain extra textual content without delay. However that measurement doesn’t make the mannequin verify an outdated rule buried sixty turns again. Liu et al. (2023) [2] identified that fashions miss particulars caught in the midst of lengthy prompts, even inside their acknowledged limits.

But that misses the true level. Even with whole recall, a mannequin nonetheless has to attach an outdated rule about database IDs to a brand new login activity. Reminiscence failure isn’t the issue. Deciding what issues is.

Intent continuity is completely different. It means carrying an outdated requirement into a brand new activity with out the consumer repeating it, whereas dropping that rule if one thing newer overrides it.

Right here is the precise cut up this text focuses on:

Retrieval asks:"What historic info may be related?"Verification asks:"Is that info nonetheless legitimate?"Intent continuity asks:"What historic intent ought to affect this activity, proper now?"

Proper now, most discuss agent reminiscence focuses purely on that first query.

Who This Is For

Construct this in the event you run coding brokers on long-running initiatives the place guidelines get acknowledged as soon as and forgotten. Consider multi-week refactors, codebases full of outdated design decisions, or groups the place whoever set a constraint three weeks in the past isn’t the particular person prompting the agent right this moment.

Skip it for fast, single-session duties that carry no historical past. Skip it in case your challenge is sufficiently small that you may simply paste your full necessities doc into each immediate. Skip it if you’re already manually repeating each rule to the agent on each flip. If a human is consistently reminding the agent what to do, the system by no means must search for previous choices.

In case your agent periods keep quick and your guidelines by no means shift, commonplace search or zero reminiscence works nice. Lengthy initiatives simply don’t work that method.

Full Pipeline Structure

I set one strict rule earlier than writing a single line of code: 100% pure Python.

No API keys, no exterior LLM, no embedding fashions, and no vector databases.

A part of the explanation was comfort. I wished anybody to clone the repo and run it in below a second with zero setup friction. However the greater motive was management. If I relied on an embedding mannequin, the benchmark outcomes would simply get twisted up in how good or unhealthy that particular mannequin occurred to be.

The verification logic is what truly does the heavy lifting right here, and I wished to show it will possibly stand fully by itself two toes.

The pipeline begins by turning uncooked chat logs into structured requirement information.

The extractor is simply easy. It scans every message for sentences that appear to be necessities, identifies the a part of the system the requirement seems to focus on, and extracts particular values when they’re current.

# extractor.py (abridged)TRIGGER_PHRASES = [    "must", "never", "required", "require", "prefer", "should",    "always", "migrating", "let's use", "default", "for now",]def _looks_like_requirement(textual content: str) -> bool:    decrease = textual content.decrease()    return any(phrase in decrease for phrase in TRIGGER_PHRASES)

Set off-Phrase Flagging vs. Floor Fact — 70 Interactions

The important thing takeaway right here is 100% recall. The extractor caught each single planted requirement within the check set.

Precision landed at 0.80 on goal. Three on a regular basis sentences tripped the filter as a result of they used set off phrases like “should”—as an illustration, telling somebody you should run to a dentist appointment.

The subsequent step cleans these up. The element classifier checks if a candidate maps to an actual system half. If it doesn’t discover a match, it drops the merchandise.

I stored these false positives within the benchmark intentionally. A key phrase filter claiming flawless precision on a tuned check set normally simply hides its weaknesses. I wished predictable, measurable habits as an alternative.

Put merely, the extractor is written to catch an excessive amount of moderately than miss a rule in silence.

Is that this code fancy? In no way. It’s only a light-weight extractor that feeds structured knowledge to the subsequent steps. An actual manufacturing app would wish one thing a lot heavier.

Part 2: Candidate Retrieval and the Area Schema

Candidate retrieval figures out which previous information to verify earlier than working any verification. It depends on two easy alerts. First, does the file share the identical system element as the present activity? Second, does it belong to a linked element listed in a site schema?

# domain_schema.pyCOMPONENT_RELATIONSHIPS = {    "auth": ["security", "api"],    "api": ["security", "testing"],    "safety": ["auth", "api"],    "database": ["deployment"],    "deployment": ["database"],    "testing": ["api"],    "ui": [],    "efficiency": [],}

I wrote this schema as soon as based mostly on common backend design guidelines. As an illustration, auth work impacts safety and API habits, whereas API work requires testing and safety evaluations.

The system applies this precise schema to each single activity with out modification.

That mounted method issues. An earlier model let me outline customized relationships for every particular person activity, which felt like dishonest. I’ll clarify why that skewed issues shortly.

Part 3: Verification

That is the a part of the pipeline that really handles intent continuity. As soon as the system pulls up candidate guidelines, verification runs two fast checks on every one: has a more moderen rule changed it, and does it apply to the present activity context?

The system figures out if a rule is outmoded utilizing a easy rule as an alternative of handbook labels. If two information share the identical element, scope, and goal key, however have completely different values, the newer one replaces the older one.

Similar Rule, Two Totally different Outcomes

Take R2 and R7. They discuss the identical key in the identical scope, so the later one wins and drops the outdated one.

Then have a look at R4 and R5. They disagree on the database engine too, however they aim completely different scopes: manufacturing versus prototype. The verification step retains each lively as a result of they apply to separate environments.

Dealing with that distinction mechanically issues loads. If the system handled them as conflicting, it could break a core use case. Writing code to compute that distinction as an alternative of hardcoding labels turned out to be probably the most vital design selection in the entire challenge.

Part 4: The Compiler and the Deterministic Agent

No matter information survive the verification stage get flattened right into a clear key-value context. From there, they go straight right into a template that mimics a coding agent.

# compiler.py — decision rule, utilized identically for each situationdef compile_context(context_records):    provenance = {}    for file in sorted(context_records, key=lambda r: r.index):        provenance[record.effect_key] = file  # later overwrites earlier    fields = {key: r.effect_value for key, r in provenance.gadgets()}    token_estimate = sum(len(r.textual content.cut up()) for r in context_records)    return fields, provenance, token_estimate

The simulated agent itself is deliberately easy. It begins with a set set of defaults, will get up to date by no matter fields it truly receives, and has zero capability to guess a rule it was by no means handed.

That’s the complete level of the setup. Each outcome you see beneath comes down fully to what every search technique managed to get better. It has nothing to do with a language mannequin having a great or unhealthy day, as a result of there isn’t a mannequin within the pipeline in any respect.

Part 5: The Checker

A single perform grades each run utilizing the very same ground-truth discipline record each time.

No method will get particular remedy or a unique rubric. The required fields for every activity are locked in earlier than any search technique runs, and each technique is measured in opposition to that very same mounted commonplace.

What Occurs on Activity T1

Right here is how the pipeline performs in opposition to one of many benchmark duties:

“Implement the brand new authentication stream.”

Floor fact: This activity depends on three unspoken constraints: the OAuth2 migration from flip 39, a backward-compatibility rule from flip 4, and an internal-ID hiding rule from flip 15. The immediate doesn’t point out any of them.

Three Circumstances, One Activity

This diagram walks by an actual supersession case from the intent-continuity experiment: an early resolution to make use of JWT-based authentication, later changed by a choice emigrate to OAuth2. Naive lexical retrieval, the sort of habits you’d get from plain key phrase or vector similarity search, finds each information and lets the newer one win purely as a result of it was talked about extra not too long ago, a coincidence of ordering moderately than an precise validity verify. Intent-aware retrieval finds the identical two information however runs an specific verification step that determines the older file is outmoded earlier than both one reaches the coding agent. Each approaches land on the proper auth technique on this specific case, which is strictly the purpose: one received there by luck, and the opposite by design, and that distinction is invisible till you check a case the place the ordering does not occur to save lots of you.

Customary key phrase search will get the auth technique proper, however solely as a result of R7 occurs to overwrite R2 throughout context meeting. The compiler simply retains the final file it sees, moderately than determining that the older JWT resolution was truly invalid. It fully misses backward compatibility and ID publicity as a result of these sentences don’t share a single phrase with “implement the brand new authentication stream.”

Intent-aware retrieval grabs these hidden necessities by the area schema as an alternative of counting on key phrase matches. The verification step explicitly determines that R2 is outmoded earlier than something reaches the agent, which is a deliberate validation step moderately than a random ordering quirk.

It additionally picks up R10, a rate-limiting constraint, proper alongside the three graded necessities. R10 is a sound historic constraint that sits outdoors the guidelines for this particular activity, proving the system captures context with out over-specializing.

Said as plainly as attainable throughout the three situations:

• Baseline: “I have no idea the previous.”

• Customary key phrase search: “I discovered one thing with matching key phrases.”

• Intent-aware: “I discovered a number of associated gadgets, checked which of them have been nonetheless legitimate, and reconstructed what truly issues.”

The Experiment: 70 Interactions, 12 Necessities, 8 Duties

I constructed an artificial challenge historical past as an alternative of utilizing actual chat logs for a similar motive the pipeline has zero exterior dependencies. I wished a floor fact I may totally confirm, moderately than a dataset the place determining what the agent ought to have recognized turns into a subjective judgment name.

The setup accommodates seventy chronologically ordered interactions: twelve real planted necessities, three entice sentences designed to journey up a key phrase extractor with out being actual guidelines, and fifty-five traces of atypical noise like standup reminders, pull request feedback, and informal chat.

The 12 Planted Necessities

The 8 Later Duties (None Restate Their Dependencies)

Measuring What It Truly Recovers

All numbers beneath come from actual runs of run_experiment.py. Nothing here’s a projection or an estimate.

Mixture Outcomes Throughout All 8 Duties

This bar chart reveals the ultimate task-completion outcomes from the intent-continuity experiment throughout the identical 8 coding duties. A coding agent with no entry to historical past handed 0 of 8 duties, breaking a requirement it was by no means advised nonetheless utilized. Naive lexical retrieval, the sort of outcome you’d count on from primary key phrase or vector-similarity search with no validity checking, handed 4 of 8. Intent-aware retrieval, which provides an specific verification step to drop outmoded or out-of-scope necessities earlier than they attain the agent, handed all 8. The hole between naive retrieval and intent-aware retrieval is the precise discovering of this experiment: retrieving associated historical past is not the identical as retrieving necessities the agent can at present belief.

Intent continuity, as carried out right here, isn’t a compression approach. I need that acknowledged explicitly moderately than left for a reader to deduce from the desk.

Intent-aware retrieval makes use of extra tokens than commonplace key phrase retrieval—199 versus 155, or roughly 28 % extra—as a result of it accurately recovers necessities that commonplace retrieval misses outright, and that trade-off prices one thing. The core declare right here is about correctness, not compression. The system used extra tokens and produced higher activity accuracy, moderately than a smaller footprint.

“Irrelevant Retrieved” Is Not a Single Quantity

Eleven of ordinary retrieval’s seventeen irrelevant hits are pure chatter matched by lexical accident, which is real junk with zero relation to the duty.

Intent-aware retrieval’s ten extras cut up evenly: 5 are true noise accurately filtered out throughout verification, and 5 are information that attain the agent with out being on that particular activity’s graded guidelines. These 5 further information are actual, at present legitimate context, not errors.

That distinction issues. Customary retrieval’s extras aren’t assured to be proper; they’re simply retrieved. For Activity T3, commonplace retrieval pulls in each the proper manufacturing database resolution and the stale prototype resolution, and the stale one wins the sphere as a result of there isn’t a verification layer to cease it.

Per-Activity Move/Fail

The Time I Virtually Shipped a Rigged Experiment

The area schema described in Part 2 went by an earlier model that was a lot narrower and far worse. I had declared these element relationships per activity as an alternative of worldwide. Activity T1 was individually advised upfront, “you additionally depend upon api and safety.” Activity T4 was individually advised, “you additionally depend upon safety and testing.” These two hand-picked declarations occurred to be precisely the parts these two duties’ appropriate solutions wanted, and nothing extra.

That isn’t a discovery mechanism. That’s a solution key dressed up as a retrieval rule, and it instantly undercut your complete premise of this challenge, which is meant to work with out being advised the place to look.

I caught it the one method that really works: I deleted the per-task trace and reran the experiment with nothing put as a replacement.

Model With vs. With out the Per-Activity Trace (Ablation)

The drop was not refined. It failed precisely the 2 duties that had been individually hand-fed their solutions—proof that the trace had been doing actual, load-bearing work your complete time. I had practically missed it just because the ultimate mixture rating appeared good.

The repair, proven in full in Part 2, was changing the per-task trace with one common schema, authored a single time, and utilized uniformly to each activity, together with the six that by no means wanted the additional assist.

Making use of it uniformly moderately than choosing it value one thing actual. Irrelevant information retrieved rose from 4 to 10, and tokens equipped rose from 161 to 199, as a result of the schema now additionally fires harmlessly for duties that by no means wanted it. The outcome stayed at 8 out of 8, however this time, it earned it.

An artificial benchmark you constructed your self is the simplest factor on this planet to unconsciously rig, since you already know the solutions earlier than you write the check. Delete the half you watched is doing an excessive amount of heavy lifting and see what breaks. It’s the solely sanity verify that really labored right here.

Sincere Design Choices

The element and key phrase dictionaries are hand-authored for this area moderately than realized. It is a managed demonstration of the verification mechanism, not a general-purpose extraction system you possibly can level at an arbitrary codebase tomorrow.

Retrieval on this experiment makes use of plain lexical phrase overlap as an alternative of an embedding mannequin or vector database. This selection retains retrieval high quality from turning into a confounding variable. If I had used a selected embedding mannequin, a skeptical reader may fairly argue the entire comparability trusted which mannequin I occurred to select. Swapping in an actual embedding mannequin would probably change the recall numbers for traditional retrieval, however it could not change the underlying argument. The verification step is what does the attention-grabbing work, and it stays agnostic to how candidates have been discovered.

Eight duties and twelve necessities make up an indication moderately than a statistically powered research. The scope is sized to be totally inspectable and reproducible, to not generalize with confidence to arbitrary manufacturing codebases.

The simulated agent is a deterministic template on goal, not an actual coding language mannequin. This ensures each result’s attributable strictly to what every retrieval technique recovered, moderately than to mannequin habits on a given day.

The checker solely assessments fields explicitly declared in every activity’s floor fact. That could be a slender rubric by design, not a common measure of code high quality.

Commerce-offs and What’s Lacking

• Actual extraction: The rule-based extractor works right here as a result of I management the dataset. A manufacturing model would wish a genuinely strong extraction entrance finish, probably a small classifier moderately than a easy trigger-phrase record.

• Embedding-based candidate retrieval: The retrieval step is a clear swap level. Drop in an embedding mannequin for candidate era and the verification layer downstream doesn’t want to vary in any respect.

• An actual coding LLM: The deterministic agent template exists particularly to isolate what every retrieval technique recovers. Changing it with an precise mannequin would check a unique speculation: whether or not the mannequin accurately makes use of the recovered context, moderately than simply whether or not the context was efficiently discovered.

• Cross-session persistence: Every part right here runs fully in-process. A light-weight persistent retailer sharing the identical file interface would permit intent continuity to outlive throughout restarts.

Closing

Retrieval will get you what is associated. Verification will get you what is legitimate. Intent continuity will get you what nonetheless issues, proper now, for the duty in entrance of you.

Most programs optimize the primary and skip the opposite two. That is why they preserve transport code that quietly breaks a choice somebody made weeks in the past.

Full code: https://github.com/Emmimal/intent-continuity/

References

[1] Lewis, P., Perez, E., Piktus, A., et al. (2020). Retrieval-Augmented Technology for Information-Intensive NLP Duties. NeurIPS 33, 9459–9474. https://arxiv.org/abs/2005.11401

[2] Liu, N. F., Lin, Okay., Hewitt, J., et al. (2023). Misplaced within the Center: How Language Fashions Use Lengthy Contexts. arXiv:2307.03172. https://arxiv.org/abs/2307.03172

Disclosure

All code on this article was written by me. It’s unique work, developed and examined on Python 3.12. All benchmark numbers come from precise runs of the system and are reproducible by cloning the repository and working run_experiment.py. Not one of the outcomes have been calculated or simulated after the very fact.

The system makes use of zero exterior dependencies and runs fully on the Python commonplace library. It doesn’t use an embedding mannequin, vector database, or LLM API. I’ve no monetary relationship with any software, library, or firm talked about on this article.

All diagrams on this article, together with the featured picture, have been created by the creator. The featured picture was generated with ChatGPT (DALL·E).