Lengthy Context Isn’t Free — I Constructed a Secure Immediate-Pruning Layer That Makes LLM Methods Work

I’ve labored on, dialog state tends to develop rapidly over time. It’s widespread to resend massive parts of the historical past on every flip—together with older device outputs, repeated RAG retrievals, and context that’s not related. As this accumulates, prompts can grow to be considerably bigger, which can improve inference price and latency, and in some circumstances have an effect on reasoning efficiency.

I constructed a deterministic pipeline that prunes this redundant state earlier than the immediate ever reaches the mannequin. The model I carried out avoids LLM calls, embeddings, and exterior dependencies. Relying strictly on customary library parts ensures each pruning choice stays absolutely deterministic and reproducible.

State monitoring executes in three distinct passes: Expired Context Elimination, Duplicate Context Elimination, and Dependency Restoration. The third go is what helps make the primary two safer in apply. It ensures that nothing a later message is dependent upon is by chance eliminated.

Whereas constructing it, I bumped into two bugs that modified the design. My first benchmark corpus used a hard and fast variety of duplicates and rancid device calls, which made discount percentages shrink as conversations grew. That didn’t mirror what I’d anticipate from real-world conduct.

My dependency restoration logic additionally went utterly untested at first as a result of my artificial knowledge by no means created a case the place a required message was truly eliminated. Each points are coated right here, together with how fixing them modified the outcomes.

After correcting the pipeline, I benchmarked it throughout three workloads: plain chat, a RAG assistant, and a tool-heavy agent. Every was examined at 5 dialog sizes, for a complete of 15 configurations on two completely different machines. Throughout these runs, all labeled required info have been preserved. The system additionally reached a steady fastened level after a single go, which suggests pruning an already pruned immediate produces no additional adjustments.

Token discount is dependent upon the workload. It’s about 2 to 4 p.c for plain chat, 27 to 32 p.c for a RAG assistant, and 33 to 34 p.c for a tool-heavy agent. Even at 2,000 turns and 131,000 tokens, preprocessing stayed underneath 50 milliseconds.

Full code, all 35 assessments, and the uncooked terminal output are included under so you’ll be able to run the pipeline your self.

Full code: https://github.com/Emmimal/prompt-pruning-layer/

Each immediate I ship retains getting heavier

I stored seeing the very same sample pop up throughout each long-running agent I constructed. A dialog begins out completely clear. Fifty turns later, it’s an absolute mess.

By that fiftieth flip, the immediate payload you’re transport out on each single request consists of the system immediate, your entire chat historical past, 4 device outputs (two of that are utterly stale as a result of the device ran twice), six retrieved chunks (three are near-duplicates as a result of the consumer circled again to an outdated matter), a SQL consequence from twenty turns in the past that’s simply sitting there, and a single consumer choice acknowledged as soon as and by no means introduced up once more.

None of these things is technically fallacious. Each single piece made sense the precise second it was injected. The actual challenge is that nothing ever will get cleared out. The immediate turns into an append-only log of each historic occasion, and also you’re dumping the entire thing onto the mannequin, each single flip, indefinitely.

That’s not only a storage drawback. Reasoning efficiency measurably degrades as enter size grows, even when the added content material is irrelevant to the duty [1], and fashions are worse at utilizing info buried in the course of a protracted context than info close to the sides [2].

The quick knee-jerk response whenever you see this bloating is simply fundamental truncation. Slice the final N messages and drop all the things else. It’s actually one line of code. The catch is it silently breaks your dialog chains in methods you received’t even notice till it blows up on an actual consumer:

Flip 3:   Consumer: My most well-liked output format is CSV.
Flip 4:   Assistant: Received it.
...
Flip 47:  Consumer: Export the outcomes.

In case your window is capped on the final 20 turns, flip 3 disappears by flip 47. The assistant loses the context that the consumer requested for a CSV. That knowledge was not stale; it was a tough dependency, and easy positional truncation can not differentiate between outdated, expendable context and outdated context {that a} later flip nonetheless depends on.

That’s the precise design constraint this mission addresses. Any mechanism that removes redundant state should distinguish between these two classes. Recency alone is inadequate.

Borrowing an concept from working techniques

Right here is the reframe I constructed the remainder of this round: an working system is continually deciding which pages keep resident in RAM and which get evicted. An extended-running LLM dialog has the very same drawback, besides nothing performs the position of the reminiscence supervisor. Context simply accumulates perpetually as a result of no course of owns the job of deciding what stops incomes its place within the immediate.

The Immediate Pruner on this article is that lacking piece.

Each piece of dialog state (a consumer flip, an assistant flip, a device output, a retrieved chunk) is a Message object with a job, a flip quantity, and a little bit of bookkeeping metadata:

@dataclass
class Message:
    id: str
    position: str
    content material: str
    flip: int
    tool_call_key: Optionally available[str] = None
    expires_after_turn: Optionally available[int] = None
    defines_keys: checklist = discipline(default_factory=checklist)

Three passes run over that checklist so as. Each is a pure operate: messages in, a filtered checklist out. No mannequin is concerned wherever within the pruning step itself.

Why there’s no mannequin contained in the pruner itself

I may have used an embedding mannequin to attain message relevance, which might have made this fuzzier and so much simpler to put in writing. I didn’t, and it’s not for the sake of purity.

As soon as a pruning choice is dependent upon a mannequin’s judgment, you lose the power to cause about what a dialog will appear like on the subsequent flip. The identical enter not ensures the identical output. A pruning layer meant to make a manufacturing system extra predictable shouldn’t be the least predictable a part of it. The whole lot right here runs on dataclasses, regex, and dict lookups, which is strictly the toolset this drawback wants.

Move 1: Expired Context Elimination

If a device will get referred to as greater than as soon as underneath the identical key (the identical search question, the identical SQL lookup, the identical file learn), solely the latest consequence remains to be reliable. The whole lot earlier underneath that secret’s expired.

def _pass1_expired_context_elimination(self, messages):
    last_occurrence = {}
    for m in messages:
        if m.tool_call_key:
            last_occurrence[m.tool_call_key] = m.id

    stored, eliminated = [], []
    for m in messages:
        if m.tool_call_key and last_occurrence[m.tool_call_key] != m.id:
            eliminated.append(m)
        else:
            stored.append(m)
    return stored, eliminated

Move 2: Duplicate Context Elimination

Retrieval pipelines always pull up an identical or near-duplicate passages, particularly when a consumer loops again to an earlier matter. This go normalizes the whitespace and casing, retains solely the primary incidence, and drops each duplicate after it.

def _pass2_duplicate_context_elimination(self, messages):
    seen, stored, eliminated = {}, [], []
    for m in messages:
        if m.position == ROLE_RETRIEVED_DOC:
            norm = " ".be part of(m.content material.decrease().break up())
            if norm in seen:
                eliminated.append(m)
                proceed
            seen[norm] = m.id
        stored.append(m)
    return stored, eliminated

Move 3: Dependency Restoration (and the bug that made me construct it correctly)

This go is the explanation the primary two are secure to run in any respect. If Move 1 or Move 2 drops a message that occurs to be the one place a still-referenced truth bought outlined, this go catches it and places it again.

The mechanism is deliberately easy: a message marks itself with a literal DEFINE: tag, and a later message references it utilizing REF:. If a REF survives into the ultimate stored set however its matching DEFINE bought dropped upstream, Dependency Restoration restores that lacking message to the checklist.

def _pass3_dependency_restoration(self, all_messages, kept_messages, removed_messages):
    kept_ids = {m.id for m in kept_messages}
    by_id = {m.id: m for m in all_messages}

    key_definer = {}
    for m in all_messages:
        for key in m.defines_keys:
            key_definer[key] = m.id

    referenced_keys = set()
    for m in kept_messages:
        referenced_keys.replace(m.references())

    restored = []
    for key in referenced_keys:
        definer_id = key_definer.get(key)
        if definer_id and definer_id not in kept_ids:
            restored_msg = by_id[definer_id]
            kept_messages.append(restored_msg)
            kept_ids.add(definer_id)
            restored.append(restored_msg)

    kept_messages.type(key=lambda m: (m.flip, m.id))
    return kept_messages, restored

Right here is the bug. I ran my first full benchmark throughout all three workloads and 5 sizes, and each single row printed “Restored (deps): 0.” Each one. My first response was that this appeared nice—an ideal security report. It wasn’t. It meant Move 3 had by no means truly restored a single message on any run at any dimension.

I went again into my corpus generator to seek out out why, and the reply was embarrassing as soon as I noticed it. My artificial conversations solely ever hooked up DEFINE markers to plain consumer messages, whereas Move 1 and Move 2 solely ever take away device outputs and retrieved paperwork. The 2 classes by no means overlapped. Dependency Restoration was sitting within the codebase, absolutely written, however utterly untested by my very own benchmark as a result of nothing I generated ever gave it a cause to fireplace.

The repair was to let some device outputs additionally outline a dependency, the identical means an actual “get consumer settings” device name may floor a truth the dialog is dependent upon later, although that precise device name may get outmoded and marked expired by Move 1. As soon as I added that, the numbers modified instantly: 2 restorations on the smallest tool-agent dialog, climbing to 127 on the largest. Required info stayed at one hundred pc preserved your entire time, however now that quantity truly meant one thing, as a result of the go being examined was able to failing and didn’t.

I need to be direct concerning the limitation that’s nonetheless right here even after the repair: dependency detection is literal identifier matching, not understanding. It catches a precise REF matching a precise DEFINE. It won’t catch a consumer paraphrasing, asking “what format did I point out earlier” with no matching tag. A semantic dependency resolver would want an embedding mannequin or an LLM name, and that’s explicitly exterior what this deterministic pipeline does. I’d quite ship a narrower assure I can show than a broader one I can’t.

Design targets, and why each exists

Deterministic. Similar enter at all times yields the identical output. Holding the mannequin out of the loop eliminates run-to-run variance and any danger of hallucinating what ought to stay within the immediate.

Dependency-safe. It by no means silently drops a truth {that a} later flip nonetheless wants. Positional truncation utterly lacks this property, which makes it non-negotiable right here. A pruner saving 40 p.c of tokens that sometimes breaks a dialog is a worse trade-off than one saving 4 p.c that by no means breaks something.

Idempotent. Operating the pruner a second time does nothing. If that isn’t true, you can not safely re-prune on each single flip of a rising dialog with out risking compounding drift.

Light-weight. The pruning step ought to by no means grow to be the precise bottleneck it was constructed to eradicate.

The benchmark: three workloads, and a mistake I nearly shipped

My first model of the artificial corpus generator picked a hard and fast variety of duplicate passages and repeated device calls (six device calls and eight duplicates) no matter how lengthy the dialog was. I ran it at 5 sizes and the discount share went down because the dialog bought longer: 9.9 p.c at 50 turns, dropping to 0.3 p.c at 2,000 turns. That runs backward from precise manufacturing site visitors, and it’s not defensible. If the quantity of waste in a benchmark is only a fixed I hand-picked, anybody studying it’s proper to ask whether or not I constructed the benchmark to show the algorithm works.

So I threw that generator out and rebuilt it round an specific workload mannequin as a substitute: retrieval-per-turn, retrieval overlap charge, device name charge, and power repetition charge. All of those have been fastened earlier than working a single benchmark, primarily based on what appeared like believable manufacturing conduct, quite than adjusted afterward to hit a quantity I favored. Three workloads got here out of that:

Regular chat. No retrieval, occasional device calls, principally bizarre back-and-forth.

RAG assistant. Retrieves paperwork on each flip, with an actual probability any given passage overlaps one thing retrieved just lately, as a result of customers revisit subjects and retrieval re-surfaces the identical chunks.

Instrument agent. Frequent calls throughout 5 device varieties (search, SQL, calculator, filesystem, net fetch), excessive repetition charge, modeling one thing that re-plans and re-queries always.

Each artificial corpus additionally ships with floor reality: each message some later message is dependent upon will get labeled required up entrance. So “did pruning maintain all the things it wanted to” is a examine in opposition to identified labels, not a guess.

Right here is the entire output. All 15 configurations, not a slice of it:

Workload	Turns	Tokens earlier than	Tokens after	Discount	Details stored	Idempotent	Overhead
Regular chat	50	1,175	1,153	1.87%	Sure	Sure	0.17 ms
Regular chat	200	4,820	4,629	3.96%	Sure	Sure	0.79 ms
Regular chat	500	12,078	11,660	3.46%	Sure	Sure	1.82 ms
Regular chat	1000	24,379	23,381	4.09%	Sure	Sure	3.79 ms
Regular chat	2000	48,241	46,514	3.58%	Sure	Sure	8.27 ms
RAG assistant	50	3,494	2,551	26.99%	Sure	Sure	0.60 ms
RAG assistant	200	14,009	9,599	31.48%	Sure	Sure	2.18 ms
RAG assistant	500	35,347	24,133	31.73%	Sure	Sure	6.19 ms
RAG assistant	1000	70,358	47,950	31.85%	Sure	Sure	11.89 ms
RAG assistant	2000	140,766	95,087	32.45%	Sure	Sure	28.95 ms
Instrument agent	50	3,279	2,176	33.64%	Sure	Sure	0.47 ms
Instrument agent	200	12,955	8,585	33.73%	Sure	Sure	2.04 ms
Instrument agent	500	32,412	21,677	33.12%	Sure	Sure	6.46 ms
Instrument agent	1000	65,366	43,351	33.68%	Sure	Sure	15.02 ms
Instrument agent	2000	131,591	87,625	33.41%	Sure	Sure	43.04 ms

Each single row says Details stored: Sure and Idempotent: Sure. Not most rows. All fifteen.

The sample by workload is sensible when you have a look at the place the waste truly comes from. Regular chat barely retrieves something and infrequently repeats a device name, so there may be nearly nothing for Move 1 or Move 2 to catch; it stays round 4 p.c irrespective of how lengthy the dialog runs. The RAG assistant retrieves each flip with actual overlap, so Duplicate Context Elimination carries a lot of the weight, touchdown round 32 p.c. The Instrument agent combines each issues (frequent device repetition and retrieval overlap) and hits the very best discount at 33 to 34 p.c.

Completely different workloads accumulate completely different sorts of waste. The pruner responds on to no matter waste is sitting in entrance of it, quite than producing a flat quantity that might recommend the benchmark was reverse-engineered to hit a goal.

If I solely bought to maintain one consequence from this entire benchmark, it’s the security property: 15 out of 15 configurations preserved one hundred pc of required info. Zero lacking dependencies, throughout three structurally completely different workloads and a 40x vary in dialog size. Truncation by place can not provide that. For me, this was an important sign when evaluating whether or not the method is perhaps usable in manufacturing.

That quantity can also be computed, not hand-counted. The benchmark script itself tallies how most of the 15 (workload, dimension) pairs preserved each required truth and what number of reached the idempotent fastened level, printing an combination abstract block after the 15 detailed runs:

============================================================
SUMMARY
============================================================
Configurations run:               15 (3 workloads x 5 sizes)
Required info preserved:         15/15
Reached fastened level (idempotent): 15/15

Workload            Token discount vary    Details preserved   Idempotent
Regular chat          1.9-4.1%                YES               YES
RAG assistant        27.0-32.5%               YES               YES
Instrument agent           33.1-33.7%               YES               YES

I wrote this examine as a result of I caught myself hand-counting desk rows for an early draft. That metric belongs within the script output, not my very own eyes squinting at a terminal log. If a take a look at run ever hits something lower than 15/15, it means I broke the pruner and have a regression to seek out, not a typo to edit within the publish.

I left one particular metric out of the ultimate numbers: tokens eliminated per millisecond of execution overhead. The code computes it—it peaks at round 4,000 tokens per millisecond on small tool-agent runs and lands between 900 and 1,700 tokens at bigger scales. It stays within the codebase as an inner discipline as a result of it helps monitor scaling prices, however it belongs exterior the primary desk. Readers can not act on it the way in which they’ll with uncooked token depend, discount share, or millisecond overhead. Three direct metrics exhibiting a transparent trade-off are higher than a fourth that acts as a novelty.

The idempotence result’s the half I favored monitoring probably the most. Proving prune(prune(x)) == prune(x) means the pipeline hits a steady fastened level on the primary go. Operating it once more on an already-pruned immediate adjustments nothing:

That guidelines out oscillation. It additionally guidelines out cumulative shrinkage throughout turns when you re-pruner on each single message of a rising dialog, which is strictly how this runs in manufacturing.

Reproducing it on a second machine

I ran the total benchmark on a Linux container working Python 3.12.3, then once more on Home windows 11 in PyCharm utilizing Python 3.12 in a separate venv. Each token depend and message depend matched precisely throughout each machines. Solely the millisecond timings moved, which is customary for various {hardware}, and even these stayed underneath 50 milliseconds for the most important, most tool-heavy dialog on each setups.

One factor I observed whereas evaluating the 2 runs and need to be straight about: prompt-build time (the time to serialize the ultimate message checklist right into a string) sometimes got here out slower after pruning than earlier than on the Regular Chat workload. Not by a lot—underneath a millisecond—however the path was backward.

My learn is that Regular Chat solely removes 2 to 4 p.c of messages, so the earlier than and after checklist sizes are practically an identical. At sub-two-millisecond operations, system jitter and rubbish assortment pauses simply swamp the precise sign. Including a warm-up name and utilizing the median of 30 runs principally stabilized the metric, however the anomaly nonetheless pops up on that workload. It’s noise at a scale the place the delta is smaller than the measurement error, so I left it uncooked quite than massaging the information.

What this benchmark intentionally doesn’t measure

This benchmark isolates token discount and pruning overhead as a result of these are the metrics the pipeline truly controls. Finish-to-end LLM latency is a totally separate variable. It is dependent upon supplier structure, batching, regional caching, and community circumstances that this mission can not see. Making an attempt to transform a token discount share immediately right into a latency delta means inventing an arbitrary conversion fixed.

The baseline actuality is easy: chopping 30 to 34 p.c of enter tokens means the mannequin does much less work per name. Normally, inference price and latency have a tendency to extend with immediate dimension [4], making this a helpful price lever. However a real latency quantity requires a dwell validation go in opposition to your particular supplier. Publishing a generic latency determine right here would imply making claims about infrastructure I don’t management, quite than evaluating the pruning layer itself.

How this matches into an actual agent loop

The pruner lives in precisely one spot: proper after the dialog historical past is pulled collectively for a flip, and simply earlier than it will get serialized into the ultimate immediate string.

from prompt_pruning import PromptPruner, PromptBuilder

pruner = PromptPruner()
builder = PromptBuilder()

def handle_turn(conversation_state, new_user_message):
    conversation_state.append(new_user_message)

    pruned_messages, report = pruner.prune(conversation_state)
    immediate = builder.construct(pruned_messages)
    response = call_llm(immediate)

    conversation_state.append(make_assistant_message(response))
    return response

As a result of the pipeline is idempotent, calling prune() each flip of a rising dialog is secure. Operating the pruner ten instances on a historical past pruned 9 instances yields the very same consequence as a clear run from scratch. This makes “run it each flip” a secure default, eliminating the necessity to monitor state or cause about earlier passes.

The one integration choice left is how REF and DEFINE tags get hooked up to messages. Right here they’re literal markers contained in the message content material, which is the best mechanism for a prototype. A manufacturing system would seemingly connect them as structured metadata on the message object so the tags by no means leak into the uncooked textual content the mannequin reads. Move 3’s logic stays the identical both means. An upstream course of nonetheless has to find out what counts as a dependency value tagging, as a result of Move 3 can solely restore what it’s explicitly instructed to trace.

What this doesn’t cowl

Dependency detection is literal, not semantic. If a reference is paraphrased and lacks an identical tag, the script will miss it.

These workloads are additionally utterly artificial. I selected the three parameter units primarily based on believable manufacturing conduct, not actual telemetry. When you’ve got manufacturing logs, regenerating these three workload classes from precise utilization is the apparent subsequent step. The numbers will shift relying on what your precise site visitors seems like.

This pipeline omits semantic compression, embeddings, and LLM-scored pruning. These are legitimate, different approaches, however avoiding them retains this implementation absolutely deterministic and dependency-free. LLMLingua is a primary instance of the realized different; it makes use of a small language mannequin to attain and drop tokens, reaching a lot larger compression ratios than this script [3]. Selecting between them is a direct trade-off: you change determinism and zero-dependency execution for tighter compression.

The token counts are additionally approximations. The script makes use of a whitespace and punctuation-boundary heuristic as a substitute of a manufacturing subword tokenizer like tiktoken. As a result of the heuristic runs constantly earlier than and after pruning, the relative discount percentages stay correct, even when absolutely the numbers don’t completely match an official tokenizer.

Lastly, there isn’t a direct latency measurement for the infrastructure causes detailed earlier.

The place I’d take this subsequent

Two extensions make sense right here quite than increasing the prevailing three passes.

The primary is a hybrid method. Preserve these three deterministic passes as a quick, secure first stage, then hand the output to an embedding-aware or LLM-scored compression device like LLMLingua. This catches semantic redundancy that literal identifier matching misses, corresponding to two passages saying the identical factor in numerous phrases.

Operating the deterministic passes first preserves the security assure. If the realized stage misbehaves, the dialog drops again to the deterministic baseline as a substitute of failing unpredictably. Architecturally, the realized go solely operates on the output of Move 3. A mannequin bug within the compression step may shrink a immediate too aggressively, however it can not reintroduce a dependency failure that the deterministic passes already cleared.

The second extension is closing the hole between artificial knowledge and manufacturing actuality. The subsequent step is regenerating these identical three workload classes from precise manufacturing traces. Holding the benchmark methodology an identical—the identical fastened parameters, ground-truth dependency labels, and 15-configuration sweep—ensures the outcomes stay immediately comparable to those revealed figures whereas changing guesses with actual telemetry.

Precise utilization logs will seemingly shift the RAG and tool-agent metrics in both path. Actual-world site visitors doesn’t mechanically imply larger compression. As an illustration, a manufacturing system with aggressive upstream deduplication may already eradicate the waste this artificial mannequin assumes. That might be a extremely helpful discovering in its personal proper, quite than a failure.

A 3rd, smaller level to notice: the REF/DEFINE conference used right here is only a placeholder. A manufacturing system ought to derive these tags mechanically from structured knowledge, device name arguments, session variables, or specific consumer settings, quite than counting on guide textual content markers.

Whereas the deterministic logic in Move 3 stays an identical both means, the precise worth of this pipeline relies upon completely on how cleanly you’ll be able to generate correct dependency tags upstream. That’s an architecture-specific integration drawback quite than one thing a general-purpose pruning library can remedy out of the field.

Manufacturing serving techniques already deal with immediate bloat as a reminiscence administration drawback on the infrastructure layer, evicting and sharing KV cache pages very similar to an working system handles bodily reminiscence [4]. You’ll be able to consider this mission as working one layer above that, on the immediate development part, earlier than the request ever reaches the infrastructure.

The 2 layers don’t compete. A smaller, deduplicated immediate offers a greater enter to a well-managed cache quite than appearing as a alternative for it.

The Core Takeaway

If a crucial truth is preserved, the pipeline must display that retention quite than assume it primarily based on the absence of apparent errors. This precept guided the design of the system.

Lengthy-running conversations don’t at all times require a bigger, smarter mannequin to resolve what to recollect. Usually, the extra strong answer is a predictable, three-pass system that may programmatically show what it didn’t lose.

Sources

[1] Levy, M., Jacoby, A., & Goldberg, Y. (2024). Similar activity, extra tokens: The influence of enter size on the reasoning efficiency of huge language fashions. In Proceedings of the 62nd Annual Assembly of the Affiliation for Computational Linguistics (Quantity 1: Lengthy Papers) (pp. 15339–15353). Affiliation for Computational Linguistics. https://doi.org/10.18653/v1/2024.acl-long.818

[2] Liu, N. F., Lin, Okay., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024). Misplaced within the center: How language fashions use lengthy contexts. Transactions of the Affiliation for Computational Linguistics, 12, 157–173. https://doi.org/10.1162/tacl_a_00638

[3] Jiang, H., Wu, Q., Lin, C.-Y., Yang, Y., & Qiu, L. (2023). LLMLingua: Compressing prompts for accelerated inference of huge language fashions. In Proceedings of the 2023 Convention on Empirical Strategies in Pure Language Processing (pp. 13358–13376). Affiliation for Computational Linguistics. https://doi.org/10.18653/v1/2023.emnlp-main.825

[4] Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., & Stoica, I. (2023). Environment friendly reminiscence administration for giant language mannequin serving with PagedAttention. In Proceedings of the twenty ninth Symposium on Working Methods Rules (SOSP ’23) (pp. 611–626). Affiliation for Computing Equipment. https://doi.org/10.1145/3600006.3613165

All code, benchmark numbers, and take a look at outcomes on this article are my very own, generated by working the included codebase immediately and reproduced on two separate machines. No proprietary datasets, copyrighted textual content, or third-party code have been utilized in constructing or benchmarking this technique. All code, the corpus generator, the pruner, the benchmark harness, and all 35 assessments, is offered within the repository linked under.

https://github.com/Emmimal/prompt-pruning-layer