Tsinghua AI Open-Sources AKV, a Cache Engineering Server for LLM Brokers on SGLang

Tsinghua College and CUHK researchers open-sourced AKV, a serving system for KV cache engineering in LLM brokers. As a substitute of summarizing previous historical past, brokers drop chosen KV cache entries and maintain reusing the remaining. Throughout 3 fashions and 4 agent benchmarks, average cost falls 27.7%. Common activity rating rises 1.7 factors. The code is Apache-2.0 licensed and builds on SGLang.

TL;DR

What it’s: Serving system plus coverage library, no weights. Examined on GPT-OSS-120B (5.1B energetic), MiniMax-M2.7 (230B whole, 10B energetic) and MiroThinker-1.7-Mini (30B whole, 3B energetic).
Runs on: NVIDIA GPUs by way of SGLang. Paper used 8 A800-80GB GPUs in BF16. Fast begin serves GPT-OSS-120B on 2 GPUs (tensor parallel 2).
Efficiency: 27.7% decrease common value and +1.7 factors common rating versus threshold summarization.
Finest: GPT-OSS-120B on SWE-bench Verified: value per activity 20.0 to 9.7 cents, rating 40.0 to 43.2.
Worst: MiniMax-M2.7 on BrowseComp-Plus scores 58.9, 3.4 factors beneath Textual content Engineering’s 62.3.

What’s cache engineering, and the way is it totally different from context engineering?

AKV is a Tsinghua serving system that lets LLM brokers drop and reposition their very own KV cache.

Context engineering trims or summarizes the textual content historical past. Enhancing that prefix invalidates each later KV state. The retained suffix should then be re-prefilled. Harnesses subsequently summarize late and aggressively, which may lose particulars from center rounds.

Cache engineering leaves the textual content trajectory untouched. It solely adjustments which KV entries keep seen to consideration. Retained entries are reused with out recomputation. They’ll nonetheless carry data from dropped messages. Within the analysis paper’s value evaluation, KV cache reads made up 82.4% of whole API value.

How does AKV work?

AKV defines 3 cache operations. Append is the usual replace for brand new tokens. Drop removes chosen entries from future consideration with out altering positions. Repos rewrites retained keys to positions 0 by means of r-1 utilizing a RoPE rotation. Values keep unchanged, so no hidden states are recomputed. In ablations, including Repos raised GPT-OSS-120B’s BrowseComp-Plus cross price from 55% to 62%.

The onerous half is serving. Equivalent tokens can produce totally different cache states underneath totally different operation histories. Plain prefix matching might reuse the incorrect states. AKV extends SGLang’s radix tree with drop and reposition nodes. It frees dropped KV solely when no energetic request nonetheless wants it. Phase-batch consideration restores evicted states in 1 prefill cross.

What does the AKV API appear like?

The interface is stateless and OpenAI-compatible. Purchasers ship the complete message historical past and two non-compulsory fields. drop_message maps a set off message index to the messages to drop. reposition lists the indices after which positions get compacted.

A CachePolicy builds these fields from a set off, a selector and a reposition rule. Constructed-in insurance policies are turn-based, token-based or model-based. A CacheTool lets the mannequin name drop_context itself.

How does AKV carry out on agent benchmarks?

The principle coverage is a tool-response rolling drop that retains the most recent 12 device responses. The Normal baseline summarizes at 96K tokens (192K for MiroThinker). Textual content Engineering summarizes at 48K. Benchmarks are BrowseComp-Plus, GAIA-Text, SWE-bench Verified and xbench-DeepSearch-2510.

Mannequin	Normal rating / value	Textual content Eng. rating / value	AKV rating / value
MiniMax-M2.7	52.4 / 5.0	52.8 / 5.7	53.7 / 4.4
GPT-OSS-120B	51.0 / 11.2	48.3 / 10.2	53.1 / 7.3
MiroThinker-1.7-Mini	44.1 / 20.4	37.1 / 18.1	45.8 / 14.8

Common cross price (%) and price (US cents per activity) throughout 4 benchmarks. Supply: Table 1 of the paper.

AKV additionally cuts value 21.9% versus Textual content Engineering. The accuracy hole opens on lengthy runs. On BrowseComp-Plus at 96K tokens and above, GPT-OSS-120B scores 37% with AKV. Normal scores 24% and Textual content Engineering 32%, per the project page.

Prices are API-equivalent estimates from printed charges. Repositioned tokens are charged on the cache-hit price, a intentionally conservative selection.

<div class="hd">
<div>
<div class="tag">Tsinghua College · arXiv 2610.28577</div>
<h3>AKV: KV Cache Engineering for LLM Brokers</h3>
<p class="sub">Brokers drop and reposition KV cache as an alternative of rewriting their textual content historical past.</p>
</div>
<div class="capsule">−27.7% avg value · +1.7 pts avg rating</div>
</div>

<div class="tabs" position="tablist">
<button class="tab on" data-t="p1">1 · The thought</button>
<button class="tab" data-t="p2">2 · Cache operations</button>
<button class="tab" data-t="p3">3 · Benchmarks</button>
<button class="tab" data-t="p4">4 · Serving pace</button>
<button class="tab" data-t="p5">5 · Limits</button>
</div>

<!– P1 –>
<div class="panel on" id="mtp-akv-p1">
<div class="row"><button class="btn on" id="mtp-akv-play"> Trim previous device outcome A</button><button class="btn" id="mtp-akv-reset">Reset</button></div>
<div class="cmp" model="margin-top:12px">
<div class="field"><h4>Context engineering (summarize textual content)</h4>
<div class="lane" id="mtp-akv-ce"></div>
<div class="meter"><i id="mtp-akv-cem" model="width:0%"></i></div>
<div class="mtxt" id="mtp-akv-cet">Re-prefill work: 0 blocks</div>
</div>
<div class="field"><h4>Cache engineering (AKV drop)</h4>
<div class="lane" id="mtp-akv-ae"></div>
<div class="meter"><i id="mtp-akv-aem" model="width:0%"></i></div>
<div class="mtxt" id="mtp-akv-aet">Re-prefill work: 0 blocks</div>
</div>
</div>
<p class="word" id="mtp-akv-n1">Each brokers maintain a consumer immediate, assistant turns and power outcomes. Press the button to take away previous device outcome <b>A</b>.</p>
<div class="lg"><span><i model="background:#5a3a6a"></i>Person</span><span><i model="background:#7d2f93"></i>Assistant</span><span><i model="background:#a64bbd"></i>Instrument outcome</span><span><i model="background:#ffb347"></i>Recomputed</span><span><i model="background:#76B900"></i>New flip</span></div>
</div>

<!– P2 –>
<div class="panel" id="mtp-akv-p2">
<div class="row">
<button class="btn" data-op="append">1. Append D</button>
<button class="btn" data-op="drop">2. Drop B</button>
<button class="btn" data-op="repos">3. Repos</button>
<button class="btn" data-op="reset2">Reset</button>
</div>
<div class="lbl">Lively KV cache (quantity = place encoding index)</div>
<div class="lane" id="mtp-akv-ops"></div>
<p class="word" id="mtp-akv-n2">Begin state: tokens A, B, C sit at positions 0, 1, 2. Step by means of the three operations.</p>
<pre id="mtp-akv-req">{"messages": […], "drop_message": {}, "reposition": []}</pre>
</div>

<!– P3 –>
<div class="panel" id="mtp-akv-p3">
<div class="lbl" model="margin-top:0">Mannequin</div>
<div class="row" id="mtp-akv-models"></div>
<div class="lbl">Benchmark</div>
<div class="row" id="mtp-akv-bench"></div>
<div class="grid2" model="margin-top:14px">
<div><div class="lbl" model="margin-top:0">Go price (%) · greater is healthier</div><div class="bars" id="mtp-akv-sc"></div></div>
<div><div class="lbl" model="margin-top:0">Price (US cents per activity) · decrease is healthier</div><div class="bars" id="mtp-akv-co"></div></div>
</div>
<p class="word" id="mtp-akv-n3"></p>
</div>

<!– P4 –>
<div class="panel" id="mtp-akv-p4">
<div class="lbl" model="margin-top:0">Concurrency: <b id="mtp-akv-cv" model="coloration:#f3eaf6">14</b> · GPT-OSS-120B · 80 long-context SWE-bench Verified duties · 8 A800 GPUs</div>
<enter kind="vary" id="mtp-akv-c" min="0" max="4" step="1" worth="3">
<div class="grid2" model="margin-top:12px">
<div><div class="lbl" model="margin-top:0">P95 request latency (s) · decrease is healthier</div><div class="bars" id="mtp-akv-lat"></div></div>
<div><div class="lbl" model="margin-top:0">Output throughput (tokens/s) · greater is healthier</div><div class="bars" id="mtp-akv-out"></div></div>
</div>
<div class="stats" model="margin-top:14px">
<div class="stat"><div class="large">1.18x</div><span>Output throughput, peak vs peak</span></div>
<div class="stat"><div class="large">1.53x</div><span>Enter throughput, peak vs peak</span></div>
<div class="stat"><div class="large">1.51x</div><span>Quicker P95 latency, peak vs peak</span></div>
<div class="stat"><div class="large">2.849ms</div><span>Full consideration per decode step (was 7.082)</span></div>
</div>
<p class="word" id="mtp-akv-n4"></p>
</div>

<!– P5 –>
<div class="panel" id="mtp-akv-p5">
<ul class="lim" id="mtp-akv-lim">
<li>Few supported mannequin households<em>Drop and Repos are enabled for Qwen, GPT-OSS and MiniMax-M2 textual content architectures. Linear consideration fashions are future work.</em></li>
<li>Self-hosted solely<em>You run the AKV server your self. It extends SGLang with cache-operation assist.</em></li>
<li>Self-reported, single {hardware} setup<em>All outcomes come from the authors, on 8 A800-80GB GPUs in BF16.</em></li>
<li>Completely different measurement home windows<em>Excessive-concurrency AKV runs have been measured by means of the primary 30 activity completions. Normal used a 1-hour window.</em></li>
<li>Not each benchmark improves<em>MiniMax-M2.7 on BrowseComp-Plus scores 58.9 with AKV, beneath Textual content Engineering at 62.3.</em></li>
<li>Dropping reasoning cache hurts<em>On SWE-bench Verified, dropping reasoning cache fell to 36.5, and dropping each to 34.5, versus 41.0 for device drop.</em></li>
</ul>
<p class="word">Faucet a row to increase it.</p>
</div>

<div class="ft">
<span>Sources: <a href="https://arxiv.org/abs/2610.28577" goal="_blank" rel="noopener">Paper</a> · <a href="https://github.com/thu-nics/AKV" goal="_blank" rel="noopener">GitHub</a> · <a href="https://fuvty.github.io/thinking_yard_project_page/tasks/akv/" goal="_blank" rel="noopener">Challenge web page</a> · numbers self-reported</span>
<span class="mtp">© Marktechpost</span>
</div>

How briskly is AKV in serving benchmarks?

Serving exams ran GPT-OSS-120B on 80 long-context SWE-bench Verified duties. At every technique’s peak setting, AKV delivers 1.18x output throughput and 1.53x enter throughput. P95 request latency is 1.51x sooner.

At concurrency 16, Normal’s P95 latency hits 229.90 seconds versus 30.63 for AKV. KV strain pushes Normal into repeated re-prefill. Per decode step, full-attention time falls from 7.082 to 2.849 ms.

On 100 SWE-bench Verified duties underneath a GPU price range, AKV reached about 47% cross price. Normal reached 40.7% and Textual content Engineering 32.3%.

Can brokers tune their very own cache coverage?

MiniMax-M2.7 refined insurance policies on 20 BrowseComp-Plus duties, then ran 100 held-out duties. The iteration-6 coverage reduce value 17.8% versus Normal at baseline accuracy. The summary experiences 7.1% financial savings past the expert-designed coverage.

Exposing cache management as a device additionally helped accuracy. GPT-OSS-120B rose from 52.1 to 61.0 on a 100-task subset. MiniMax-M2.7’s value rose, although, from 5.4 to five.9 cents.

AKV vs different KV cache approaches

Metrics	AKV	Threshold summarization	SGLang RadixAttention	StreamingLLM	SnapKV
Works on	KV cache, agent-directed	Textual content historical past	KV cache, prefix reuse	KV cache, mounted window	Immediate KV cache
What will get dropped	Chosen by coverage or mannequin device name	Historical past older than a token threshold	Nothing, shared prefixes reused	Center tokens; retains sinks and up to date tokens	Low-attention immediate positions
Re-prefill after discount	No	Sure, retained suffix	Not relevant	No cache refresh wanted	Not relevant
Goal workload	Multi-turn tool-calling brokers	Agent harnesses	Common and agentic serving	Streaming dialogue	Lengthy prompts
Reported headline	-27.7% value, +1.7 pts	Baseline in AKV paper	As much as 6.4x throughput	As much as 22.2x vs sliding-window recompute	3.6x sooner era at 16K
Fashions	Qwen, GPT-OSS, MiniMax-M2	Any	Broad	Llama-2, MPT, Falcon, Pythia	Llama, Mistral, Mixtral
Effective-tuning wanted	No	No	No	No	No
License	Apache-2.0	Harness-dependent	Apache-2.0	MIT	Apache-2.0

The principle numbers come from every challenge’s personal paper and use totally different setups. They don’t seem to be instantly comparable.

Key Takeaways

AKV lets brokers drop and reposition KV cache as an alternative of rewriting historical past.
Common value falls 27.7% and rating rises 1.7 factors throughout 3 fashions.
Repos lifts GPT-OSS-120B on BrowseComp-Plus from 55% to 62%.
At concurrency 16, P95 latency drops from 229.90 to 30.63 seconds.
Assist covers solely Qwen, GPT-OSS and MiniMax-M2 textual content fashions as we speak.

Take a look at the Paper, GitHub Repo and Project Page. Additionally, be happy to observe us on Twitter and don’t overlook to affix our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.

Have to accomplice with us for selling your GitHub Repo OR Hugging Face Web page OR Product Launch OR Webinar and so forth.? Connect with us

The submit Tsinghua AI Open-Sources AKV, a Cache Engineering Server for LLM Brokers on SGLang appeared first on MarkTechPost.

Tsinghua AI Open-Sources AKV, a Cache Engineering Server for LLM Brokers on SGLang

TL;DR

What’s cache engineering, and the way is it totally different from context engineering?

How does AKV work?

What does the AKV API appear like?

How does AKV carry out on agent benchmarks?

How briskly is AKV in serving benchmarks?

Can brokers tune their very own cache coverage?

AKV vs different KV cache approaches

Key Takeaways

Leases Helped Me Stop My 14-Yr Nursing Job (And Begin My Personal Enterprise)

Converter

Editors Pick

Newsletter

Categories

Related Posts

Leave a Comment Cancel Reply