NVIDIA researchers, with Princeton College and the College of Maryland, have launched PivotOPD, an on-policy distillation technique for multi-turn LLM brokers. PivotOPD on-policy distillation trains an agent to keep away from its most damaging early mistake, and to get better when it occurs anyway. In opposition to 13 baselines, it posts one of the best common on ALFWorld, WebShop and Search-based QA for Qwen3-1.7B and Qwen3-8B college students. The takeaway: restoration is learnable, and normal OPD not often teaches it.
TL;DR
- Dimension: A coaching technique, not a mannequin. Examined on Qwen3-1.7B and Qwen3-8B college students, plus a Nemotron-3.5-SFT pupil on SWE-Bench Verified.
- Runs on: Skilled on NVIDIA H100 nodes. Provides 0 inference price, so the educated agent runs wherever its base mannequin runs.
- Efficiency: First on all 8 per-benchmark averages towards 13 baselines, throughout 3 seeds.
- Greatest: Recovers from 72.7% of replayed pivotal errors, vs 20.3% for normal OPD.
- Worst: 55.9% on ALFWorld “Look” duties with the 1.7B pupil, vs 83.9% for SOD.
- Backside line: Greatest: teaches restoration that outcome-only RL can not attain. Worst: relies on replayable environments and a trainer whose pivots match the oracle in 77.8% of failed rollouts.
What’s a pivotal mistake in a multi-turn agent?
A pivotal mistake is an motion that lengthens the shortest remaining path to ending a process, or makes it unsolvable. ALFWorld’s symbolic oracle measures this at each flip.
Throughout Qwen3-8B, Qwen3-30B-A3B and Qwen3-235B-A22B, 59% of failed rollouts (155 of 262) contained one. The primary pivotal flip arrived early, at a median of flip 8 to 12 out of 30. The brokers then wasted 18 to 21 extra turns with out recovering.
In replays of Qwen3-8B failures, correcting the pivotal flip raised success from 8% to 59%. Leaving the error in place and forcing the suitable motion for the following 2 turns nonetheless reached 58%.
Why does normal on-policy distillation miss it?
Customary OPD lowered the held-out failure charge from 79% to 56%. Failures after a pivotal flip solely fell from 51% to 49%. The right motion stayed beneath 1% likelihood at each pivotal flip, so 8 rollouts not often pattern it.
Final result-based RL shares the blind spot: if each rollout fails, the group-relative benefit is 0.
How does PivotOPD work?
PivotOPD provides 3 elements to group-based RL, mixed in a single PPO replace.
- Pivot detection: A bigger trainer mannequin reads every rollout and its final result in hindsight. It picks candidate turns and names a gold motion at every. A flip counts as pivotal when the coed’s motion differs from the gold motion. On ALFWorld, detected pivots land inside 1 flip of the oracle’s pivot in 77.8% of failed rollouts on common.
- Preventive distillation (reverse KL): A frozen copy of the coed, hinted with the gold motion, re-scores the coed’s personal response. This pushes the coed away from the dedicated mistake.
- Restoration distillation (ahead KL): After every pivot, the trainer names a restoration motion for as much as Okay turns. The hinted self-teacher writes restoration responses, and the unhinted pupil trains on them. Ahead KL is mass-covering, so it lifts actions the coed nearly by no means samples.
The trainer solely names actions. Token-level targets come from the coed’s personal hinted distribution.
How does PivotOPD carry out on agent benchmarks?
With the 1.7B pupil, PivotOPD averages 73.7% on ALFWorld, 5.5 factors above SDAR. It averages 44.5% on Search-based QA, 5.9 factors above RLSD. On WebShop it beats RLSD by 1.2 in rating however by 14.1 in success charge (76.6%).
With the 8B pupil, it reaches 93.0% on ALFWorld, 47.4% on Search-based QA and 81.9% WebShop success. Margins are smaller, no less than 1.8 factors.
With Qwen3-8B as its personal trainer, PivotOPD nonetheless wins all 3 benchmarks by no less than 1.5 factors, 3.9 on common.
On SWE-Bench Verified, a Nemotron-3.5-SFT pupil taught by Nemotron-3-Tremendous went from 62.8% to 66.0%. Customary OPD reached 63.0%, and the trainer scores 73.0%.
Restoration is the standout. Throughout 72 replayed pivotal errors, PivotOPD recovered 72.7% of the time, vs 8.3% for the bottom mannequin, 20.3% for normal OPD and 45.8% for preventive-only. It averaged 9.7 turns to get better, towards an optimum 6.2.
<iframe id="mtp-pivotopd-frame" title="PivotOPD interactive explainer" loading="lazy" peak="600" fashion="peak:600px;" srcdoc="<!DOCTYPE html>
<html lang="en"><head><meta charset="utf-8"><meta identify="viewport" content material="width=device-width,initial-scale=1">
<title>PivotOPD Explainer</title>
<fashion>
html,physique{margin:0;padding:0;background:#111!vital;}
#mtp-pivotopd{max-width:880px;margin:0 auto;background:#111!vital;coloration:#e8e8e8!vital;font-family:Inter,"Segoe UI",system-ui,sans-serif;border:1px strong #2a2a2a!vital;border-radius:14px;overflow:hidden;box-sizing:border-box;}
#mtp-pivotopd *{box-sizing:border-box;}
#mtp-pivotopd hr,#mtp-pivotopd p:empty,#mtp-pivotopd del,#mtp-pivotopd s{show:none!vital;}
#mtp-pivotopd .hd{padding:20px 24px 14px;border-bottom:1px strong #222!vital;background:linear-gradient(135deg,#151a0c,#111)!vital;}
#mtp-pivotopd .kick{font-size:11px;letter-spacing:.14em;text-transform:uppercase;coloration:#76B900!vital;font-weight:700;}
#mtp-pivotopd h2{margin:6px 0 4px;font-size:22px;coloration:#fff!vital;line-height:1.25;}
#mtp-pivotopd .sub{margin:0;font-size:13.5px;coloration:#a8a8a8!vital;}
#mtp-pivotopd .tabs{show:flex;hole:6px;padding:12px 16px;background:#141414!vital;border-bottom:1px strong #222!vital;flex-wrap:wrap;}
#mtp-pivotopd .tab{background:#1c1c1c!vital;coloration:#cfcfcf!vital;border:1px strong #2c2c2c!vital;border-radius:999px;padding:7px 14px;font-size:13px;cursor:pointer;font-family:inherit;transition:all .2s;}
#mtp-pivotopd .tab:hover{border-color:#76B900!vital;coloration:#fff!vital;}
#mtp-pivotopd .tab.on{background:#76B900!vital;coloration:#0b0b0b!vital;border-color:#76B900!vital;font-weight:700;}
#mtp-pivotopd .panel{show:none;padding:20px 24px 22px;animation:mtpfade .35s ease;}
#mtp-pivotopd .panel.on{show:block;}
@keyframes mtpfade{from{opacity:0;rework:translateY(6px)}to{opacity:1;rework:none}}
#mtp-pivotopd p{font-size:14px;line-height:1.55;coloration:#d6d6d6!vital;margin:0 0 12px;}
#mtp-pivotopd .muted{coloration:#8f8f8f!vital;font-size:12px;}
#mtp-pivotopd .turns{show:flex;hole:4px;margin:14px 0 8px;flex-wrap:wrap;}
#mtp-pivotopd .t{width:22px;peak:22px;border-radius:5px;background:#2a2a2a!vital;show:flex;align-items:heart;justify-content:heart;font-size:10px;coloration:#777!vital;transition:background .25s,rework .25s;}
#mtp-pivotopd .t.okay{background:#4a5a7a!vital;coloration:#e6ecff!vital;}
#mtp-pivotopd .t.pv{background:#e0603a!vital;coloration:#fff!vital;rework:scale(1.15);}
#mtp-pivotopd .t.wa{background:#3a2a2a!vital;coloration:#b88!vital;}
#mtp-pivotopd .t.rc{background:#76B900!vital;coloration:#0b0b0b!vital;}
#mtp-pivotopd .legend{show:flex;hole:14px;flex-wrap:wrap;font-size:12px;coloration:#aaa!vital;margin-bottom:10px;}
#mtp-pivotopd .legend i{show:inline-block;width:10px;peak:10px;border-radius:3px;margin-right:5px;vertical-align:-1px;}
#mtp-pivotopd .btn{background:clear!vital;coloration:#76B900!vital;border:1px strong #76B900!vital;border-radius:8px;padding:7px 14px;font-size:13px;cursor:pointer;font-family:inherit;margin-right:6px;}
#mtp-pivotopd .btn:hover{background:#76B900!vital;coloration:#0b0b0b!vital;}
#mtp-pivotopd .stats{show:grid;grid-template-columns:repeat(3,1fr);hole:10px;margin-top:14px;}
#mtp-pivotopd .stat{background:#181818!vital;border:1px strong #262626!vital;border-radius:10px;padding:12px;}
#mtp-pivotopd .stat b{show:block;font-size:24px;coloration:#76B900!vital;}
#mtp-pivotopd .stat span{font-size:12px;coloration:#aaa!vital;line-height:1.35;show:block;}
#mtp-pivotopd .steps{show:grid;grid-template-columns:repeat(3,1fr);hole:10px;margin:6px 0 14px;}
#mtp-pivotopd .step{background:#181818!vital;border:1px strong #262626!vital;border-radius:10px;padding:12px;cursor:pointer;transition:all .25s;opacity:.55;}
#mtp-pivotopd .step.on{opacity:1;border-color:#76B900!vital;box-shadow:0 0 0 1px #76B900 inset,0 0 18px rgba(118,185,0,.18);}
#mtp-pivotopd .step .n{font-size:11px;coloration:#76B900!vital;font-weight:700;letter-spacing:.1em;}
#mtp-pivotopd .step h4{margin:4px 0 4px;font-size:14px;coloration:#fff!vital;}
#mtp-pivotopd .step small{font-size:12px;coloration:#9a9a9a!vital;}
#mtp-pivotopd .element{background:#151a0c!vital;border:1px strong #2f4200!vital;border-radius:10px;padding:14px;min-height:96px;}
#mtp-pivotopd .element h5{margin:0 0 6px;font-size:14px;coloration:#9fe02a!vital;}
#mtp-pivotopd .stream{show:flex;align-items:heart;hole:6px;margin:10px 0 2px;flex-wrap:wrap;font-size:12px;}
#mtp-pivotopd .chip{padding:4px 9px;border-radius:6px;background:#222!vital;coloration:#ddd!vital;border:1px strong #333!vital;}
#mtp-pivotopd .chip.g{background:#76B900!vital;coloration:#0b0b0b!vital;border-color:#76B900!vital;font-weight:700;}
#mtp-pivotopd .chip.r{background:#e0603a!vital;coloration:#fff!vital;border-color:#e0603a!vital;}
#mtp-pivotopd .arr{coloration:#666!vital;}
#mtp-pivotopd .tog{show:inline-flex;border:1px strong #333!vital;border-radius:8px;overflow:hidden;margin-bottom:12px;}
#mtp-pivotopd .tog button{background:#1a1a1a!vital;coloration:#bbb!vital;border:0!vital;padding:6px 12px;font-size:12px;cursor:pointer;font-family:inherit;}
#mtp-pivotopd .tog button.on{background:#76B900!vital;coloration:#0b0b0b!vital;font-weight:700;}
#mtp-pivotopd .bench{margin-bottom:14px;}
#mtp-pivotopd .bench h6{margin:0 0 6px;font-size:12.5px;coloration:#cfcfcf!vital;font-weight:600;}
#mtp-pivotopd .row{show:flex;align-items:heart;hole:8px;margin:4px 0;font-size:12px;}
#mtp-pivotopd .row .lb{width:92px;coloration:#aaa!vital;flex:none;}
#mtp-pivotopd .row .tr{show:block;flex:1;peak:16px;background:#1d1d1d!vital;border-radius:4px;overflow:hidden;}
#mtp-pivotopd .row .bar{show:block;peak:100%;width:0;background:#4a5a7a!vital;border-radius:4px;transition:width 1s cubic-bezier(.2,.8,.2,1);}
#mtp-pivotopd .row.me .bar{background:#76B900!vital;}
#mtp-pivotopd .row.me .lb{coloration:#76B900!vital;font-weight:700;}
#mtp-pivotopd .row .v{width:40px;text-align:proper;coloration:#ddd!vital;flex:none;}
#mtp-pivotopd .case{show:grid;grid-template-columns:1fr 1fr;hole:10px;margin-top:12px;}
#mtp-pivotopd .col{background:#181818!vital;border:1px strong #262626!vital;border-radius:10px;padding:10px;}
#mtp-pivotopd .col h6{margin:0 0 6px;font-size:12px;coloration:#fff!vital;}
#mtp-pivotopd .ln{font-size:12px;padding:5px 7px;border-radius:6px;margin:3px 0;coloration:#ccc!vital;background:#202020!vital;opacity:0;rework:translateX(-6px);transition:all .3s;}
#mtp-pivotopd .ln.present{opacity:1;rework:none;}
#mtp-pivotopd .ln.dangerous{background:#3a1d16!vital;coloration:#f3b5a2!vital;}
#mtp-pivotopd .ln.good{background:#1f2c08!vital;coloration:#c6f07a!vital;}
#mtp-pivotopd .ft{padding:10px 24px;border-top:1px strong #222!vital;font-size:12px;coloration:#777!vital;show:flex;justify-content:space-between;flex-wrap:wrap;hole:6px;background:#0f0f0f!vital;}
#mtp-pivotopd .ft b{coloration:#76B900!vital;}
#mtp-pivotopd a{coloration:#76B900!vital;text-decoration:none;}
@media (max-width:640px){
#mtp-pivotopd .hd,#mtp-pivotopd .panel{padding-left:16px;padding-right:16px;}
#mtp-pivotopd h2{font-size:19px;}
#mtp-pivotopd .stats,#mtp-pivotopd .steps,#mtp-pivotopd .case{grid-template-columns:1fr;}
#mtp-pivotopd .t{width:18px;peak:18px;font-size:9px;}
#mtp-pivotopd .row .lb{width:74px;}
}
</fashion></head><physique>
<div id="mtp-pivotopd">
<div class="hd">
<div class="kick">NVIDIA Analysis · Interactive explainer</div>
<h2>PivotOPD: educating brokers to get better from pivotal errors</h2>
<p class="sub">On-policy distillation that stops early agent errors and trains restoration from them.</p>
</div>
<div class="tabs" position="tablist">
<button class="tab on" data-p="0">1. The issue</button>
<button class="tab" data-p="1">2. The way it works</button>
<button class="tab" data-p="2">3. Benchmarks</button>
<button class="tab" data-p="3">4. Restoration</button>
</div>
<div class="panel on" data-p="0">
<p>In ALFWorld, 59% of failed rollouts throughout 3 Qwen3 fashions include a <b fashion="coloration:#e0603a">pivotal mistake</b>: an motion that lengthens the shortest path to ending the duty. It lands early, then the agent wastes the remainder of the 30-turn episode.</p>
<div class="legend"><span><i fashion="background:#4a5a7a"></i>Right flip</span><span><i fashion="background:#e0603a"></i>Pivotal mistake</span><span><i fashion="background:#3a2a2a"></i>Wasted flip</span><span><i fashion="background:#76B900"></i>Restoration</span></div>
<div class="turns" id="mtp-turns"></div>
<div fashion="margin:10px 0 2px"><button class="btn" id="mtp-play1">Play failed rollout</button><button class="btn" id="mtp-play2">Play with restoration</button></div>
<p class="muted" id="mtp-cap">Illustrative 30-turn episode. Pivot proven at flip 10, contained in the reported median vary of 8 to 12.</p>
<div class="stats">
<div class="stat"><b data-c="59">0%</b><span>replayed success after fixing the pivotal flip (from 8%)</span></div>
<div class="stat"><b data-c="58">0%</b><span>success when solely the two turns after the error are guided</span></div>
<div class="stat"><b>51% → 49%</b><span>pivotal-turn failures after normal OPD barely transfer</span></div>
</div>
</div>
<div class="panel" data-p="1">
<p>PivotOPD provides 3 elements to group-based RL, all utilized in a single PPO replace. Click on a step or use the buttons.</p>
<div class="steps">
<div class="step on" data-s="0"><div class="n">STEP 01</div><h4>Pivot detection</h4><small>Trainer finds the flip and the gold motion</small></div>
<div class="step" data-s="1"><div class="n">STEP 02</div><h4>Preventive distillation</h4><small>Reverse KL, on the coed's personal response</small></div>
<div class="step" data-s="2"><div class="n">STEP 03</div><h4>Restoration distillation</h4><small>Ahead KL, on self-teacher responses</small></div>
</div>
<div class="element" id="mtp-detail"></div>
<div fashion="margin-top:12px"><button class="btn" id="mtp-prev"> Prev</button><button class="btn" id="mtp-next">Subsequent 
</button></div>
</div>
<div class="panel" data-p="2">
<div class="tog" id="mtp-tog"><button class="on" data-m="s">Qwen3-1.7B</button><button data-m="l">Qwen3-8B</button></div>
<div id="mtp-bench"></div>
<p class="muted">Averages over 3 seeds from Desk 1 of the paper. In contrast with the strongest baselines. PivotOPD ranks first on all 8 per-benchmark averages towards 13 baselines.</p>
<div class="bench"><h6>SWE-Bench Verified resolve charge (Nemotron-3.5-SFT pupil)</h6><div id="mtp-swe"></div></div>
</div>
<div class="panel" data-p="3">
<p>72 oracle-labeled pivotal errors, replayed 8 instances per coverage. How usually does every coverage end the duty anyway?</p>
<div id="mtp-rec"></div>
<p class="muted">Common turns to get better: PivotOPD 9.7, normal OPD 12.3, base 13.4, optimum 6.2.</p>
<button class="btn" id="mtp-case">Replay case examine</button>
<div class="case">
<div class="col"><h6>Base mannequin</h6><div id="mtp-ca"></div></div>
<div class="col"><h6 fashion="coloration:#76B900">PivotOPD mannequin</h6><div id="mtp-cb"></div></div>
</div>
</div>
<div class="ft"><span>Supply: <a href="https://arxiv.org/abs/2609.40285" goal="_blank" rel="noopener">arXiv 2609.40285</a></span><b>© Marktechpost</b></div>
</div>
<script>
(operate(){
var R=doc.getElementById(‘mtp-pivotopd’);
operate $(s){return R.querySelector(s);} operate $$(s){return R.querySelectorAll(s);}
operate resize(){attempt{guardian.postMessage({mtpPivotOPDHeight:R.offsetHeight+40},’*’);}catch(e){}}
var proven={};
$$(‘.tab’).forEach(operate(b){b.onclick=operate(){
$$(‘.tab’).forEach(operate(x){x.classList.take away(‘on’)});b.classList.add(‘on’);
var p=b.getAttribute(‘data-p’);
$$(‘.panel’).forEach(operate(x){x.classList.toggle(‘on’,x.getAttribute(‘data-p’)===p)});
if(p===’2′)bench(mode); if(p===’3′)rec(); setTimeout(resize,50);};});
/* panel 1 */
var T=$(‘#mtp-turns’),cells=[];
for(var i=1;i<=30;i++){var d=doc.createElement(‘div’);d.className=”t”;d.textContent=i;T.appendChild(d);cells.push(d);}
var timer;
operate play(get better){clearInterval(timer);cells.forEach(operate(c){c.className=”t”});var okay=0;
timer=setInterval(operate(){var c=cells[k];var n=okay+1;
if(n<10)c.className=”t okay”;else if(n===10)c.className=”t pv”;
else if(get better){c.className=n<=12?’t rc’:(n<=16?’t okay’:’t’);}
else c.className=”t wa”;
okay++; if(get better&&n===16){clearInterval(timer);$(‘#mtp-cap’).textContent=”Guided restoration after the error, then process accomplished. Replays present 58% success with 2 guided turns.”;}
if(okay>=30){clearInterval(timer);$(‘#mtp-cap’).textContent=”No restoration: the remaining 20 turns are wasted. Actual brokers wasted 18 to 21 activates common.”;}
},90);}
$(‘#mtp-play1’).onclick=operate(){play(false)};$(‘#mtp-play2’).onclick=operate(){play(true)};
operate rely(){ $$(‘.stat b[data-c]’).forEach(operate(b){var t=+b.getAttribute(‘data-c’),v=0;var iv=setInterval(operate(){v+=2;if(v>=t){v=t;clearInterval(iv);}b.textContent=v+’%’;},20);});}
/* panel 2 */
var S=[
{h:’1. Pivot detection’,b:’A larger teacher reads each rollout and its outcome in hindsight, picks candidate turns, and names a gold action. A turn is pivotal when the student’s action differs. Detected pivots land within 1 turn of the oracle’s in 77.8% of failed ALFWorld rollouts on average.’,f:[‘<span class="chip">Rollout + outcome</span>’,'<span class="arr">→</span>’,'<span class="chip">Teacher model</span>’,'<span class="arr">→</span>’,'<span class="chip r">Pivotal turn t</span>’,'<span class="chip g">Gold action</span>’]},
{h:’2. Preventive distillation (reverse KL)’,b:’The frozen pupil, hinted with the gold motion, turns into a privileged self-teacher. It re-scores the response the coed really wrote, which shifts likelihood away from the error. Weight w_prev is saved small (0.001).’,f:[‘<span class="chip">Student response at t</span>’,'<span class="arr">→</span>’,'<span class="chip">Self-teacher + hint(gold)</span>’,'<span class="arr">→</span>’,'<span class="chip g">Reverse KL update</span>’]},
{h:’3. Restoration distillation (ahead KL)’,b:’After the pivot, the trainer names a restoration motion for every of the following Okay turns. The hinted self-teacher writes the restoration response and the unhinted pupil trains on it. Ahead KL is mass-covering, so it lifts actions the coed nearly by no means samples. Greatest Okay: 2 on ALFWorld, 1 on WebShop and Search QA.’,f:[‘<span class="chip r">Post-mistake state</span>’,'<span class="arr">→</span>’,'<span class="chip">Teacher: recovery action</span>’,'<span class="arr">→</span>’,'<span class="chip">Self-teacher writes response</span>’,'<span class="arr">→</span>’,'<span class="chip g">Forward KL update</span>’]}
];
var si=0;
operate step(i){si=(i+3)%3;$$(‘.step’).forEach(operate(x){x.classList.toggle(‘on’,+x.getAttribute(‘data-s’)===si)});
var s=S[si];$(‘#mtp-detail’).innerHTML='<h5>’+s.h+'</h5><p fashion="margin:0">’+s.b+'</p><div class="stream">’+s.f.be part of(”)+'</div>’;setTimeout(resize,30);}
$$(‘.step’).forEach(operate(x){x.onclick=operate(){step(+x.getAttribute(‘data-s’))}});
$(‘#mtp-prev’).onclick=operate(){step(si-1)};$(‘#mtp-next’).onclick=operate(){step(si+1)};step(0);
/* panel 3 */
var B={s:[
[‘ALFWorld avg success (%)’,[[‘PivotOPD’,73.7],[‘SDAR’,68.2],[‘OPID’,61.3],[‘RLSD’,60.7],[‘GRPO’,42.7]]],
[‘Search-based QA avg accuracy (%)’,[[‘PivotOPD’,44.5],[‘RLSD’,38.6],[‘SOD’,38.2],[‘GRPO’,37.7]]],
[‘WebShop success rate (%)’,[[‘PivotOPD’,76.6],[‘OPID’,68.0],[‘RLSD’,62.5],[‘GRPO’,57.0]]]],
l:[
[‘ALFWorld avg success (%)’,[[‘PivotOPD’,93.0],[‘RLSD’,90.8],[‘Skill-SD’,88.6],[‘GRPO’,35.3]]],
[‘Search-based QA avg accuracy (%)’,[[‘PivotOPD’,47.4],[‘OPID’,45.6],[‘OPSD’,43.3],[‘GRPO’,37.5]]],
[‘WebShop success rate (%)’,[[‘PivotOPD’,81.9],[‘SDAR’,79.7],[‘RLSD’,78.9],[‘GRPO’,75.0]]]]};
operate bars(el,rows,max){el.innerHTML=rows.map(operate(r){return ‘<div class="row’+(r[0]===’PivotOPD’?’ me’:”)+’"><span class="lb">’+r[0]+'</span><span class="tr"><span class="bar" data-w="’+(r[1]/max*100)+’"></span></span><span class="v">’+r[1].toFixed(1)+'</span></div>’}).be part of(”);
requestAnimationFrame(operate(){setTimeout(operate(){el.querySelectorAll(‘.bar’).forEach(operate(b){b.fashion.width=b.getAttribute(‘data-w’)+’%’})},40)});}
var mode=”s”;
operate bench(m){mode=m;var W=$(‘#mtp-bench’);W.innerHTML=”;B[m].forEach(operate(g){var d=doc.createElement(‘div’);d.className=”bench”;d.innerHTML='<h6>’+g[0]+'</h6><div></div>’;W.appendChild(d);bars(d.lastChild,g[1],100);});
bars($(‘#mtp-swe’),[[‘Teacher’,73.0],[‘PivotOPD’,66.0],[‘Std OPD’,63.0],[‘Student’,62.8]],100);setTimeout(resize,60);}
$$(‘#mtp-tog button’).forEach(operate(b){b.onclick=operate(){$$(‘#mtp-tog button’).forEach(operate(x){x.classList.take away(‘on’)});b.classList.add(‘on’);bench(b.getAttribute(‘data-m’));}});
/* panel 4 */
operate rec(){bars($(‘#mtp-rec’),[[‘PivotOPD’,72.7],[‘Prev. only’,45.8],[‘Std OPD’,20.3],[‘Base’,8.3]],100);setTimeout(resize,60);}
var CA=[[‘Turns 1-2: no egg in the fridge’,”],[‘Turn 3: takes tomato from fridge’,’bad’],[‘Turn 4: goes to microwave’,”],[‘Turn 5: moves tomato to microwave’,’bad’],[‘Turns 6-30: never finds the egg’,’bad’],[‘✕ Failure’,’bad’]];
var CB=[[‘Turns 1-2: no egg in the fridge’,”],[‘Turn 3: takes tomato from fridge’,’bad’],[‘Turn 4: recovery begins, goes to table’,’good’],[‘Turn 5: sets the tomato aside’,’good’],[‘Turn 6: takes egg from table’,’good’],[‘Turns 7-11: cools egg, moves it to microwave’,’good’],[‘✓ Success’,’good’]];
operate fill(el,arr){el.innerHTML=arr.map(operate(a){return ‘<div class="ln ‘+a[1]+’">’+a[0]+'</div>’}).be part of(”);}
operate caseplay(){fill($(‘#mtp-ca’),CA);fill($(‘#mtp-cb’),CB);var a=$$(‘#mtp-ca .ln’),b=$$(‘#mtp-cb .ln’);var okay=0;var iv=setInterval(operate(){if(a[k])a[k].classList.add(‘present’);if(b[k])b[k].classList.add(‘present’);okay++;if(okay>7)clearInterval(iv);},380);setTimeout(resize,60);}
$(‘#mtp-case’).onclick=caseplay;
fill($(‘#mtp-ca’),CA);fill($(‘#mtp-cb’),CB);$$(‘#mtp-pivotopd .ln’).forEach(operate(x){x.classList.add(‘present’)});
rely();
window.addEventListener(‘load’,resize);window.addEventListener(‘resize’,resize);setTimeout(resize,300);
})();
</script>
</physique></html>
“>
window.addEventListener(“message”,operate(e){if(e.information&&e.information.mtpPivotOPDHeight){var f=doc.getElementById(“mtp-pivotopd-frame”);if(f&&e.supply===f.contentWindow){f.fashion.peak=e.information.mtpPivotOPDHeight+”px”;}}});
How does PivotOPD examine with different agent distillation strategies?
| Metrics | PivotOPD | OPSD (standard OPD) | OPID | SDAR | RLSD |
|---|---|---|---|---|---|
| Core concept | Stop and get better at teacher-detected pivotal turns | Privileged self-teacher on each response | Hierarchical expertise from on-policy hindsight | OPSD as sigmoid-gated auxiliary loss | Self-distillation units replace measurement, RLVR units path |
| Trains on teacher-written restoration responses | Sure (ahead KL) | No | No | No | No |
| ALFWorld avg, Qwen3-1.7B | 73.7 | 12.4 | 61.3 | 68.2 | 60.7 |
| Search QA avg, Qwen3-1.7B | 44.5 | 36.8 | 37.5 | 37.1 | 38.6 |
| WebShop success, Qwen3-1.7B | 76.6 | 10.1 | 68.0 | 57.0 | 62.5 |
| ALFWorld avg, Qwen3-8B | 93.0 | 46.7 | 83.7 | 83.8 | 90.8 |
| Code | Coming soon | GitHub | GitHub | Not disclosed | Not disclosed |
Rating supply: PivotOPD Table 1.
What does it price to coach, and may you run it?
PivotOPD adjustments solely coaching, so inference prices nothing additional. On ALFWorld with the 1.7B pupil, on 4 H100 GPUs, the overhead over GRPO is 12.4% at Okay = 1. It jumps to 94.2% at Okay = 2, as a result of later restoration turns want setting replay. The chosen funds is Okay = 2 on ALFWorld and Okay = 1 on WebShop and Search-based QA.
Key Takeaways
- Over half of failed agent rollouts hinge on 1 early, recoverable mistake.
- Customary OPD barely touches these failures: 51% to 49%.
- PivotOPD recovers from 72.7% of replayed errors, vs 20.3% for OPD.
- Greatest common towards 13 baselines on ALFWorld, WebShop and Search QA.
- 0 inference overhead, however code isn’t but launched.
Take a look at the paper and the project page. All credit score goes to the researcher of this mission. Additionally, be happy to observe us on Twitter and don’t neglect to affix our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
The submit NVIDIA PivotOPD Teaches Multi-Flip AI Brokers to Recuperate From Pivotal Errors appeared first on MarkTechPost.
