Agent LLMs typically fail in the identical manner again and again. The Stanford analysis staff discovered that the offender was a scarcity of reusable performance. their system is hintdiagnose these gaps and prepare them immediately.
Hint is an abbreviation of Tbreak bones Rcommon agentleman’s failure CCompetency-targeted coaching Esurroundings. Launched as open supply underneath the MIT license.
What drawback does TRACE remedy?
To know the design, first think about why the agent fails. They lack sure expertise wanted for the duty, similar to retrieving the suitable data or validating assumptions.
Two mainstream fixes cut back computational energy. Direct RL or SFT provides sparse rewards that by no means reveal which expertise have been lacking. In depth artificial information is untargeted, so your finances is directed towards expertise your fashions have already got.
Nevertheless, TRACE observes that failures usually are not random. Most failed trajectories are attributable to a collection of small deficiencies. Subsequently, every repeated deficit can turn out to be its personal dense and verifiable coaching sign.
How does tracing work?
Contemplating this outcome, TRACE runs an automatic four-step pipeline. Every step is carried out by the LLM agent in accordance with markdown prompts.
Step 1: Contrastive competency evaluation
The bottom agent generates the rollout within the goal surroundings. The evaluation agent splits them into success and failure units. Then label all trajectory-ability pairs as follows: NA, PRESENTor LACKING.
Options are preserved solely when distinction is excessive and protection is excessive. Particularly, the distinction hole ought to clear δ = 0.20 and the protection ought to clear ρ = 0.10. In consequence, the pipeline retains expertise which are lacking and failing.
Step 2: Synthesizing the goal surroundings
The era agent then builds one synthesis surroundings for every retained characteristic. Every surroundings isolates a single performance whereas sustaining the schema and format of the goal instrument.
Process situations are procedurally generated from random seeds. Technology and validation are algorithmic, so rewards don’t require human labels or LLM judges.
Step 3: Coaching the characteristic adapter
Every characteristic then will get one LoRA (low-rank adaptation) adapter skilled in an artificial surroundings. The coaching algorithm is GRPO (Group Relative Coverage Optimization). The bottom mannequin stays solely frozen.
GRPO teams rollouts by shared seed, so the eventualities inside a bunch are equivalent. The rewards are then normalized inside every group to isolate the contribution of the coverage.
Step 4: MoE configuration with token-level routing
Lastly, TRACE configures the adapter right into a Combination-of-Consultants (MoE) mannequin. The spine and adapters stay frozen and solely the light-weight token-level gates are skilled.
Throughout inference, every token is routed from prime 1 to a single characteristic adapter. This permits the mannequin to change specialists mid-trajectory.
interactive explainer
TRACE diagnoses lacking capabilities in brokers, builds one verifiable surroundings for every hole, trains every LoRA professional, after which routes tokens between specialists. Step by way of the pipeline under.
1 · Contrastive competency evaluation
Cut up rollouts into cross/fail and keep gaps separating them.
Retained if Δ ≥ 0.20 and Cov ≥ 0.10.
2 · Synthesis of goal surroundings
One seeded, mechanically verifiable surroundings is generated for every characteristic.
3 · Competency Adapter Coaching (GRPO)
Rollouts share seeds. Rewards are normalized inside teams.
🔒 Base mannequin is frozen · Solely Δc is up to date
4 · MoE configuration · Token-level routing
The discovered gate routes the highest 1 of every token to a single characteristic professional.
Choose the duty above to route that token.
Outcomes・τ²-Bench total cross price (Qwen3-30B-A3B)
Focused coaching and MoE configuration outperform immediate optimization and single adapter baselines.
What options did we discover?
In truth, on the τ² bench, 4 deficits have been recovered against this evaluation. These have been structured information inference, multi-step job completion, assumption validation, and gear invocation accuracy.
Remarkably, this outcome was steady over 10 unbiased runs. Structured information inference alone lined the biggest portion of failed duties. Completion of multi-step duties adopted intently behind.
Utilization and examples
To make these capabilities stick, think about three particular duties. Every maps to a separate failure mode that’s lined by TRACE.
- First, think about an airline agent liable for customer support. A person requested to cancel a fundamental financial system flight booked 14 days in the past. Prerequisite validation checks insurance coverage eligibility earlier than making a name.
cancel_reservation. - Now think about a composite retail request. A person canceled two reservations and requested a change to a 3rd reservation. Finishing a multi-step job prevents the agent from exiting after the primary subtask.
- Lastly, think about SWE bench-validated coding brokers. Discovering the related operate or file appropriately is a characteristic. Required for bug fixes and API name updates.
How do the outcomes examine?
With these options skilled, TRACE was examined on two backbones and two benchmarks. τ²-Bench measures customer support cross charges throughout 50 airways and 114 retail operations. SWE Bench Validated measures Cross@1 on 500 actual GitHub points.
| spine | methodology | τ²-Complete Bench (%) | SWE Bench Validated Path@1 (%) |
|---|---|---|---|
| Quen 3-30B-A3B | base | 32.9 | 26.0 |
| GEPA (immediate optimization) | 39.6 | 31.0 | |
| SWE-RL | — | 32.6 | |
| Single operate GRPO (our firm) | 40.3 | 36.6 | |
| hint (ours) | 48.2 | 41.0 | |
| Quen 3.6-27B | base | 50.0 | 68.0 |
| GEPA (immediate optimization) | 53.0 | 69.6 | |
| SWE-RL | — | 70.4 | |
| hint (ours) | 59.1 | 73.2 |
For Qwen3-30B-A3B, TRACE improved τ²-Bench by +15.3 factors and SWE-bench Verified improved by +15 factors Cross@1. It outperformed the strongest exterior baselines, GEPA and SWE-RL, by +8.6 and +8.4 factors.
Moreover, TRACE has good pattern effectivity. We beat the ultimate scores of GRPO and GEPA utilizing lower than 1 / 4 of the rollout. The ultimate accuracy improved by +10.4 factors and +8.6 factors on the τ² bench. Moreover, a 3rd benchmark, ToolSandBox, confirmed the identical sample.
On Qwen3.6-27B, TRACE reached 73.2% Cross@1 in SWE bench validation. This 27B open weight mannequin topped GPT-5.2-Codex (72.8%), GLM 5, and Claude 4.5 Sonnet on the general public leaderboard.
implementation
For AI professionals, pipelines are benchmark-agnostic and pushed by markdown prompts. After the surroundings is generated, every characteristic adapter is skilled in opposition to a working vLLM server.
MODEL="Qwen/Qwen3-30B-A3B-Instruct-2507" # set your spine
NAME="structured_data_reasoning" # set the potential
N_GPUS=4
# 1 — serve the frozen base mannequin, with LoRA hot-swapping enabled
export VLLM_ALLOW_RUNTIME_LORA_UPDATING=True
vllm serve "$MODEL" --port 8000 --tensor-parallel-size 4
--enable-lora --max-loras 2 --max-lora-rank 32
--enable-auto-tool-choice --tool-call-parser hermes # qwen3_xml for Qwen3.6
# 2 — prepare one LoRA adapter per functionality with GRPO
export VLLM_BASE_URLS=http://localhost:8000
torchrun --nproc_per_node="$N_GPUS" --master-port=29501 -m prepare
--game "capability_$NAME" --model "$MODEL"Every LoRA adapter provides roughly 1.6 billion trainable parameters, representing 5.3% of the spine. The skilled MoE gate provides solely 491,760 parameters in whole. The default thresholds are ρ = 0.10, δ = 0.20, and 8-of-10 cross-run consistency.
Essential factors
- The distinction of failure is sensible. Evaluating profitable and unsuccessful trajectories reveals trainable defects relatively than frequent error labels.
- Pattern effectivity is excessive within the goal surroundings. Each rollout comprises a excessive density of indicators as a result of every verifiable surroundings is given one characteristic.
- The composition collapses. Particular person specialists with token-level routing outperformed the perfect single adapter different by greater than 7 factors.
- Coaching is best than prompting. The skilled adapter scales monotonically, however the prompt-only optimization plateaus after 4 options.
- The 27B mannequin topped the leaderboard. Qwen3.6-27B achieved 73.2% Cross@1 in SWE bench verification over GPT-5.2-Codex.
