Most fine-tuning teaches a mannequin to supply a greater reply to a single immediate. Agentic duties are totally different. An agent that recordsdata an expense report or triages a help ticket takes a sequence of actions, and every step relies on what got here earlier than. There’s hardly ever one appropriate sequence to mimic, and the one dependable suggestions is whether or not the entire sequence succeeded. Supervised fine-tuning has no labels for that, and single-turn reinforcement studying can’t credit score a call made three steps earlier than the end result.
Multi-Flip Reinforcement Studying (MTRL) on Amazon SageMaker AI trains fashions on precisely that sign. SageMaker AI sends prompts to an agent you host on Amazon Bedrock AgentCore, or by yourself infrastructure, corresponding to Amazon Elastic Kubernetes Service (Amazon EKS), Amazon Elastic Compute Cloud (Amazon EC2), or AWS Fargate, by means of an AWS Lambda forwarder. The instance right here makes use of AgentCore. The agent calls the coverage mannequin being skilled, acts in your surroundings over as many turns as the duty wants, and experiences one reward when the episode ends. You don’t specify the proper trajectory, solely the right way to rating the outcome.
This submit makes use of Wordle, a recreation during which a participant has six tries to guess a hidden five-letter phrase, as the instance. After every guess, each letter turns inexperienced whether it is in the proper place, yellow if it’s within the phrase however in a unique place, and grey if it isn’t within the phrase. The sport fits MTRL as a result of the principles are few, every guess relies on the suggestions from the final, and the end result is scored objectively. Wordle is a educating instance, and frontier fashions already play it nicely. In apply, you’d use MTRL to make a small, low-cost mannequin good at a multi-step activity particular to what you are promoting, corresponding to an inside workflow that public fashions had been by no means skilled on. The identical pipeline applies to that setting.
In our run, MTRL took gpt-oss-20b from a 61 to an 84 % resolve charge on held-out phrases at its default reasoning effort, and from 15 to 69 % on the low effort it skilled at, for about $339 in coaching tokens. On this submit, we stroll by means of the pipeline end-to-end: deploying the agent and reward to AgentCore, making ready the dataset, coaching, monitoring in MLflow, and evaluating at serving settings. We additionally share the reward-design errors and coaching failures we hit. Your datasets, agent, and skilled mannequin keep in your AWS account and are usually not used to coach the bottom fashions. All code is within the accompanying aws-samples repository.
Answer overview
The system has two halves. SageMaker AI owns the coaching job and the coverage mannequin. AgentCore hosts the agent, the Wordle surroundings, and the reward operate. The next diagram exhibits how they join.
Determine 1: MTRL on SageMaker AI, the place the coaching job invokes the AgentCore-hosted agent as soon as per rollout to pattern the coverage mannequin, play the sport, and report a reward
For each rollout, the MTRL service invokes the AgentCore runtime, a functionality of Amazon Bedrock AgentCore, with one coaching row, which holds one secret phrase. A Strands agent within the runtime calls the coverage mannequin by means of the SageMaker Job Runtime’s OpenAI-compatible endpoint. The mannequin performs as much as six guesses by means of one instrument, guess_word, and after every guess the surroundings returns per-letter suggestions and a abstract of what’s identified. When the sport ends, the agent returns one reward. Group Relative Policy Optimization (GRPO) makes use of the unfold of rewards throughout a number of rollouts of the identical phrase to replace a Low-Rank Adaptation (LoRA) adapter on the coverage mannequin.
Three design factors matter for what follows:
- The agent and reward stay in a single place. A single Python file deployed to AgentCore holds the surroundings, the instrument, and the scoring operate.
- The dataset carries solely the key phrase. The phrase lists bundled with the agent are constructed from the open-source SCOWL phrase lists and are used just for scoring.
- The reward is outcome-dominant. A resolve pays 1.05–1.55 relying on velocity, a loss pays -0.5, and by no means making a legitimate guess pays -1.5. A small shaping time period rewards good technique inside every tier.
Supported fashions and AWS Areas
MTRL is serverless: you choose a base mannequin and the service provisions the coaching and sampling infrastructure. On the time of writing, it helps the next base fashions, every skilled with a LoRA adapter:
| Base mannequin | AWS Areas |
| Amazon Nova 2 Lite | US East (N. Virginia) (us-east-1), US West (Oregon) (us-west-2) |
GPT-OSS-20B |
US East (N. Virginia) (us-east-1), US West (Oregon) (us-west-2) |
| Gemma 4 31B (instruction-tuned) | US West (Oregon) (us-west-2) |
| Qwen 3.6 27B | US West (Oregon) (us-west-2) |
We selected GPT-OSS-20B as a result of it has the bottom per-token worth of the 4 and its reasoning effort is controllable per request. See the MTRL documentation for the present checklist.
Pricing
You don’t select cases or measurement a cluster. You pay per million tokens in three dimensions, priced by base mannequin:
- Prefill: enter tokens the coverage mannequin processes. These accumulate in a multi-turn activity, as a result of each flip re-reads the historical past.
- Pattern: tokens the coverage mannequin generates, together with reasoning tokens.
- Prepare: tokens processed within the backward move that updates the LoRA adapter.
The 100-step run consumed 491.6 million prefill, 159.7 million pattern, and 643.3 million prepare tokens throughout 105,486 rollouts in 12 hours 40 minutes. At GPT-OSS-20B checklist costs within the US East (N. Virginia) Area (us-east-1) on the time of writing, that’s about $339:
| Dimension | Tokens | Worth per 1M | Value |
| Prefill | 491,616,806 | $0.12 | $58.99 |
| Pattern | 159,737,098 | $0.30 | $47.92 |
| Prepare | 643,308,554 | $0.36 | $231.59 |
| Whole | 1,294,662,458 | $338.50 |
The prepare dimension dominates and scales with trajectory size, so shorter video games and fewer verbose reasoning reduce the invoice. The AgentCore runtime payments individually for the vCPU and reminiscence the agent makes use of. For this run, that was about $79, from the runtime’s Amazon CloudWatch utilization metrics at us-east-1 checklist costs. DescribeJob experiences the token counts beneath BillableTokenUsage. See Amazon SageMaker AI pricing for present charges.
Stipulations
To observe this walkthrough, you want the next. See additionally the MTRL conditions.
- Python 3.10 or later, uv, Node 20 or later, and the AgentCore CLI (
npm set up -g @aws/agentcore). See Get began with Amazon Bedrock AgentCore for set up and the IAM permissions the CLI wants. - botocore 1.43.37 or later, on the machine that launches the job. That is the primary model that features the
AgentRFTCreateJoboperation. - Two AWS Id and Entry Administration (IAM) roles, described in Step 1.
- An Amazon Easy Storage Service (Amazon S3) bucket in the identical Area because the agent. The coaching job should additionally run in that Area, as a result of
CreateJobrejects a cross-Area runtime ARN.
Step 1: Create the IAM roles
MTRL makes use of two roles, and certainly one of them has a belief coverage that the final SageMaker AI documentation doesn’t point out.
Agent runtime function. agentcore deploy creates this function for you. It trusts bedrock-agentcore.amazonaws.com and wishes the AmazonSageMakerJobRuntimeAccess managed coverage, which lets the agent name the coverage mannequin and report rewards. The CDK stack within the pattern repository attaches this coverage routinely.
Coaching job function. You create this one. It should belief job.sagemaker.amazonaws.com with each sts:AssumeRole and sts:TagSession, and it wants the AmazonSageMakerJobFullAccess managed coverage.
Notice: The belief principal is job.sagemaker.amazonaws.com, not sagemaker.amazonaws.com. With the generic principal, CreateJob fails with “Couldn’t assume function,” and the message names the right principal solely after retries.
Step 2: Deploy the agent to Amazon Bedrock AgentCore
Clone the sample repository, set your account and Area in Wordle/agentcore/aws-targets.json, and deploy:
The CLI builds the agent’s dependencies for the runtime’s Linux ARM64 platform, creates the execution function with AmazonSageMakerJobRuntimeAccess hooked up, and deploys by means of AWS CloudFormation. We advocate it over importing a code package deal on the console, the place it’s frequent to get the roughly 700 MB of compiled dependencies or the coverage attachment improper. Notice the runtime ARN within the output for the coaching job.
Determine 2: The Wordle agent runtime within the Amazon Bedrock AgentCore console with standing Prepared
The runtime reads two surroundings variables from agentcore.json. REASONING_EFFORT=low units reasoning_effort on each request to the coverage mannequin. In our analysis jobs, low effort used 63 to 83 % fewer pattern tokens per recreation than medium. REASONING_PROMPT_HINT=true additionally provides “Reasoning: low” to the system immediate, which is how gpt-oss reads the setting in its Concord chat format. Neither variable provides the mannequin any trace in regards to the secret phrase.
Determine 3: Runtime surroundings variables exhibiting REASONING_EFFORT set to low
The agent is one file, most important.py. Simplified, its entrypoint seems like the next. On your personal activity, you alter the duty parsing, the instruments, and the reward:
Vital: The sagemaker_rft_handler decorator doesn’t begin an HTTP server by itself. Wrap it with BedrockAgentCoreApp and @app.entrypoint, with the entrypoint decorator outermost. With out this, each invocation instances out after 30 seconds.
Step 3: Confirm the runtime
Earlier than spending GPU time, verify the entire chain works with a payload formed just like the one the coaching service sends. The repository consists of one as Wordle/payload.json:
The pretend job ARN is predicted to fail, and the best way it fails is the proof. Put your individual account ID into it first: for a job ARN in one other account, the service returns a 403 earlier than it checks whether or not the job exists, which seems like a damaged IAM setup. From the Wordle listing, run:
Within the runtime’s Amazon CloudWatch logs, you see the agent parse the immediate, authenticate to the Job Runtime endpoint, and obtain a 400 error studying “Couldn’t discover job.” That failure confirms that parsing, authentication, and error reporting work. The command itself prints {“reward”: -1.5}, as a result of the agent by no means reached the coverage mannequin. A 403 studying “not licensed to entry this useful resource” means the job ARN nonetheless names one other account. A 30-second timeout as a substitute means the HTTP server will not be beginning (see the Vital notice in Step 2).
Step 4: Put together the dataset
The MTRL dataset has no goal responses, transcripts, or labels. Every row is one activity for the agent to aim. Listed here are the primary three rows of the coaching file:
The service reads the immediate column (or the primary column) and passes the string to your agent verbatim, with out parsing it. We made the string a small JSON object with three fields:
- immediate: the instruction the agent exhibits the coverage mannequin, an identical on each row.
- reply: the key phrase. Solely the surroundings and the reward operate use it. The coverage mannequin by no means sees it.
- id: a row ID for tracing a rollout within the logs again to its supply row.
The pattern repository’s make_dataset.py attracts 640 coaching phrases (5 full batches of 128) and 128 validation phrases from the reply checklist with no overlap, so the validation resolve charge measures generalization. The service additionally accepts Apache Parquet, JSON arrays, and CSV. Add each recordsdata to Amazon S3:
As a result of the service by no means inspects the immediate string, a manufacturing activity can pack a full message checklist, instrument configuration, or reward specification into it. See the MTRL documentation for examples, together with the right way to shield delicate content material within the string.
Step 5: Launch the coaching job
The launcher makes use of the MultiTurnRLTrainer class from the sagemaker-train SDK. Level it on the runtime ARN from Step 2, the function from Step 1, the datasets from Step 4, and an MLflow app for metrics. The MLflow app is required, so when you don’t have one, create it first with aws sagemaker create-mlflow-app:
The launcher overrides three hyperparameters and leaves each different one on the gpt-oss-20b default. It units sampling_max_tokens to 8192, the service most, as a result of on the default of 4096 gpt-oss typically ran out of tokens whereas reasoning, earlier than its first instrument name. It units max_epochs to twenty and max_steps to 100: 640 rows on the default global_batch_size of 128 is 5 steps per epoch, and with the default group_size of 8, every step performs 1,024 video games. The job strikes by means of Beginning, Downloading, and Pending earlier than reaching Coaching. After it’s coaching, your AgentCore runtime begins receiving rollout invocations.
Vital: Don’t redeploy the agent whereas a job is operating. Redeploying restarts the runtime, in-flight rollouts see no sampling requests, and the service fails all the job.
Step 6: Monitor coaching in MLflow
The service logs reward, token, and switch statistics at each step and runs validation each val_every steps. The metric to observe is val/reward/pass_at_1, the fraction of held-out phrases solved. Each resolve scores a minimum of 1.01, so it clears the service’s success_threshold of 1.0 and move@1 counts it. The next chart is constructed from this run’s MLflow historical past:
Determine 4: Coaching curves for the 100-step run, with imply reward per recreation on the left and validation resolve charge on the proper
Coaching reward climbs to about +0.57 inside 15 steps and holds there. Validation reward stays decrease, principally due to temperature 1.0 versus 0 moderately than overfitting (see Step 7). The validation resolve charge jumps from 2 to 31 % within the first 10 steps, then strikes between 35 and 48 %. With 128 phrases, these swings are principally noise. We didn’t take a look at whether or not extra steps assist at serving settings.
The runtime’s OpenTelemetry traces in CloudWatch report each mannequin flip and power name. Right here is one recreation the skilled mannequin performed throughout an analysis job (medium effort, temperature 1.0), with the reasoning trimmed. The key phrase was PASTE:
A trained-model rollout from the AgentCore runtime traces. The mannequin retains inexperienced letters in place, by no means reuses a grey letter, and makes use of every yellow letter to rule out positions, although nothing within the coaching information confirmed it how.
Step 7: Consider the skilled mannequin at serving settings
When the job completes, its web page in SageMaker AI Studio exhibits the standing and the mannequin package deal group that holds the skilled mannequin. DescribeJob, with JobCategory set to AgentRFT, returns the output mannequin package deal ARN and the billable token counts.
Determine 5: The finished coaching job in SageMaker AI Studio on the Jobs, Coaching, Multi-turn RL tab
An analysis job (JobCategory set to AgentRFTEvaluation) performs your analysis prompts by means of the identical AgentCore runtime and reward, on the bottom mannequin or the skilled package deal, and logs eval/reward/pass_at_1 to MLflow. In contrast to coaching validation, you’ll be able to set the temperature. The repository’s launcher runs evaluations again to again, as a result of the default quota permits one analysis job at a time (you’ll be able to request a rise):
Determine 6: Reasoning effort and sampling temperature at every stage, the place the coaching job fixes validation at temperature 0 and the deployed effort whereas analysis jobs allow you to select
Every 128-word analysis took 4 to seven minutes. Reasoning effort comes from the agent’s surroundings variables, so to vary it you edit REASONING_EFFORT, redeploy, and consider. We evaluated each fashions at low and medium effort and at temperature 0 and 1.0:
Determine 7: Base mannequin versus skilled mannequin beneath 4 decoding settings, with resolve charge on the left and imply reward per recreation on the proper
The grey section of every bar is the bottom mannequin and the blue section is the acquire from coaching. The low-effort, temperature-0 pair is the coaching job’s validation at steps 0 and 100. The medium-effort, temperature-1.0 skilled result’s the imply of two runs (80 and 88 %).
At temperature 0, each fashions typically finish the sport with out a resolve, at both effort. With sampling on, medium effort provides 15 factors for the skilled mannequin and 46 for the bottom mannequin. The adapter provides 14–54 factors in each setting, so what coaching taught transfers throughout settings.
The coaching job’s personal validation curve in MLflow instructed a unique story: it peaked at 48 % at step 90 and ended at 37 %, scored at temperature 0. The hyperparameters reference recommends temperature 0 for analysis as a result of grasping decoding makes curves deterministic and comparable throughout runs. We report temperature 1.0 as a result of that’s how we serve the mannequin. At medium effort and temperature 1.0, the skilled mannequin solves 84 % of held-out phrases, up from 61 % for the bottom mannequin.
Tip: Consider on the settings you’ll serve at. The coaching curve says this adapter ends at 37 %. At serving settings it solves 84 %. The analysis jobs that present this value about $1.20 in tokens and took about an hour.
Step 8: Clear up
Coaching and analysis jobs cease billing once they full, however the AgentCore runtime, its IAM function, the MLflow app, and the S3 objects persist. From the challenge listing:
As a result of agentcore take away all rewrites the recordsdata beneath agentcore/, run git checkout -- agentcore/ if you wish to redeploy later. Deleting the AgentCore-Wordle-default CloudFormation stack removes the identical sources and leaves the recordsdata alone. The coaching job function and the mannequin packages are usually not billed.
Deep dive: Designing the reward
Our reward wanted one revision, and the error is a standard one.
The primary design summed the per-guess scoring from the wordle-lora-rl challenge over the entire recreation. That scoring works there as a result of every guess is an unbiased coaching pattern. Summed into one reward per recreation, it produced two inversions we discovered solely by studying traces:
- A misplaced recreation paid optimistic reward. Six turns of bonuses added as much as about +0.49.
- Fixing on flip 1 scored under fixing on flip 3. A primary-guess resolve earned +0.99 and a third-guess resolve earned +1.61, as a result of the additional turns collected extra bonuses.
The repair separates a dominant end result time period from a small, bounded shaping time period:
The end result time period is ready completely by how the sport ended:
| Final result | Worth |
| Solved on guess n | 1.05 + 0.5 * (6 – n) / 5, so 1.55 on guess 1 all the way down to 1.05 on guess 6 |
| Performed all six guesses, unsolved | -0.5 |
| By no means made a legitimate guess | -1.5 |
The shaping time period is that per-guess scoring, summed over the sport:
- Penalties for contradicting a identified inexperienced, yellow, or grey letter, for repeating a guess, and for phrases exterior the dictionary.
- An entropy bonus for a robust opening phrase.
- Bonuses for attempting new letters and for shrinking the set of remaining potentialities.
The sum is clipped to [-1, 1] and scaled by 0.04 for a resolve and 0.15 for a loss, so it separates video games with the identical end result however by no means strikes a recreation into one other tier. With typical shaping values:
| Situation | Reward |
| Remedy on guess 1 | +1.550 |
| Remedy on guess 3 | +1.375 |
| Remedy on guess 6 | +1.069 |
| Misplaced after six guesses | -0.427 |
| By no means made a legitimate guess | -1.500 |
A sooner resolve now all the time beats a slower one, each resolve beats each loss, and by no means enjoying is the worst end result. Our first repair used a single weight of 0.15, which let a guess-3 resolve with full shaping (+1.45) outscore a guess-2 resolve with the minimal (+1.25). Adjoining guess counts are 0.1 aside, so a smaller resolve weight of 0.04 removes the overlap, and the worst resolve (+1.01) nonetheless clears the 1.0 success threshold. Shaping nonetheless issues as a result of GRPO learns from reward variance inside a gaggle, so two misplaced video games that differ in clue consistency nonetheless educate the mannequin one thing. Because the SageMaker AI documentation advises, begin with an outcome-based reward, add shaping sparingly, and examine the reward on actual trajectories.
Deep dive: What went improper, within the order you’ll hit it
Every of those value a minimum of one coaching run.
- Bearer tokens beneath concurrency. Minting a token with a contemporary botocore session per name starved the credential supplier at 32 concurrent rollouts, and caching tokens for an hour returned 403 errors as credentials rotated. The repair is one self-refreshing session, a brief cache, and a re-sign on 403.
- A rollout that by no means reaches the coverage mannequin fails the entire job. When each mannequin name in a rollout failed authentication, the service ended the job with “No sampling requests had been acquired.” Returning an error standing fails it too, so all the time return a reward. A recreation the place the mannequin is sampled however by no means makes a legitimate guess is ok: it scores -1.5, and the 100-step run had 51 of them in about 105,000 rollouts.
- Software calls drift into plain textual content. At too excessive a studying charge, the mannequin wrote instrument calls as textual content, each rollout acquired the identical flooring reward, and the gradient vanished. The agent now recovers guesses from uncooked textual content with a small formatting penalty.
- The plateau. Coaching reward stopped rising after about step 15 and validation stopped enhancing after about step 30, although at temperature 0 validation understates the mannequin (Step 7). A curriculum that seeds video games with prior guesses, or extra rollouts per phrase than the default
group_sizeof 8, is what we might strive subsequent.
Conclusion
MTRL on SageMaker AI trains a mannequin on the end result of a complete episode. That matches agentic duties, the place the proper trajectory is unknown and solely the outcome might be scored. We used a single episode-level reward. When intermediate steps might be scored too, MTRL additionally accepts one reward per flip by means of the UpdateReward API. On this submit, an open-weight 20B mannequin reached an 84 % resolve charge at serving settings, up from 61 %, for about $339 in coaching tokens and about $79 in AgentCore runtime utilization. The 37 % the coaching curve experiences is an artifact of how the coaching job scores validation, so consider on the settings you’ll serve at.
The identical pipeline applies to inside procedures, instruments, and workflows that public fashions had been by no means skilled on.
To get began, clone the sample repository, observe the previous steps, and run the smoke take a look at in Step 3 earlier than committing GPU time. For the underlying APIs, see the SageMaker AI MTRL documentation.
In regards to the authors
