Organizations constructing generative AI functions normally examine fashions the identical approach: {dollars} per million tokens. It’s the quantity on each pricing web page, so it turns into the quantity in each spreadsheet. However manufacturing workloads don’t purchase tokens. They purchase outcomes: a resolved help ticket, a accomplished analysis temporary, an accurate monetary abstract. Between the pricing web page and the end result sit multipliers the sticker value ignores: how typically the mannequin is true, what number of tokens it must get there, and, for agentic workloads, what number of turns it takes, as a result of each flip re-sends the rising dialog.
On this put up, we share outcomes from an open-source benchmarking harness that measures these multipliers throughout OpenAI fashions on Amazon Bedrock (gpt-5.6-luna, gpt-5.6-terra, and gpt-5.6-sol) and two extensively used cost-efficient fashions on the OpenAI API (gpt-5.4-mini and gpt-5.4-nano). We selected the latter two because the cost-optimized baselines many groups begin from, not as like-for-like generational friends, as a result of “we run mini or nano right this moment. Is a more recent mannequin on Amazon Bedrock value it?” is the query we hear most. We deal with three questions:
- What does an accurate reply price? Not what a token prices.
- What do agent trajectories price? Flip rely can dominate the invoice.
- Can it produce work an expert would settle for? That is measured on actual occupational deliverables, not quiz questions.
The core benchmarks listed below are reproducible: the harness, openai-on-aws/benchmarks-openai, runs one similar code path (the OpenAI Responses API) towards each functions, and we encourage you to run it by yourself duties earlier than making any determination.
Answer overview
The harness evaluates all 5 fashions by the identical Responses API consumer, switching the backend and mannequin ID whereas holding the analysis logic fixed. The outcomes nonetheless replicate variations within the fashions, supplier infrastructure, and model-specific configuration. Particularly, the fashions on Amazon Bedrock ran with reasoning disabled, whereas the OpenAI API baselines ran at their defaults. It is a comparability of sensible deployment configurations, not a managed estimate of intrinsic mannequin functionality.
We measure three issues: single-call accuracy and price on benchmarks that also separate frontier fashions (AIME competitors arithmetic, GPQA Diamond graduate-level science, and MMLU-Professional), multi-turn agent trajectories on stay web-research duties, and rubric-graded skilled deliverables. Grading combines deterministic checks with a big language mannequin (LLM) decide (gpt-5.5, not one of many evaluated fashions) utilizing frozen prompts whose hashes are recorded in each outcome file. Every run writes a timestamped outcome JSON, and each quantity and chart on this put up is generated from these recordsdata at construct time.
The best way to learn the outcomes: pattern sizes vary from 48–198 gadgets. Deal with small gaps as directional until accompanied by uncertainty estimates, and reproduce the analysis by yourself workload earlier than deciding on a mannequin.
Measuring the price of an accurate reply
For every benchmark we divide a mannequin’s whole spend, throughout proper and unsuitable makes an attempt alike, by its variety of right solutions. This estimates the noticed price of an accurate reply in our pattern. The next graph exhibits accuracy and price per right reply on AIME for all 5 fashions. The sample repeats on GPQA Diamond and MMLU-Professional (full tables within the repo).
Two issues stand out:
- Functionality tiers are seen on this pattern. Sol solves 75 % of AIME issues to mini’s 37 %, and leads GPQA Diamond (68 % versus 43 %) and MMLU-Professional (82 % versus 59 %). If makes an attempt had been unbiased and accuracy remained fixed, 37 % accuracy would indicate roughly 2.7 makes an attempt per success on common. Actual retries are correlated, so measure your precise retry technique fairly than assuming that estimate transfers straight.
- Token effectivity determined the bill even earlier than costs moved, after which costs moved. At its unique record value (roughly 1.5× mini’s), luna was already 25 % cheaper per right AIME reply on this configuration, as a result of with reasoning disabled it used fewer billed tokens than mini at its defaults. Following the July 30, 2026 value discount for GPT-5.6 Luna and Terra on Amazon Bedrock (luna −80 %, terra −20 %), our recorded price per right AIME reply is $0.0021 for luna versus $0.0139 for mini. Throughout these samples, luna has the bottom noticed price per right reply, together with towards nano, despite the fact that nano’s nominal token value is barely decrease below the value assumptions recorded within the outcome recordsdata.Pricing notice: the outcome recordsdata at the moment use $0.22/$1.32 per 1M enter/output tokens for luna and $2.20/$13.20 for terra. Affirm the relevant Amazon Bedrock inference tier and area towards the stay pricing web page earlier than publication. The July 30 announcement describes first-party value parity, whereas pricing-page updates might lag.For patrons, the extra helpful model-selection query shouldn’t be solely “which is least expensive per token?” however “which is least expensive per final result, at a top quality stage my enterprise accepts?”
Measuring agent trajectory price
Single-call pricing misses a defining characteristic of our agentic workload. The harness makes use of client-managed historical past with retailer: false, so each flip re-sends the system immediate, prior instrument outcomes, and dialog context. Per-turn context grows roughly linearly. Cumulative billed enter can due to this fact develop roughly quadratically with flip rely. Every flip additionally provides a spherical journey of latency. A mannequin that finishes in 5 turns as a substitute of eight can save greater than the 37 % discount in turns alone suggests.
To measure this on an actual workload, we ran a 50-question stratified pattern of DeepSearchQA (multi-step web-research questions) by a stay agent loop with actual web_search and fetch_page instruments. Solutions had been scored utilizing a deterministic pre-pass adopted by a frozen gpt-5.5 autorater. A query passes at F1 ≥ 0.7. The next graph exhibits how flip rely drives input-token quantity (left) and what every passing reply finally ends up costing (proper).
Determine 2: Flip rely driving input-token quantity and price per passing DeepSearchQA reply
The left panel is the mechanism. The fitting panel is the consequence. Mini took probably the most turns per query of any mannequin (7.6, largely re-search loops), and each further flip re-sent a context containing gathered search outcomes. By the top, that reached 2.3× terra’s input-token quantity (114k versus 50k tokens per query). On this pattern, terra’s greater token value was offset by fewer turns and better high quality: it price $0.31 per passing reply versus mini’s $0.40, with imply F1 of 0.50 versus 0.39. The three GPT-5.6 configurations recorded greater imply F1 than each baselines, whereas luna mixed fewer turns than mini with $0.05 per passing reply versus mini’s $0.40. Nano’s nominal token value was decrease, however at an 18 % go fee its noticed price per passing reply was $0.07 versus luna’s $0.05. With 50 questions, deal with shut gaps as directional.
The shopper takeaway: flip effectivity is a pricing variable, and it’s invisible on the pricing web page. In case your brokers chain instrument calls (analysis, multi-hop lookups, iterative retrieval), benchmark trajectory price alongside name price.
Evaluating skilled deliverables with GDPval
Benchmarks grade solutions. A lot of what clients ship is paperwork similar to compliance briefs, monetary plans, and care protocols, the place “right” is a rubric, not a string match. So we ran a 48-task slice of GDPval: actual occupational deliverables created by professionals averaging 14 years of expertise, every graded towards its human-authored rubric. A deliverable passes at ≥70% of weighted rubric factors. The next graph exhibits go charges and price per passing deliverable.
Determine 3: GDPval go charges and price per passing deliverable
All three gpt-5.6 configurations achieved greater noticed rubric scores than mini and nano with reasoning disabled. The most important noticed variations appeared in regulation, nursing, and financial-advice duties, the place rubrics demand particular caveats, construction, and completeness. The per-category samples are small, so deal with this sample as exploratory. Luna scored greater than mini on 31 of 48 deliverables, decrease on 9, and tied on 8. It handed 27 deliverables to mini’s 20.
This workload is a single name with no flip results, so earlier than the repricing, mini and nano had been cheaper per passing deliverable and gpt-5.6 purchased go fee alone. After repricing, luna has the bottom noticed price on this pattern: $0.010 per passing deliverable versus mini’s $0.030 and nano’s $0.012, at a 56 % go fee to mini’s 42 % and nano’s 35 %. Terra and sol nonetheless carry a premium per go whereas recording greater rubric efficiency. Whether or not that premium is justified relies on the price of overview and rework within the goal workflow.
GDPval caveat: outputs had been capped at 8,192 tokens, truncating 6 luna, 9 terra, 7 sol, 0 mini, and 1 nano deliverables. These are actual outcomes below the examined cap. The next cap might enhance high quality, but it surely may also enhance price. Take a look at each collectively.
Placing it collectively: a choice framework
In case you run gpt-5.4-mini or nano right this moment, the migration query comes right down to workload form:
| Your workload seems to be like… | Begin with | Why |
| Excessive-volume, low-complexity duties. Failures are low-cost | gpt-5.6-luna on Amazon Bedrock | Luna recorded the bottom price per profitable final result throughout these samples. Nano retains a barely decrease nominal token value below the recorded value assumptions |
| Interactive apps. Accuracy issues. Latency service-level targets (SLOs) | Benchmark gpt-5.6-luna on Amazon Bedrock first | Robust noticed price per success. Latency outcomes are a point-in-time regional snapshot. Re-measure them towards your SLOs |
| Brokers that chain instrument calls (analysis, multi-hop) | Benchmark gpt-5.6-luna and -terra on Amazon Bedrock | Each recorded greater DeepSearchQA F1 with fewer turns than mini. Luna price about 8× much less per passing reply on this 50-question pattern |
| High quality-gated doc manufacturing | gpt-5.6-luna on Amazon Bedrock | Highest go fee among the many cost-efficient fashions on GDPval (27/48 versus mini’s 20/48) on the lowest price per passing deliverable ($0.010 versus $0.030) |
| Accuracy is a tough gate on genuinely tough work | Benchmark gpt-5.6-sol on Amazon Bedrock | A special noticed functionality and value tier (AIME 75%, GDPval 31/48). Take into account it when lower-cost configurations don’t clear your high quality gate |
We additionally measured same-model latency on each functions with the identical streaming harness. In our July 2026 runs (us-west-2, single-region, and point-in-time), median time-to-first-token on Amazon Bedrock averaged 21 % decrease for luna and 5 % decrease for terra throughout the 12 matched configurations. Shared companies differ with load, so deal with these as a snapshot and re-measure. Luna’s throughput averaged 43 % greater on Amazon Bedrock at ≥500-token outputs (terra +4 %). The noticed worst-case TTFT-to-median ratios had been 2.1–2.5× on Amazon Bedrock versus 4.6–6.6× on the OpenAI API. These maxima point out tail variability however usually are not estimates of p99 latency. Sol behaves otherwise: it’s a deep-reasoning mannequin with inherently lengthy and variable time-to-first-token, and its Amazon Bedrock runs used us-east-1. Per-configuration element for all three fashions is documented within the repository’s performance section, and efficiency/run_all.sh reproduces the comparability by yourself account.
Run it in your workload
These outcomes replicate our pattern sizes and configuration selections: 50–198 questions per suite (DeepSearchQA 50, AIME 60, MMLU-Professional 140, GPQA Diamond 198), 48 deliverables, and reasoning disabled for the fashions on Amazon Bedrock, a deliberate price ground. Enabling it raises each high quality and spend. Your duties usually are not our duties. The methodology is the sturdy half:
Swap in 50–100 of your individual duties with known-good outputs and the identical scripts produce cost-per-success numbers in your area, in your account.
Conclusion
The per-token value is one enter to a choice, not the choice. Within the configurations we examined, normalizing price by profitable outcomes modified the mannequin rating after accuracy, token effectivity, and trajectory size had been included. Pricing pages additionally transfer, because the July 2026 GPT-5.6 reductions confirmed. Outline the end result and high quality threshold your software wants, measure the total price of successes and failures, and re-run the analysis when costs, fashions, or workload patterns change.
To get began with OpenAI fashions on Amazon Bedrock, see the Amazon Bedrock documentation. To breed these benchmarks by yourself duties, clone openai-on-aws/benchmarks-openai and comply with the steps within the earlier part.
Concerning the authors
