Wednesday, November 18, 2026
banner
Top Selling Multipurpose WP Theme

Serving massive language fashions (LLMs) effectively requires ample GPU reminiscence and cautious coordination of immediate processing and token technology. GLM-5.3’s FP8 weights, for instance, exceed the 640 GB of GPU reminiscence on a single ml.p5.48xlarge occasion, earlier than accounting for KV cache and runtime overhead. Lengthy prompts may sluggish token technology when each share the identical GPUs.

Amazon SageMaker HyperPod Inference now helps NVIDIA Dynamo, combining Dynamo’s distributed inference capabilities with the infrastructure administration of HyperPod to deal with these challenges. Dynamo is an open supply distributed inference framework that helps multi-node deployment, disaggregated prefill and decode, and KV-aware routing with engines reminiscent of vLLM, SGLang, and TensorRT-LLM. HyperPod helps these deployments with high-performance Elastic Cloth Adapter (EFA) networking, coordinated GPU scheduling, and node well being monitoring and restoration. On this publish, we illustrate multi-node and disaggregated serving via a deployment instance, share benchmark outcomes, and description greatest practices for optimizing inference on HyperPod.

How the mixing works

The HyperPod Inference add-on installs the Dynamo customized useful resource definitions and these serving and scheduling parts:

  • HyperPod Inference Operator manages endpoint deployment via an InferenceEndpointConfig (IEC) customized useful resource. Its dynamoSpec part provides the Dynamo configuration.
  • Dynamo operator manages assets outlined in a DynamoGraphDeployment (DGD), together with the frontend, mannequin staff, reproduction counts, and useful resource necessities.
  • Grove coordinates associated teams of pods, together with multi-node staff, and their scaling relationships.
  • Kubernetes AI (KAI) Scheduler locations pods on GPU assets, utilizing gang scheduling to schedule the pods required for a multi-node employee collectively.

AWS Deep Studying Containers (DLCs) present the Dynamo runtime, whereas the add-on manages built-in element variations. To undertake a supported Dynamo launch, replace the add-on.

Deployment structure

The next diagram reveals the deployment paths, serving parts, and infrastructure administration for disaggregated inference on SageMaker HyperPod. The Dynamo frontend routes requests to prefill staff, which course of prompts, and decode staff, which generate tokens. These separate swimming pools permit unbiased scaling and tuning. Each swimming pools load the total mannequin weights, and KV cache transfers between them over EFA.

Dynamo additionally helps aggregated serving, the place prefill and decode run in the identical employee. In both topology, staff can run on one occasion or span a number of situations utilizing tensor and pipeline parallelism. Dynamo’s KV-aware router considers reusable prefix KV cache and energetic employee load in each topologies. Prefix reuse can scale back immediate recomputation and enhance time to first token, whereas load consciousness helps keep away from overloading staff. For extra particulars on the Dynamo structure, see the Dynamo Knowledge Base overview.

HyperPod supplies node, GPU, and EFA well being monitoring and manages node substitute, whereas the orchestration parts recreate affected staff. Deployments configured via InferenceEndpointConfig with dynamoSpec assist invocation via SageMaker endpoints.

Determine 1: Deployment paths, serving parts, and infrastructure administration for NVIDIA Dynamo on Amazon SageMaker HyperPod Inference

Two methods to configure your deployment

Whether or not you’re new to HyperPod, new to Dynamo, or each, the add-on provides two deployment paths to suit your workflow:

Deliver-your-own DynamoGraphDeployment (DGD) lets current Dynamo customers proceed utilizing upstream documentation, recipes, and manifests, with the Dynamo operator managing the deployment. To convey an current DGD to HyperPod, use a supported container picture and configure storage, networking, and pod placement to your cluster.

InferenceEndpointConfig with dynamoSpec lets current HyperPod Inference customers deploy a brand new Dynamo endpoint utilizing the identical configuration format and OpenAI-compatible interface they already know. For the inference engine, you possibly can select vLLM, SGLang, or TensorRT-LLM, with vLLM utilized by default. The operator additionally provides defaults for KV-aware routing and reproduction counts.

Each paths assist single-instance and multi-node deployments. In both case, HyperPod manages node well being and substitute. The desk summarizes the tasks for every path.

Deliver-your-own DGD InferenceEndpointConfig with dynamoSpec
You write Full DGD manifest IEC containing dynamoSpec
Greatest for Groups wanting direct management of the serving graph or reuse of current DGDs Groups wanting managed endpoint setup via IEC
Mannequin weights You configure downloading or mounting the weights The operator phases weights from Amazon Easy Storage Service (Amazon S3) or Amazon FSx
Endpoint, TLS, registration You configure and handle these assets Managed via the HyperPod Inference Operator
Autoscaling You configure and handle scaling Kubernetes Occasion-driven Autoscaling (KEDA) via autoScalingSpec for aggregated and disaggregated deployments

Deployment instance: GLM-5.3

The next excerpts use the InferenceEndpointConfig (IEC) deployment path with vLLM on ml.p5.48xlarge situations. Dynamo-specific settings belong below spec.dynamoSpec. Mannequin supply and endpoint settings are included within the full manifest.

Multi-node aggregated serving

This configuration runs one model-worker reproduction throughout two situations (16 GPUs), with two frontend replicas for request dealing with. Setting dynamoSpec.frontend.routingMode to kv-aware activates KV-aware routing.

spec:
  replicas: 1
  dynamoSpec:
    modelServer: vllm
    frontend:
      replicas: 2
      routingMode: kv-aware
    aggregation:
      nodesPerReplica: 2
      tensorParallelism: 16

Right here, spec.replicas controls the variety of model-worker replicas, whereas frontend.replicas controls the variety of frontend replicas.

Multi-node disaggregated serving

Exchange aggregation below spec.dynamoSpec with the next disaggregation block, holding the previous frontend configuration. Duplicate counts are set individually for prefill and decode. This instance runs one prefill reproduction and two decode replicas, every spanning two situations, for a complete of six situations.

disaggregation:
  prefill:
    nodesPerReplica: 2
    tensorParallelism: 16
    replicas: 1
  decode:
    nodesPerReplica: 2
    tensorParallelism: 16
    replicas: 2

Apply the deployment

Apply the whole manifest with kubectl apply. The HyperPod Inference Operator makes the configured mannequin weights out there to the employees and generates the DGD. Grove and KAI Scheduler coordinate placement in order that the pods required for every multi-node employee are scheduled collectively.

After the SageMaker endpoint is prepared and configured for chat completions, purchasers can ship an OpenAI-compatible request payload via the SageMaker Runtime API:

aws sagemaker-runtime invoke-endpoint 
--endpoint-name glm53 
--content-type software/json 
--body '{"mannequin":"zai-org/GLM-5.3","messages":[{"role":"user","content":"Hello"}],"max_tokens":128}' 
--cli-binary-format raw-in-base64-out 
out.json

Benchmarks

We benchmarked Dynamo on SageMaker HyperPod throughout two fashions to check serving configurations.

Disaggregated prefill and decode for Qwen3-235B

We used AIPerf to check aggregated and disaggregated serving for Qwen3-235B-A22B-FP8 utilizing Dynamo with TensorRT-LLM on two p5en.48xlarge situations (16 H200 GPUs). Requests contained 4,000 enter tokens and 200 output tokens. We retained measured factors that met two targets: P90 time to first token under one second, and median technology velocity of not less than 50 tokens per second per consumer. Amongst these factors, disaggregated serving achieved 2.05× the best aggregated throughput: 207 versus 101 output tokens per second per GPU, at concurrency 80 and 32, respectively.

Throughput comparison for Qwen3-235B: disaggregated serving at 207 versus aggregated at 101 output tokens per second per GPU

Determine 2: Aggregated in comparison with disaggregated serving throughput for Qwen3-235B-A22B-FP8

Multi-node aggregated serving for Qwen3.5-397B

Dynamo helps multi-node serving in each aggregated and disaggregated modes for fashions that exceed a single occasion’s GPU reminiscence. To judge a employee spanning nodes, we used AIPerf to benchmark multi-node aggregated serving for Qwen3.5-397B-A17B-FP8 with Dynamo and vLLM, utilizing TP4 × PP2 and eight energetic H200 GPUs. This mannequin suits on one occasion, so in follow you wouldn’t break up it. It serves right here as a managed baseline. We in contrast eight GPUs on one p5en.48xlarge in opposition to 4 GPUs on every of two situations, utilizing the identical mannequin, parallelism, and workload configuration. With an agentic workload that includes excessive prefix reuse at concurrency 32, the two-node configuration delivered 126.4 output tokens per second per GPU, retaining 99.1 % of single-node throughput. This reveals that Dynamo on SageMaker HyperPod can serve a mannequin that wants a couple of occasion throughout nodes. The node boundary itself doesn’t turn out to be a major throughput penalty.

Throughput comparison for Qwen3.5-397B: two-node serving at 126.4 output tokens per second per GPU, 99.1% of single-node throughput

Determine 3: Single-node in comparison with two-node throughput for multi-node aggregated serving of Qwen3.5-397B-A17B-FP8

Greatest practices for optimized inference

NVIDIA Dynamo supplies recipes and instruments that can assist you choose a serving configuration, benchmark efficiency, and tune your deployment on SageMaker HyperPod.

  • Begin with an identical recipe – Select a Dynamo recipe to your mannequin, inference engine, and GPU sort. Account for mannequin weights, KV cache, and runtime reminiscence when sizing staff. For supported configurations, AISimulate may also help evaluate aggregated and disaggregated serving, select parallelism, and measurement prefill and decode employee swimming pools earlier than testing on GPUs.
  • Confirm communication earlier than benchmarking – For deployments spanning situations, affirm that communication makes use of the supposed EFA transport, together with KV-cache transfers between prefill and decode staff. See the Dynamo EFA guide.
  • Benchmark consultant visitors – Replay your visitors with AIPerf, or use a generated agentic trace with consultant immediate lengths, output lengths, and prefix reuse. Sweep concurrency for every configuration, retain factors assembly your SLAs, and select primarily based on value and latency priorities.
  • Tune from measured bottlenecks – Alter topology and employee allocation earlier than engine settings, altering one variable at a time. Observe the performance tuning guide, or use Dynamo optimization agent skills to help with benchmarking and tuning. Validate every candidate in your goal HyperPod deployment.

Conclusion

NVIDIA Dynamo on SageMaker HyperPod Inference simplifies deploying and working massive fashions with multi-node serving, disaggregated prefill and decode, and KV-aware routing. The managed add-on installs the serving parts, and configuration defaults simplify endpoint setup. HyperPod manages mannequin weights, auto-scales clusters primarily based on GPU and inference metrics, screens node well being, and replaces failed nodes. Collectively, these capabilities scale back undifferentiated infrastructure work so groups can concentrate on their fashions and functions. Get began with the setup directions and full deployment examples within the linked repository.

Acknowledgements

The authors want to thank Eliuth Triana, Felipe Garcia, Sanjana Checker, Junyuan Yang, Jiahong Liu, Anish Maddipoti, Jacob Kern and Abdullahi Olaoye from NVIDIA for his or her contributions to this publish. We additionally want to thank Can Solar, Ophelia Yang, Aryaman Kothari, David Oh, and Ayush Kumar from AWS for his or her contributions.


Concerning the authors

Vinay Arora

Vinay Arora

Vinay is a Specialist Answer Architect for Generative AI at AWS, the place he collaborates with clients in designing cutting-edge AI options, leveraging AWS applied sciences and Infrastructure for the previous 6 years. Previous to AWS, Vinay has over 20 years of expertise in finance—together with roles at banks and hedge funds—he has constructed danger fashions, buying and selling techniques, and market information platforms. Vinay holds a grasp’s diploma in laptop science and enterprise administration.

Piyush Daftary

Piyush Daftary

Piyush is a Principal Software program Engineer at AWS, the place he builds inference techniques for giant language fashions on Amazon SageMaker HyperPod. He focuses on optimized routing, disaggregated serving, and autoscaling architectures that assist clients extract most efficiency from their GPU fleets. His background spans distributed techniques, databases, and search applied sciences. Exterior of labor, he enjoys touring and spending time with household.

Xuan Lu

Xuan Lu

Xuan is a Software program Improvement Engineer at AWS, the place he works on Amazon SageMaker HyperPod Inference to construct and scale large-scale LLM inference techniques. His work focuses on distributed inference, clever request routing, and efficiency optimization, with expertise throughout Kubernetes-based orchestration and LLM serving infrastructure. Exterior of labor, he enjoys touring, exploring nature, and studying science fiction.

Preethi Ravula

Preethi Ravula

Preethi is a Options Structure Supervisor for Generative AI, Knowledge, Storage & Analytics on the ISV workforce at AWS. She leads a workforce that helps software program distributors construct and scale generative AI on AWS. She works on AI technique and inference optimization, and she or he develops frameworks and playbooks that assist clients run massive language mannequin workloads quicker and extra cost-effectively on AWS Trainium and NVIDIA GPUs. Preethi has spoken at GHC 2025, SJSU, and Girls in Tech Dublin. She additionally champions AI training via an AWS program that teaches college students to construct functions with AWS generative AI companies.

Sai Devulapalli

Sai Devulapalli

Sai has been main the Amazon SageMaker AI world-wide GTM workforce over the previous six years. He works intently with customer-facing AI Specialist leaders throughout AWS world-wide and engages with C/VP-level stakeholders at clients to know their wants and provide steering in constructing and executing on their firm’s AI platform methods. Sai works with SageMaker AI Service management to assist information service roadmap to higher serve our buyer wants and has been in AI/ML platform area over the previous 16 years, main enterprise growth and product administration groups at Dell EMC and Pivotal and previous to that, main Knowledge Analytics and ML follow for Ericsson in North America.

Aryaman Kothari

Aryaman is a Senior Product Supervisor at Amazon SageMaker AI, the place he leads AI inference on Amazon SageMaker HyperPod, serving to enterprises run AI fashions at scale for manufacturing workloads. A Chicago Sales space graduate, Aryaman beforehand led product and go-to-market for Quantiphi’s Conversational AI enterprise. Exterior work, he enjoys enjoying soccer, making chai, and mountaineering.

Kshitiz Gupta

Kshitiz Gupta

Kshitiz is a Senior Options Architect at NVIDIA, the place he helps cloud clients design and optimize GPU-accelerated information and AI techniques. His work spans information processing, vector database, and LLM inference, partnering intently with AWS and Amazon groups to convey these capabilities into manufacturing. Exterior of labor, he enjoys music, yoga, and mountaineering.

Marc Karp

Marc Karp

Marc is a Senior Options Architect at NVIDIA, the place he helps clients deploy and scale AI inference on GPUs. He works throughout the inference stack, from mannequin optimization and distributed serving to the engines that energy them. Exterior of labor, he enjoys touring and exploring new locations.

Wenhan Tan

Wenhan Tan

Wenhan is a options architect at NVIDIA, aiding clients in adopting NVIDIA AI options at large-scale. His work focuses on accelerating deep studying functions and addressing inference and coaching challenges.

banner
Top Selling Multipurpose WP Theme

Converter

Top Selling Multipurpose WP Theme

Newsletter

Subscribe my Newsletter for new blog posts, tips & new photos. Let's stay updated!

banner
Top Selling Multipurpose WP Theme

Leave a Comment

banner
Top Selling Multipurpose WP Theme

Latest

Best selling

22000,00 $
16000,00 $
6500,00 $
5999,00 $

Top rated

6500,00 $
22000,00 $
900000,00 $

Products

Knowledge Unleashed
Knowledge Unleashed

Welcome to Ivugangingo!

At Ivugangingo, we're passionate about delivering insightful content that empowers and informs our readers across a spectrum of crucial topics. Whether you're delving into the world of insurance, navigating the complexities of cryptocurrency, or seeking wellness tips in health and fitness, we've got you covered.