Serving massive language fashions (LLMs) effectively requires ample GPU reminiscence and cautious coordination of immediate processing and token technology. GLM-5.3’s FP8 weights, for instance, exceed the 640 GB of GPU reminiscence on a single ml.p5.48xlarge occasion, earlier than accounting for KV cache and runtime overhead. Lengthy prompts may sluggish token technology when each share the identical GPUs.
Amazon SageMaker HyperPod Inference now helps NVIDIA Dynamo, combining Dynamo’s distributed inference capabilities with the infrastructure administration of HyperPod to deal with these challenges. Dynamo is an open supply distributed inference framework that helps multi-node deployment, disaggregated prefill and decode, and KV-aware routing with engines reminiscent of vLLM, SGLang, and TensorRT-LLM. HyperPod helps these deployments with high-performance Elastic Cloth Adapter (EFA) networking, coordinated GPU scheduling, and node well being monitoring and restoration. On this publish, we illustrate multi-node and disaggregated serving via a deployment instance, share benchmark outcomes, and description greatest practices for optimizing inference on HyperPod.
How the mixing works
The HyperPod Inference add-on installs the Dynamo customized useful resource definitions and these serving and scheduling parts:
- HyperPod Inference Operator manages endpoint deployment via an InferenceEndpointConfig (IEC) customized useful resource. Its
dynamoSpecpart provides the Dynamo configuration. - Dynamo operator manages assets outlined in a DynamoGraphDeployment (DGD), together with the frontend, mannequin staff, reproduction counts, and useful resource necessities.
- Grove coordinates associated teams of pods, together with multi-node staff, and their scaling relationships.
- Kubernetes AI (KAI) Scheduler locations pods on GPU assets, utilizing gang scheduling to schedule the pods required for a multi-node employee collectively.
AWS Deep Studying Containers (DLCs) present the Dynamo runtime, whereas the add-on manages built-in element variations. To undertake a supported Dynamo launch, replace the add-on.
Deployment structure
The next diagram reveals the deployment paths, serving parts, and infrastructure administration for disaggregated inference on SageMaker HyperPod. The Dynamo frontend routes requests to prefill staff, which course of prompts, and decode staff, which generate tokens. These separate swimming pools permit unbiased scaling and tuning. Each swimming pools load the total mannequin weights, and KV cache transfers between them over EFA.
Dynamo additionally helps aggregated serving, the place prefill and decode run in the identical employee. In both topology, staff can run on one occasion or span a number of situations utilizing tensor and pipeline parallelism. Dynamo’s KV-aware router considers reusable prefix KV cache and energetic employee load in each topologies. Prefix reuse can scale back immediate recomputation and enhance time to first token, whereas load consciousness helps keep away from overloading staff. For extra particulars on the Dynamo structure, see the Dynamo Knowledge Base overview.
HyperPod supplies node, GPU, and EFA well being monitoring and manages node substitute, whereas the orchestration parts recreate affected staff. Deployments configured via InferenceEndpointConfig with dynamoSpec assist invocation via SageMaker endpoints.
Determine 1: Deployment paths, serving parts, and infrastructure administration for NVIDIA Dynamo on Amazon SageMaker HyperPod Inference
Two methods to configure your deployment
Whether or not you’re new to HyperPod, new to Dynamo, or each, the add-on provides two deployment paths to suit your workflow:
Deliver-your-own DynamoGraphDeployment (DGD) lets current Dynamo customers proceed utilizing upstream documentation, recipes, and manifests, with the Dynamo operator managing the deployment. To convey an current DGD to HyperPod, use a supported container picture and configure storage, networking, and pod placement to your cluster.
InferenceEndpointConfig with dynamoSpec lets current HyperPod Inference customers deploy a brand new Dynamo endpoint utilizing the identical configuration format and OpenAI-compatible interface they already know. For the inference engine, you possibly can select vLLM, SGLang, or TensorRT-LLM, with vLLM utilized by default. The operator additionally provides defaults for KV-aware routing and reproduction counts.
Each paths assist single-instance and multi-node deployments. In both case, HyperPod manages node well being and substitute. The desk summarizes the tasks for every path.
| Deliver-your-own DGD | InferenceEndpointConfig with dynamoSpec | |
| You write | Full DGD manifest | IEC containing dynamoSpec |
| Greatest for | Groups wanting direct management of the serving graph or reuse of current DGDs | Groups wanting managed endpoint setup via IEC |
| Mannequin weights | You configure downloading or mounting the weights | The operator phases weights from Amazon Easy Storage Service (Amazon S3) or Amazon FSx |
| Endpoint, TLS, registration | You configure and handle these assets | Managed via the HyperPod Inference Operator |
| Autoscaling | You configure and handle scaling | Kubernetes Occasion-driven Autoscaling (KEDA) via autoScalingSpec for aggregated and disaggregated deployments |
Deployment instance: GLM-5.3
The next excerpts use the InferenceEndpointConfig (IEC) deployment path with vLLM on ml.p5.48xlarge situations. Dynamo-specific settings belong below spec.dynamoSpec. Mannequin supply and endpoint settings are included within the full manifest.
Multi-node aggregated serving
This configuration runs one model-worker reproduction throughout two situations (16 GPUs), with two frontend replicas for request dealing with. Setting dynamoSpec.frontend.routingMode to kv-aware activates KV-aware routing.
Right here, spec.replicas controls the variety of model-worker replicas, whereas frontend.replicas controls the variety of frontend replicas.
Multi-node disaggregated serving
Exchange aggregation below spec.dynamoSpec with the next disaggregation block, holding the previous frontend configuration. Duplicate counts are set individually for prefill and decode. This instance runs one prefill reproduction and two decode replicas, every spanning two situations, for a complete of six situations.
Apply the deployment
Apply the whole manifest with kubectl apply. The HyperPod Inference Operator makes the configured mannequin weights out there to the employees and generates the DGD. Grove and KAI Scheduler coordinate placement in order that the pods required for every multi-node employee are scheduled collectively.
After the SageMaker endpoint is prepared and configured for chat completions, purchasers can ship an OpenAI-compatible request payload via the SageMaker Runtime API:
Benchmarks
We benchmarked Dynamo on SageMaker HyperPod throughout two fashions to check serving configurations.
Disaggregated prefill and decode for Qwen3-235B
We used AIPerf to check aggregated and disaggregated serving for Qwen3-235B-A22B-FP8 utilizing Dynamo with TensorRT-LLM on two p5en.48xlarge situations (16 H200 GPUs). Requests contained 4,000 enter tokens and 200 output tokens. We retained measured factors that met two targets: P90 time to first token under one second, and median technology velocity of not less than 50 tokens per second per consumer. Amongst these factors, disaggregated serving achieved 2.05× the best aggregated throughput: 207 versus 101 output tokens per second per GPU, at concurrency 80 and 32, respectively.
Multi-node aggregated serving for Qwen3.5-397B
Dynamo helps multi-node serving in each aggregated and disaggregated modes for fashions that exceed a single occasion’s GPU reminiscence. To judge a employee spanning nodes, we used AIPerf to benchmark multi-node aggregated serving for Qwen3.5-397B-A17B-FP8 with Dynamo and vLLM, utilizing TP4 × PP2 and eight energetic H200 GPUs. This mannequin suits on one occasion, so in follow you wouldn’t break up it. It serves right here as a managed baseline. We in contrast eight GPUs on one p5en.48xlarge in opposition to 4 GPUs on every of two situations, utilizing the identical mannequin, parallelism, and workload configuration. With an agentic workload that includes excessive prefix reuse at concurrency 32, the two-node configuration delivered 126.4 output tokens per second per GPU, retaining 99.1 % of single-node throughput. This reveals that Dynamo on SageMaker HyperPod can serve a mannequin that wants a couple of occasion throughout nodes. The node boundary itself doesn’t turn out to be a major throughput penalty.
Determine 3: Single-node in comparison with two-node throughput for multi-node aggregated serving of Qwen3.5-397B-A17B-FP8
Greatest practices for optimized inference
NVIDIA Dynamo supplies recipes and instruments that can assist you choose a serving configuration, benchmark efficiency, and tune your deployment on SageMaker HyperPod.
- Begin with an identical recipe – Select a Dynamo recipe to your mannequin, inference engine, and GPU sort. Account for mannequin weights, KV cache, and runtime reminiscence when sizing staff. For supported configurations, AISimulate may also help evaluate aggregated and disaggregated serving, select parallelism, and measurement prefill and decode employee swimming pools earlier than testing on GPUs.
- Confirm communication earlier than benchmarking – For deployments spanning situations, affirm that communication makes use of the supposed EFA transport, together with KV-cache transfers between prefill and decode staff. See the Dynamo EFA guide.
- Benchmark consultant visitors – Replay your visitors with AIPerf, or use a generated agentic trace with consultant immediate lengths, output lengths, and prefix reuse. Sweep concurrency for every configuration, retain factors assembly your SLAs, and select primarily based on value and latency priorities.
- Tune from measured bottlenecks – Alter topology and employee allocation earlier than engine settings, altering one variable at a time. Observe the performance tuning guide, or use Dynamo optimization agent skills to help with benchmarking and tuning. Validate every candidate in your goal HyperPod deployment.
Conclusion
NVIDIA Dynamo on SageMaker HyperPod Inference simplifies deploying and working massive fashions with multi-node serving, disaggregated prefill and decode, and KV-aware routing. The managed add-on installs the serving parts, and configuration defaults simplify endpoint setup. HyperPod manages mannequin weights, auto-scales clusters primarily based on GPU and inference metrics, screens node well being, and replaces failed nodes. Collectively, these capabilities scale back undifferentiated infrastructure work so groups can concentrate on their fashions and functions. Get began with the setup directions and full deployment examples within the linked repository.
Acknowledgements
The authors want to thank Eliuth Triana, Felipe Garcia, Sanjana Checker, Junyuan Yang, Jiahong Liu, Anish Maddipoti, Jacob Kern and Abdullahi Olaoye from NVIDIA for his or her contributions to this publish. We additionally want to thank Can Solar, Ophelia Yang, Aryaman Kothari, David Oh, and Ayush Kumar from AWS for his or her contributions.
Concerning the authors


