Deploying Qwen3.8-2.4T-A95B on Amazon SageMaker HyperPod with vLLM

On August 12, 2026, Alibaba’s Qwen crew launched Qwen3.8-2.4T-A95B. That is the primary time a Qwen-Max-class mannequin has been made obtainable as open weights. With 2.4 trillion complete parameters (95 billion activated per token), a hybrid linear-plus-full-attention structure, and native context as much as 262K tokens (extensible to 1M), Qwen3.8 targets probably the most demanding agentic and reasoning workloads. These embrace multi-step coding, long-horizon planning, and autonomous instrument use.

Open weights fashions provide you with full management. Information stays inside your infrastructure, inference habits might be personalized, and there are not any per-token API charges at scale. The trade-off is operational: internet hosting a 2.4T-parameter mannequin requires purpose-built GPU infrastructure and an optimized serving stack.

On this publish we present tips on how to deploy Qwen3.8-2.4T-A95B on Amazon SageMaker HyperPod utilizing vLLM on a ml.p6-b300 occasion (8× NVIDIA B300 Blackwell Extremely GPUs). We cowl the total path from cluster provisioning to an OpenAI-compatible endpoint, together with vLLM configuration for NVFP4 quantization, built-in reasoning, instrument calling, and native Multi-Token Prediction (MTP) speculative decoding.

That is the second publish in our sequence on deploying open trillion-parameter fashions on Amazon SageMaker HyperPod. For the primary publish overlaying Kimi K3, see Deploying Kimi K3 on Amazon SageMaker HyperPod and Amazon EKS.

Qwen3.8-2.4T-A95B at a look

Qwen3.8-2.4T-A95B (the open-weight launch of Qwen3.8-Max) is the biggest and most succesful mannequin within the Qwen household. The next is a abstract of the important thing architectural particulars related to deployment.

Structure

The hybrid consideration design is essential to environment friendly long-context inference. Gated DeltaNet layers (69 of 92) use linear consideration with a bounded recurrent state, changing the rising KV-cache with a fixed-size reminiscence. Gated Consideration layers (23 of 92) use full quadratic consideration for high-fidelity token interactions. This 3:1 ratio retains each compute and reminiscence bounded as context scales towards 1M tokens. This can be a vital property for agentic workloads that accumulate instrument outputs, code, and reasoning traces throughout many turns.

The fine-grained MoE distributes capability throughout 512 small specialists quite than just a few giant ones, enhancing routing effectivity and specialization. Solely roughly 95B parameters are energetic per ahead go, so serving prices observe activated parameters, not the total 2.4T.

Capabilities and reasoning management

Qwen3.8 is designed for agentic execution: multi-step coding, autonomous instrument use, long-horizon planning, and sophisticated analysis workflows. It contains built-in reasoning controls by means of the reasoning_effort parameter (low, medium, excessive), so builders can commerce compute for reasoning depth per request. Dial up for arduous multi-step issues and dial down for high-throughput duties.

Mannequin weights and quantization

The open weights are printed on Hugging Face in the usual Transformers format. Neighborhood quantizations embrace MXFP4 and NVFP4 (W4A4), which compress the mannequin to roughly 1.2 TB, becoming on a single 8-GPU node with B300 Blackwell Extremely GPUs.

Benchmark highlights

In accordance with the seller’s benchmarking outcomes, Qwen3.8-2.4T-A95 exhibits explicit energy in analysis workflows (PaperBench 93.0), instruction following (IFBench 82.8), and terminal-based coding (86.6). It performs comparably with main frontier fashions throughout most classes, with remaining headroom on more durable repository-level duties (SWE-bench Professional) and common instrument use (Toolathlon). For organizations evaluating self-hosted alternate options to proprietary APIs, these outcomes place Qwen3.8-2.4T-A95 as a reputable frontier-class choice, notably for coding brokers and analysis pipelines.

Why Amazon SageMaker HyperPod for giant MoE inference

Deploying a 2.4T-parameter mannequin is just not solely a GPU drawback. It requires orchestration that handles mannequin obtain, container scheduling, well being monitoring, autoscaling, and node failures with out guide intervention. Amazon SageMaker HyperPod is purpose-built for this class of workload.

EKS-orchestrated clusters. HyperPod clusters use Amazon Elastic Kubernetes Service (Amazon EKS) because the management aircraft. You get the total Kubernetes panorama (kubectl, Helm charts, customized useful resource definitions), whereas AWS manages the underlying infrastructure lifecycle: networking, storage, GPU driver set up, and the NVIDIA machine plugin.

Inference Operator. The HyperPod Inference Operator (put in routinely or as an EKS Add-on) offers a single customized useful resource definition (CRD), InferenceEndpointConfig, that declaratively specifies your mannequin, container picture, GPU useful resource requests, and vLLM launch arguments. The operator handles:

• Mannequin weight obtain (from Hugging Face Hub, Amazon Easy Storage Service (Amazon S3), or Amazon FSx).
• Container scheduling and GPU allocation.
• Well being checks and readiness gates.
• Rolling updates and endpoint lifecycle administration.
• Autoscaling by means of KEDA with Amazon CloudWatch or Prometheus metrics.

Reserved capability with Versatile Coaching Plans. The ml.p6-b300.48xlarge occasion sort requires reserved capability. Versatile Coaching Plans present dedicated GPU reservations that may be allotted on to your HyperPod cluster. There’s no rivalry with on-demand swimming pools and no cold-start capability threat.

Resilience. HyperPod repeatedly displays node well being and routinely replaces degraded nodes. For sustained inference workloads working 24/7, this alleviates the operational overhead of manually detecting and recovering from {hardware} failures.

Further inference options (Inference Operator v3.x):

• Disaggregated Prefill and Decode (DPD) – separates prefill and decode onto distinct GPU swimming pools for predictable per-token latency beneath concurrent load.
• Inference information seize – log inputs/outputs on the endpoint, load balancer, or pod degree.
• Native NVMe mannequin deployment – load weights from node-local storage to cut back cold-start latency.
• Amazon Route 53 DNS administration – computerized customized area information to your endpoints.

In brief: You write a YAML manifest describing what to deploy. HyperPod handles how to run it reliably at scale.

Infrastructure sizing: Matching {hardware} to the mannequin

The p6-b300 occasion

The ml.p6-b300.48xlarge offers the compute density required for single-node serving of Qwen3.8:

Why NVFP4 quantization

At BF16 precision, Qwen3.8’s 2.4T parameters require roughly 4.8 TB of reminiscence for weights alone, exceeding a single 8-GPU node. NVFP4 (W4A4) quantization compresses weights to roughly 4 bits per parameter, bringing the overall weight footprint to roughly 1.2 TB. This suits comfortably throughout the 2.1 TB of combination GPU reminiscence on a p6-b300 occasion, leaving headroom for KV-cache and activations.

Reminiscence price range

A tough breakdown for a single p6-b300 node:

The hybrid consideration structure is a key benefit right here: the 69 DeltaNet layers keep a fixed-size recurrent state no matter context size, not like conventional fashions the place KV-cache grows linearly with each layer. Solely the 23 full-attention layers contribute to context-dependent reminiscence progress.

Throughput expectations

Reference numbers from NVIDIA’s Day-0 benchmarks on GB300 NVL72 (FP8, 72 GPUs): >4K tokens/sec/GPU, >350 tokens/sec/consumer. A single 8-GPU p6-b300 node with NVFP4 will ship proportionally decrease combination throughput however stays well-suited for manufacturing inference workloads with reasonable concurrency.

Capability procurement

The ml.p6-b300.48xlarge occasion sort isn’t obtainable on-demand. You need to procure capability by means of a Versatile Coaching Plan, a dedicated reservation of GPU availability to your HyperPod cluster. Set the goal Availability Zone to match your plan’s allocation when configuring the occasion group.

vLLM configuration deep dive

This part particulars the vLLM serving parameters for Qwen3.8 on a single p6-b300 node. The configuration is knowledgeable by the vLLM recipe for Qwen3.8 on B300 (NVFP4).

Base serving command

The total vllm serve invocation:

vllm serve Inferact/Qwen3.8-2.4T-A95B-NVFP4 
    --tensor-parallel-size 8 
    --quantization nvfp4 
    --load-format fastsafetensors 
    --trust-remote-code 
    --enable-prefix-caching 
    --moe-backend auto 
    --reasoning-parser qwen3 
    --enable-auto-tool-choice 
    --tool-call-parser qwen3 
    --speculative-config '{"methodology":"mtp","num_speculative_tokens":1}' 
    --served-model-name Qwen3.8

Key flags defined:

• --tensor-parallel-size 8 – shards the mannequin throughout all 8 B300 GPUs.
• --quantization nvfp4 – prompts NVIDIA FP4 (W4A4) quantization so the two.4T mannequin suits in 2.1 TB of GPU reminiscence.
• --load-format fastsafetensors – makes use of accelerated weight deserialization for quicker cold-start.
• --trust-remote-code – required for Qwen3.8’s customized modeling code on Hugging Face.
• --enable-prefix-caching – reuses computed KV-cache throughout requests that share immediate prefixes. Vital for multi-turn agentic conversations the place the system immediate and dialog historical past repeat.
• --moe-backend auto – lets vLLM choose the optimum MoE dispatch kernel for the {hardware}.

Reasoning (pondering mode)

The --reasoning-parser qwen3 flag extracts reasoning content material from the mannequin’s <assume>...</assume> output blocks. Key behaviors:

• Qwen3.8 reasoning is enabled by default – no further flag wanted on the mannequin aspect.
• The API response separates reasoning_content (the pondering hint) from content material (the ultimate reply).
• To disable pondering per-request, go extra_body={"chat_template_kwargs": {"enable_thinking": False}} within the consumer name.
• Structured output (guided_json, guided_regex) works alongside reasoning – the structured output engine constrains solely the content material subject.

Software calling (operate calling)

The --enable-auto-tool-choice and --tool-call-parser qwen3 flags allow OpenAI-compatible operate calling:

• Helps tool_choice values: auto, required, none, and named capabilities.
• Software calls are parsed from the content material subject solely — the reasoning_content is just not parsed for operate calls. This implies the mannequin can purpose about which instrument to name, then emit the structured name individually.
• When tool_choice="auto" and strict: true is ready on a instrument definition, vLLM enforces schema-constrained decoding for instrument arguments, facilitating legitimate JSON output.

Speculative decoding (native MTP)

The --speculative-config '{"methodology":"mtp","num_speculative_tokens":1}' flag allows Multi-Token Prediction utilizing Qwen3.8’s built-in draft heads:

• Qwen3.8 was skilled with MTP – light-weight draft heads are bundled within the mannequin weights. No separate draft mannequin obtain or configuration is required.
• The draft head predicts the following N tokens in parallel, then verifies them in a single ahead go. Accepted tokens skip particular person decode steps, rising throughput.
• num_speculative_tokens: 1 is the protected place to begin. Enhance to 2–3 for throughput-sensitive workloads when you’ve validated that the acceptance price stays excessive (monitor by means of vLLM’s /metrics endpoint).
• MTP provides minimal latency overhead on the draft step as a result of the heads reuse the mannequin’s present hidden states.

Deployment walkthrough on SageMaker HyperPod

The entire deployment manifests and scripts used on this publish can be found in our GitHub repository.

Stipulations

Earlier than deploying the mannequin, you want a working HyperPod cluster with p6-b300 capability:

1. Create a HyperPod cluster with EKS orchestration. Within the Amazon SageMaker AI console, navigate to HyperPod Clusters, then select Create. Select Orchestrated by Amazon EKS and both Fast Setup (default networking and IAM) or Customized Setup with an present digital personal cloud (VPC) and subnets. Ensure that Use default Helm charts and add-ons is chosen so the Inference Operator is put in routinely.
2. Provision a Versatile Coaching Plan. Underneath the occasion group configuration, choose Coaching plan because the capability supply. Create or connect a plan overlaying ml.p6-b300.48xlarge with the occasion rely and length you want. Set the Goal Availability Zone to match the plan’s allocation.
3. Add a p6-b300 employee group. Add an occasion group with ml.p6-b300.48xlarge and at the least 1 occasion. Watch for the cluster to achieve Energetic state with wholesome GPU nodes.
4. Confirm entry. Verify you possibly can attain the cluster:

   kubectl get nodes
   # Anticipate node(s) with nvidia.com/gpu: 8 capability

InferenceEndpointConfig manifest

Apply the next InferenceEndpointConfig to deploy Qwen3.8 with the vLLM configuration:

apiVersion: inference.sagemaker.aws.amazon.com/v1
form: InferenceEndpointConfig
metadata:
  title: qwen38
spec:
  modelName: qwen38
  instanceType: ml.p6-b300.48xlarge
  invocationEndpoint: v1/chat/completions
  replicas: 1
  modelSourceConfig:
    huggingFaceModel:
      modelId: Inferact/Qwen3.8-2.4T-A95B-NVFP4
    modelSourceType: huggingface
  employee:
    picture: vllm/vllm-openai:qwen38
    modelInvocationPort:
      containerPort: 8000
      title: http
    modelVolumeMount:
      mountPath: /decide/ml/mannequin
      title: model-weights
    assets:
      limits:
        nvidia.com/gpu: 8
      requests:
        nvidia.com/gpu: 8
    args:
      - "--model"
      - "/decide/ml/mannequin"
      - "--serving-model-name"
      - "Qwen3.8"
      - "--linear-backend"
      - "flashinfer_cutedsl"
      - "--trust-remote-code"
      - "--enable-prefix-caching"
      - "--enable-auto-tool-choice"
      - "--tool-call-parser"
      - "qwen3_coder"
      - "--reasoning-parser"
      - "qwen3"
      - "--served-model-name"
      - "Qwen3.8"
      - "--tensor-parallel-size"
      - "8"
    environmentVariables:
      - title: "VLLM_ENGINE_READY_TIMEOUT_S"
        worth: "1800"

This manifest can be obtainable within the GitHub repository.

Making use of and monitoring

Apply the manifest:

kubectl apply -f qwen.yaml

Monitor the deployment progress:

# Watch the InferenceEndpointConfig standing
kubectl get inferenceendpointconfig qwen38 -w

# Examine pod standing (mannequin obtain and container startup)
kubectl get pods -l model-name=qwen38

# View vLLM startup logs
kubectl logs -f <pod-name> --tail=100

The deployment proceeds by means of these phases: mannequin obtain (roughly 1.2 TB from Hugging Face, time depends upon community bandwidth) then weight loading (fastsafetensors deserialization to GPU reminiscence) then well being checks go and endpoint prepared. On a contemporary deployment with no cached weights, anticipate 15–half-hour for the total sequence. Subsequent restarts with native NVMe caching are considerably quicker.

After the endpoint exhibits Prepared, you possibly can ship requests to the service:

# Get the service endpoint
kubectl get svc -l model-name=qwen38

# Fast well being test
curl http://<service-endpoint>:8000/well being

Inference in motion: Calling the endpoint

After the endpoint is prepared, it exposes an OpenAI-compatible API. You should utilize the usual OpenAI Python SDK, curl, or one other HTTP consumer (notice that on this deployment instance the endpoint isn’t uncovered to the general public web).

Fundamental chat completion (with reasoning)

from openai import OpenAI

consumer = OpenAI(
    base_url="http://<service-endpoint>:8000/v1",
    api_key="unused",  # vLLM doesn't require auth by default
)

response = consumer.chat.completions.create(
    mannequin="Qwen3.8",
    messages=[{"role": "user", "content": "Explain the trade-offs of MoE vs dense models for inference."}],
    temperature=0.6,
    top_p=0.95,
)

# Reasoning hint (the mannequin's pondering)
print("Considering:", response.decisions[0].message.reasoning_content)

# Ultimate reply
print("Reply:", response.decisions[0].message.content material)

To manage reasoning depth per request, go reasoning_effort:

response = consumer.chat.completions.create(
    mannequin="Qwen3.8",
    messages=[{"role": "user", "content": "What is 2+2?"}],
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},  # disable pondering completely
)

For streaming responses, test for the reasoning_content attribute on every chunk’s delta:

stream = consumer.chat.completions.create(
    mannequin="Qwen3.8",
    messages=[{"role": "user", "content": "Write a Python quicksort."}],
    stream=True,
)

for chunk in stream:
    delta = chunk.decisions[0].delta
    if hasattr(delta, "reasoning_content") and delta.reasoning_content:
        print(delta.reasoning_content, finish="", flush=True)
    elif delta.content material:
        print(delta.content material, finish="", flush=True)

Software calling instance

instruments = [{
    "type": "function",
    "function": {
        "name": "get_stock_price",
        "description": "Get the current stock price for a ticker symbol",
        "parameters": {
            "type": "object",
            "properties": {
                "ticker": {"type": "string", "description": "Stock ticker, e.g. 'AMZN'"}
            },
            "required": ["ticker"],
            "additionalProperties": False,
        },
        "strict": True,
    }
}]

response = consumer.chat.completions.create(
    mannequin="Qwen3.8",
    messages=[{"role": "user", "content": "What's Amazon's stock price right now?"}],
    instruments=instruments,
    tool_choice="auto",
)

# The mannequin causes internally, then emits a structured instrument name
print("Considering:", response.decisions[0].message.reasoning_content)

tool_call = response.decisions[0].message.tool_calls[0].operate
print(f"Operate: {tool_call.title}, Args: {tool_call.arguments}")

With strict: True set on the instrument definition, vLLM makes certain the arguments conform to the JSON schema. No post-validation is required.

Really useful sampling parameters

For many use circumstances with Qwen3.8:

• temperature=0.6 – balances creativity and coherence. Use 0.0 for deterministic outputs (for instance, structured extraction).
• top_p=0.95 – customary nucleus sampling.
• top_k=20 – limits vocabulary at every step and reduces degenerate outputs on lengthy generations.
• max_tokens – set generously when pondering is enabled, for the reason that reasoning_content counts towards the token price range. For complicated reasoning duties, 32,768–65,536 is an inexpensive ceiling.

A curl instance for fast validation:

curl http://<service-endpoint>:8000/v1/chat/completions 
  -H "Content material-Kind: software/json" 
  -d '{
    "mannequin": "Qwen3.8",
    "messages": [{"role": "user", "content": "Hello, Qwen3.8!"}],
    "temperature": 0.6,
    "max_tokens": 256
  }'

Efficiency tuning ideas

With the endpoint working, the next tuning levers allow you to optimize to your particular workload sample.

Benchmark outcomes

We benchmarked Qwen3.8-2.4T-A95B on a single p6-b300 occasion (8× B300 GPUs) utilizing 512 requests with 1,024 enter tokens and 1,024 output tokens at a concurrency of 32. We examined 4 configurations to isolate the affect of Knowledgeable Parallelism (EP) and Multi-Token Prediction (MTP) speculative decoding:

• TP – Tensor Parallelism solely (TP=8), our baseline.
• TP+MTP – TP=8 with native MTP speculative decoding (num_speculative_tokens: 1).
• TP+EP – TP=8 with Knowledgeable Parallelism enabled.
• TP+EP+MTP – TP=8 with each EP and MTP enabled.

Determine 2: Benchmark comparability of TP, MTP, and EP configurations on a p6-b300 node

Key findings (share enchancment in comparison with the TP baseline):

Evaluation:

• MTP is the dominant optimization for TTFT. Enabling speculative decoding (even with just one draft token) cuts time-to-first-token (TTFT) by practically 59 %, from 1,244 ms to 513 ms. It’s because the MTP draft head predicts the primary output tokens in parallel with the ultimate prefill steps, overlapping compute.
• EP alone offers modest features (roughly 3.5 % TTFT, roughly 1 % throughput). The profit is extra pronounced at greater concurrency ranges the place skilled routing rivalry turns into a bottleneck.
• The mixture of EP+MTP delivers the most effective total consequence: 59.7 % TTFT discount, 12.2 % latency discount, and 12.6 % greater output throughput. The 2 optimizations are complementary: EP reduces skilled dispatch overhead whereas MTP reduces decode latency.
• Inter-token latency (not proven in chart) additionally improves: 17.97 ms (TP) to 17.33 ms (TP+MTP) to 16.36 ms (TP+EP+MTP), a 9 % discount with the total configuration.

Advice: For manufacturing deployments, allow each EP and MTP (--enable-expert-parallel + --speculative-config '{"methodology":"mtp","num_speculative_tokens":1}'). The mixed configuration delivers the most effective latency and throughput profile with minimal extra complexity.

Prefix caching

Enabled with --enable-prefix-caching (already in our config). vLLM’s Computerized Prefix Caching (APC) reuses KV-cache blocks throughout requests that share the identical immediate prefix. That is widespread in multi-turn agentic conversations the place the system immediate and dialog historical past repeat. For workloads with excessive prefix overlap, this may cut back time-to-first-token (TTFT) by 50–80 % on repeated turns. No draw back for workloads with out prefix sharing. Unused cached blocks are evicted routinely.

Speculative decoding (MTP) tuning

Our config begins with num_speculative_tokens: 1 (one further token drafted per step). Tuning steerage:

• Enhance to 2–3 for throughput-sensitive, low-concurrency workloads. Every extra speculative token will increase the variety of tokens that may be accepted per step, but in addition will increase draft overhead and verification price.
• Monitor acceptance price by means of the vLLM /metrics endpoint (spec_decode_acceptance_rate). If acceptance stays above 70–80 %, rising num_speculative_tokens is worthwhile. Beneath 50 %, cut back it or disable hypothesis.
• Excessive-concurrency caveat: at excessive queries per second (QPS), speculative decoding consumes extra GPU compute for drafting and verification. Underneath saturation, the overhead can cut back combination throughput. Think about disabling MTP beneath heavy batch load and enabling it just for latency-sensitive single-stream requests.
• MTP offers the most effective features for memory-bound decode workloads (lengthy outputs, low batch measurement) — precisely the sample for agentic reasoning duties with prolonged <assume> traces.

Reminiscence and context size

--gpu-memory-utilization (default: 0.9) controls how a lot GPU reminiscence vLLM pre-allocates for KV-cache. For Qwen3.8 with NVFP4, the mannequin weights occupy roughly 57 % of GPU reminiscence, leaving roughly 43 % (roughly 900 GB) for cache and overhead. Tuning:

• Hold at 0.9 for max throughput (extra KV-cache slots = extra concurrent requests).
• Scale back to 0.85 for those who observe out-of-memory (OOM) errors throughout long-context requests. This sacrifices batch capability however helps forestall preemption cascades.

--max-model-len caps the utmost sequence size vLLM will settle for. Setting this decrease than the mannequin’s full 262K context provides vLLM extra KV-cache slots for shorter requests, enhancing concurrency. Set it to match your precise workload’s most context requirement. For instance, 32,768 for typical coding brokers or 131,072 for long-document evaluation.

Batching and scheduling

--max-num-seqs (default: 256) limits concurrent sequences in a batch. For MoE fashions with giant per-token compute, lowering this to 64–128 decreases scheduling overhead and makes certain every request will get enough GPU consideration, enhancing per-request latency at the price of combination throughput.

--max-num-batched-tokens controls the overall token price range per scheduling step (prefill + decode mixed). vLLM V1 makes use of chunked prefill by default. Giant prefills are cut up into chunks and interleaved with decode steps:

• Decrease values (for instance, 8192) present higher inter-token latency (ITL), as a result of decodes aren’t stalled by giant prefills.
• Larger values (for instance, 32768+) result in higher TTFT, since extra prefill tokens are processed per batch.
• For agentic workloads with blended brief and lengthy requests, begin at 16384 and modify based mostly on noticed P99 ITL.

MoE backend choice

--moe-backend auto lets vLLM select the optimum kernel. On Blackwell GPUs, this sometimes selects fused MoE kernels that deal with sparse skilled dispatch as grouped-GEMM operations, minimizing per-expert scheduling overhead. If profiling (utilizing nsys or vLLM’s built-in profiler) reveals MoE dispatch as a bottleneck, you possibly can experiment with specific backends, however auto is the really helpful place to begin.

Knowledgeable parallelism (superior)

For deployments needing greater concurrency than a single TP=8 configuration permits, vLLM helps Knowledgeable Parallelism (EP) the place MoE specialists are distributed throughout GPUs independently of consideration heads. On a single p6-b300 node, you can mix TP=4 + EP=2. This halves the variety of specialists per GPU, liberating reminiscence for bigger KV-cache at the price of elevated all-to-all communication throughout skilled dispatch. Profile earlier than adopting: on NVLink-connected GPUs the communication price is low, however features rely in your concurrency goal.

Monitoring

vLLM exposes Prometheus metrics at /metrics. Key gauges to observe:

• vllm:num_requests_running – present batch measurement. Spikes point out queuing.
• vllm:gpu_cache_usage_perc – KV-cache utilization. Sustained >95 % alerts preemption threat.
• vllm:num_preemptions_total – if this climbs, improve gpu_memory_utilization or cut back max_num_seqs.
• vllm:spec_decode_acceptance_rate – MTP acceptance price. Tune num_speculative_tokens accordingly.
• vllm:e2e_request_latency_seconds – end-to-end latency distribution for service degree settlement (SLA) monitoring.

For HyperPod deployments, these metrics might be scraped by Amazon Managed Prometheus and visualized in Grafana, or fed into KEDA for autoscaling selections.

Clear up

To keep away from ongoing prices, take away the assets in reverse order:

# Delete the inference endpoint
kubectl delete inferenceendpointconfig qwen38

# (Non-compulsory) Scale down or delete the HyperPod cluster
# through the SageMaker AI console or CLI

In case you not want GPU capability, launch or let your Versatile Coaching Plan expire. Cluster deletion stops compute prices. The coaching plan dedication stays till its time period ends no matter utilization.

Conclusion and subsequent steps

On this publish we deployed Qwen3.8-2.4T-A95B, a 2.4-trillion-parameter open-weight mannequin, on a single p6-b300 occasion utilizing Amazon SageMaker HyperPod and vLLM. The deployment exposes a production-ready OpenAI-compatible endpoint with built-in reasoning, instrument calling, and native MTP speculative decoding, all managed by means of a single Kubernetes manifest.

From right here you possibly can:

• Benchmark along with your workload – run vllm bench serve or your individual analysis harness towards the endpoint to determine baseline throughput and latency to your particular immediate distribution.
• Combine into agentic pipelines – join the endpoint to your orchestration framework (LangChain, CrewAI, or customized) utilizing the usual OpenAI SDK. The tool-calling and reasoning outputs are appropriate with out extra configuration.
• Positive-tune to your area – use NVIDIA NeMo or Amazon SageMaker coaching jobs to supply domain-adapted checkpoints, then redeploy with the identical manifest by swapping the mannequin ID.
• Add guardrails – layer I/O filtering (for instance, NVIDIA NeMo Guardrails or a customized proxy) in entrance of the vLLM endpoint for content material security and coverage compliance.

References

In regards to the authors