Cut back LLM latency with prefix-aware routing on Amazon SageMaker Inference

While you construct an software on prime of a giant language mannequin (LLM), the immediate you ship to the mannequin sometimes has two components. There’s a set half that units up context (directions, reference paperwork, dialog historical past) and a variable half that accommodates the precise person enter. Take a customer support bot for example. Every request begins with the identical block of textual content: “You’re a assist agent for AnyCompany. Listed here are our insurance policies…” adopted by regardless of the buyer typed. The directions on the prime is perhaps 3,000 tokens. The client’s query on the backside is perhaps 50 tokens.

Because of this throughout a whole lot or hundreds of requests, your mannequin is processing that very same 3,000-token starting over and over.

LLM serving frameworks like vLLM and TensorRT-LLM have an answer for this. They cache the computed key-value (KV) pairs for immediate prefixes which were seen earlier than. When the identical starting exhibits up in a brand new request, the mannequin reuses the cached computation and solely processes the brand new tokens on the finish. That is referred to as prefix caching, and it could actually scale back time-to-first-token (TTFT) considerably.

However there’s an issue once you scale past a single occasion. When you have a fleet of machines behind an endpoint, requests get distributed throughout all of them. That very same 3,000-token prefix lands on occasion A for one request, occasion B for the following, occasion C after that. Every occasion computes it from scratch as a result of none of them see it regularly sufficient to construct a dependable cache. The prefix caching function is there, however the routing layer spreads requests too thinly for it to assist.

As we speak, Amazon SageMaker Inference introduces prefix-aware routing. It’s a new routing technique that appears originally of every request and persistently sends requests with the identical starting to the identical occasion. The KV cache on that occasion truly builds up and will get reused. In our benchmarks on Llama 3.1 70B, this diminished P50 TTFT by as much as 77 p.c and elevated throughput by as much as 16 p.c. It additionally pushed KV cache hit charges from roughly 25 p.c to over 80 p.c.

What prefix-aware routing does

When a request arrives at your endpoint, Amazon SageMaker appears originally of the payload and makes use of it to resolve which occasion ought to deal with it. The identical starting goes to the identical occasion. Totally different beginnings unfold throughout totally different cases. If 10 requests share a prefix or starting, all 10 go to the identical machine, and that machine’s cache stays heat for that prefix.

You don’t must tag requests or handle affinity your self. The endpoint handles it based mostly on the content material of the request.

There are two built-in safeguards:

Overload safety. If one prefix is extraordinarily fashionable and the goal occasion is already at capability, the endpoint routes the request to a much less busy occasion as an alternative. You configure the concurrency restrict, and the endpoint respects it. You would possibly miss a cache hit on that one request, however you keep away from overwhelming a single machine.

Secure conduct throughout scaling. While you add or take away cases, most requests proceed going to the identical occasion they have been going to earlier than. Solely a small fraction of visitors shifts to account for the modified fleet. Your caches don’t get invalidated each time you scale.

Efficiency benchmarks

We benchmarked prefix-aware routing towards the default random routing baseline utilizing Llama 3.1 70B Instruct on 7 ml.p5.48xlarge cases with vLLM (prefix caching enabled). We ran 16 check configurations masking single mannequin endpoints, inference part endpoints, the native Invoke API, and the OpenAI-compatible API. All checks accomplished with 100% success fee.

Lengthy context workloads

8,000-token shared prefixes, sustained over 1 hour:

• P90 TTFT: diminished by 33–37 p.c.
• P50 TTFT: diminished by 71–77 p.c.
• KV cache hit fee: from roughly 25–82 p.c.
• Throughput: elevated 15–16 p.c.

Brief context workloads

Variable-length ShareGPT-style conversations, half-hour:

• P90 TTFT: diminished by 24–37 p.c.
• P50 TTFT: diminished by 13–16 p.c.
• KV cache hit fee: from roughly 30–80 p.c.
• Throughput: elevated 1.7–2.0 p.c.

The longer your shared prefix, the larger the win. Lengthy context workloads profit essentially the most as a result of there may be extra computation to skip on every cache hit. Brief context workloads nonetheless profit, however the shared prefixes are smaller so the financial savings per request are proportionally smaller.

Routing overhead

The prefix-aware routing logic provides 1.3–1.9 milliseconds per request. Mannequin TTFT in these checks ranged 63–280 milliseconds. The routing price is negligible.

Visitors distribution remained balanced throughout all situations. Every of the 7 cases obtained 13.3–15.4 p.c of requests, inside 1 p.c of a really perfect even cut up. No scorching spots.

Routing methods on SageMaker Inference

With this launch, Amazon SageMaker Inference provides three routing methods for real-time endpoints:

RANDOM (default): Distributes requests uniformly throughout cases. Advisable for general-purpose workloads, non-LLM fashions, or a scenario the place requests are interchangeable and there’s no profit to sending particular requests to particular cases.

LEAST_OUTSTANDING_REQUESTS: Sends every request to the occasion with the fewest in-flight requests. Advisable when request processing occasions fluctuate and also you need to maintain all cases equally busy. Helps stop gradual requests from piling up on one machine whereas others sit idle.

PREFIX_AWARE (new): Sends requests sharing the identical immediate prefix to the identical occasion. Advisable for LLM workloads the place many requests share frequent textual content originally and your serving framework has prefix caching enabled.

You set the technique per manufacturing variant in your endpoint configuration. You may change between them by updating the endpoint configuration with out redeploying your mannequin.

When to make use of prefix-aware routing

The function delivers worth when your requests share textual content originally. Listed here are the patterns the place it has essentially the most impression:

Retrieval Augmented Technology (RAG) purposes. You retrieve a doc and prepend it earlier than the person’s query. When a number of customers ask questions on the identical doc, all of them share that doc as a prefix. Prefix-aware routing sends them to the identical occasion, the place the KV cache for that doc is already heat.

Multi-turn conversations. Every flip in a dialog consists of the complete historical past of earlier turns. Because the dialog grows, that shared historical past turns into an extended and costlier prefix. Routing on that prefix retains the dialog’s cache on one occasion throughout turns.

Templated bots and assistants. Bots with lengthy, structured directions (insurance policies, formatting guidelines, persona definitions) ship those self same directions with each request. Solely the person message on the finish modifications. Prefix-aware routing signifies that costly instruction block will get processed as soon as, not hundreds of occasions.

Code completion. Coding assistants embrace file contents as context. Whereas a developer works in the identical file, each completion request shares that file content material as a prefix.

Methods to allow it

You configure prefix-aware routing once you create your endpoint configuration. Two parameters management the conduct:

PrefixLength (1024–65536): How a lot of the request to make use of for routing. For the native Amazon SageMaker Invoke API, that is bytes from the start of the request physique. For the OpenAI-compatible API, that is characters from the extracted message textual content. Set this to cowl your shared prefix plus sufficient distinctive content material to unfold totally different workloads throughout cases.

ConcurrencyThreshold (1–1024): The utmost in-flight requests on the goal occasion earlier than overflow kicks in. If the goal occasion is at this restrict, the request goes to a much less loaded occasion as an alternative.

Right here is an instance:

aws sagemaker create-endpoint-config 
    --endpoint-config-name example-llm-config 
    --production-variants '[{
        "VariantName": "AllTraffic",
        "ModelName": "example-llm-model",
        "InitialInstanceCount": 3,
        "InstanceType": "ml.p5.48xlarge",
        "RoutingConfig": {
            "RoutingStrategy": "PREFIX_AWARE",
            "PrefixAwareRoutingConfig": {
                "PrefixLength": 4096,
                "ConcurrencyThreshold": 10
            }
        }
    }]'

Then create your endpoint as ordinary:

aws sagemaker create-endpoint 
    --endpoint-name example-llm-endpoint 
    --endpoint-config-name example-llm-config

No modifications to your mannequin container or serving framework are wanted. Prefix-aware routing operates fully on the endpoint routing layer.

Invoking the endpoint

Nothing modifications about the way you name the endpoint. The identical InvokeEndpoint and InvokeEndpointWithResponseStream APIs work precisely as earlier than:

aws sagemaker-runtime invoke-endpoint 
    --endpoint-name example-llm-endpoint 
    --content-type software/json 
    --body fileb://request.json 
    output.json

Identical for the OpenAI-compatible Chat Completion API:

from openai import OpenAI
from sagemaker.core.token_generator import generate_token

shopper = OpenAI(
    base_url=f"https://runtime.sagemaker.us-west-2.amazonaws.com"
             f"/endpoints/example-llm-endpoint/openai/v1",
    api_key=generate_token(area="us-west-2")
)
response = shopper.chat.completions.create(
    mannequin="example-model",
    messages=[
        {"role": "user", "content": "What is your return policy?"},
    ],
)

Multi-tenant prefix isolation

If totally different tenants share the identical immediate directions however you need them routed individually (to maintain cache contexts unbiased), cross an elective ID:

Native Invoke API: set the X-Amzn-SageMaker-Prefix-Conscious-Id header (as much as 64 ASCII characters).

OpenAI API: embrace the prompt_cache_key area within the request physique.

This ID combines with the prefix in order that requests with similar prefixes however totally different IDs land on totally different cases.

Inference elements and LoRA adapters

Prefix-aware routing works with inference part endpoints and dynamic Low-Rank Adaptation (LoRA) adapters. For inference elements, it behaves the identical as single mannequin endpoints. For LoRA adapters, it operates inside the adapter’s sticky occasion set, utilizing prefix-based choice among the many cases that have already got the adapter loaded.

Sensible steerage

Allow prefix caching in your serving framework. Prefix-aware routing will get repeated prefixes to the identical occasion, however your container wants prefix caching turned on to truly retailer and reuse these KV pairs. In vLLM, that is enabled by default in current variations. Different frameworks would possibly require specific configuration.

Hold request serialization constant. For the native Invoke API, PrefixLength operates on uncooked bytes. JSON whitespace, key ordering, and formatting all have an effect on routing. For those who serialize the identical immediate in another way throughout requests, they may find yourself on totally different cases. Use constant serialization.

Dimension PrefixLength rigorously. Too brief and all requests with the identical brief prefix get funneled to at least one occasion, triggering overflow. Too lengthy and small payload variations (like temperature values) scatter requests that ought to keep collectively. Begin with the size of your shared prefix plus a modest buffer.

You want a minimum of two cases. With one occasion, all requests go to the identical place no matter technique.

Monitor cache hit charges. Allow SageMaker detailed observability to trace KV cache hit charges on the mannequin stage. This confirms whether or not prefix-aware routing is working in your particular workload.

Conclusion

Prefix-aware routing is obtainable right now on SageMaker real-time inference endpoints. Replace your AWS SDK or CLI to the most recent model to entry the brand new RoutingStrategy and PrefixAwareRoutingConfig parameters. Seek advice from this pocket book for examples of tips on how to allow it throughout endpoint creation.

In regards to the authors