Improve generative AI inference throughput by as much as 2x whereas lowering prices by as much as 50% in Amazon SageMaker with the brand new Inference Optimization Toolkit – Half 2

As generative synthetic intelligence (AI) inference turns into more and more necessary to enterprises, clients are exploring the way to scale their generative AI operations and the way to combine generative AI fashions into current workflows. Mannequin optimization has emerged as a key step for organizations to stability cost-efficiency and responsiveness to enhance productiveness. Nevertheless, value and efficiency necessities fluctuate considerably throughout use circumstances. For chat purposes, minimizing latency is vital to ship an interactive expertise, whereas real-time purposes equivalent to suggestions require maximizing throughput. Navigating these trade-offs is a significant problem in quickly adopting generative AI, as completely different optimization methods should be rigorously chosen and evaluated.

To beat these challenges, we’re completely satisfied to introduce the Inference Optimization Toolkit, a totally managed mannequin optimization characteristic in Amazon SageMaker. This new characteristic reduces the price of generative AI fashions equivalent to Llama 3, Mistral, and Mixtral fashions by as much as 50%, whereas delivering as much as 2x greater throughput. For instance, the Llama 3-70B mannequin can now obtain as much as 2400 tokens/sec on an ml.p5.48xlarge occasion, in comparison with the earlier 1200 tokens/sec with out optimization.

This inference optimization toolkit makes use of the most recent generative AI mannequin optimization methods, together with: compile, Quantizationand Speculative Decoding It reduces the time it takes to optimize generative AI fashions from months to hours, enabling you to attain the perfect price-performance to your use case. For compilation, the toolkit makes use of the Neuron compiler to optimize the mannequin’s computational graph for particular {hardware}, equivalent to AWS Inferentia, to hurry up execution time and scale back useful resource utilization. For quantization, the toolkit leverages Activation-aware Weight Quantization (AWQ) to effectively scale back the scale and reminiscence footprint of your mannequin whereas sustaining high quality. For speculative decoding, the toolkit makes use of a quicker draft mannequin to foretell candidate outputs in parallel, enhancing inference pace for lengthy textual content technology duties. For extra details about every approach, see Optimizing Mannequin Inference in Amazon SageMaker. For extra data and benchmark outcomes on standard open supply fashions, see Scale back prices by as much as 50% whereas growing generative AI inference throughput by as much as 2X in Amazon SageMaker with the brand new inference optimization toolkit – Half 1.

This put up describes the way to get began with the Mannequin Inference Optimization Toolkit, which is supported by Amazon SageMaker JumpStart, and Amazon SageMaker Python SDKSageMaker JumpStart is a totally managed mannequin hub the place you’ll be able to discover, fine-tune, and deploy standard open-source fashions with just some clicks. You should use pre-optimized fashions or create your personal {custom} optimizations. Alternatively, you are able to do this utilizing the SageMaker Python SDK, as proven within the following diagram: NoteFor a whole record of supported fashions, see Optimizing Mannequin Inference with Amazon SageMaker.

Utilizing pre-optimized fashions with SageMaker JumpStart

The Inference Optimization Toolkit supplies pre-optimized fashions which can be optimized for best-in-class value efficiency at scale with out compromising accuracy. You may select a configuration based mostly on the latency and throughput necessities of your use case and deploy it with one click on.

Take the Meta-Llama-3-8b mannequin from SageMaker JumpStart for instance: increase From the Mannequin web page, in Deployment configuration, you’ll be able to increase the mannequin configuration choices, choose the variety of concurrent customers, and deploy the optimized mannequin.

Deploying pre-optimized fashions utilizing the SageMaker Python SDK

You may also use the SageMaker Python SDK to deploy pre-optimized generative AI fashions with just some strains of code. The next code: ModelBuilder A category for SageMaker JumpStart fashions. ModelBuilder is a category within the SageMaker Python SDK that offers you fine-grained management over numerous features of deployment, equivalent to occasion sort, community isolation, and useful resource allocation. You should use it to transform framework fashions (equivalent to XGBoost or PyTorch) or inference specs into SageMaker-compatible fashions and create deployable mannequin cases. For extra data, see Creating Fashions Utilizing ModelBuilder in Amazon SageMaker.

sample_input = {
    "inputs": "Hey, I am a language mannequin,",
    "parameters": {"max_new_tokens":128, "do_sample":True}
}

sample_output = [
    {
        "generated_text": "Hello, I'm a language model, and I'm here to help you with your English."
    }
]
schema_builder = SchemaBuilder(sample_input, sample_output)

builder = ModelBuilder(
    mannequin="meta-textgeneration-llama-3-8b", # JumpStart mannequin ID
    schema_builder=schema_builder,
    role_arn=role_arn,
)

Use the next code to record the obtainable pre-benchmark configurations:

builder.display_benchmark_metrics()

Select the appropriate one instance_type and config_name Choose from the record based mostly in your necessities for variety of concurrent customers, latency, and throughput. Within the desk above, you’ll be able to see the latency and throughput at numerous concurrency ranges for a selected occasion sort and configuration title. If the configuration title is lmi-optimizedwhich implies the configuration has been pre-optimized by SageMaker. Then, .construct() Run the optimization job. After the job is full, you’ll be able to deploy it to an endpoint to check the mannequin predictions. See the next code:

# set deployment config with pre-configured optimization
bulder.set_deployment_config(
    instance_type="ml.g5.12xlarge", 
    config_name="lmi-optimized"
)

# construct the deployable mannequin
mannequin = builder.construct()

# deploy the mannequin to a SageMaker endpoint
predictor = mannequin.deploy(accept_eula=True)

# use pattern enter payload to check the deployed endpoint
predictor.predict(sample_input)

Making a {custom} optimization utilizing the Inference Optimization Toolkit

Along with creating pre-optimized fashions, you may as well create {custom} optimizations based mostly on the occasion sort you choose. The next desk reveals an entire record of accessible combos. Within the subsequent sections, we first talk about compiling on AWS Inferentia, after which we discover different optimization methods for GPU cases.

Compiling from SageMaker JumpStart

To compile, we select the identical Meta-Llama-3-8b mannequin from SageMaker JumpStart. Optimize On the mannequin web page. On the optimization settings web page, you’ll be able to choose ml.inf2.8xlarge because the occasion sort. Then, specify the output Amazon Easy Storage Service (Amazon S3) location for the optimized artifacts. For instance, for a big mannequin like Llama 2 70B, the compilation job can take greater than an hour. Due to this fact, we suggest that you just use the Inference Optimization Toolkit to carry out ahead-of-time compilation. That manner, you solely have to compile as soon as.

Compiling with the SageMaker Python SDK

Within the SageMaker Python SDK, you’ll be able to configure compilation by altering setting variables within the .optimize() operate. For extra data, see compilation_configrefer LMI NeuronX Model Pre-Compilation Tutorial.

compiled_model = builder.optimize(
    instance_type="ml.inf2.8xlarge",
    accept_eula=True,
    compilation_config={
        "OverrideEnvironment": {
            "OPTION_TENSOR_PARALLEL_DEGREE": "2",
            "OPTION_N_POSITIONS": "2048",
            "OPTION_DTYPE": "fp16",
            "OPTION_ROLLING_BATCH": "auto",
            "OPTION_MAX_ROLLING_BATCH_SIZE": "4",
            "OPTION_NEURON_OPTIMIZE_LEVEL": "2",
        }
   },
   output_path=f"s3://{output_bucket_name}/compiled/"
)

# deploy the compiled mannequin to a SageMaker endpoint
predictor = compiled_model.deploy(accept_eula=True)

# use pattern enter payload to check the deployed endpoint
predictor.predict(sample_input)

Quantization and Inferential Decoding from SageMaker JumpStart

When optimizing your mannequin on a GPU, ml.g5.12xlarge is the default deployment occasion sort for Llama-3-8b. You may select quantization, speculative decoding, or each as your optimization choices. Quantization makes use of AWQ to cut back your mannequin weights to a low-bit (INT4) illustration. Lastly, you’ll be able to present an output S3 URL to retailer the optimized artifacts.

Speculative decoding means that you can enhance latency and throughput through the use of a draft mannequin offered by SageMaker, bringing your personal draft mannequin from the general public Hugging Face mannequin hub, or importing from your personal S3 bucket.

As soon as the optimization job is full, you’ll be able to deploy the mannequin or run additional analysis jobs on the optimized mannequin. Within the SageMaker Studio UI, you’ll be able to select to make use of the default instance dataset or present your personal dataset utilizing an S3 URI. On the time of writing, the efficiency analysis possibility is barely obtainable from the Amazon SageMaker Studio UI.

Quantization and Speculative Decoding with the SageMaker Python SDK

Beneath is the SageMaker Python SDK code snippet for quantization. quantization_config Attribute .optimize() operate.

optimized_model = builder.optimize(
    instance_type="ml.g5.12xlarge",
    accept_eula=True,
    quantization_config={
        "OverrideEnvironment": {
            "OPTION_QUANTIZE": "awq"
        }
    },
    output_path=f"s3://{output_bucket_name}/quantized/"
)

# deploy the optimized mannequin to a SageMaker endpoint
predictor = optimized_model.deploy(accept_eula=True)

# use pattern enter payload to check the deployed endpoint
predictor.predict(sample_input)

For speculative decoding, speculative_decoding_config Configure your SageMaker or {custom} mannequin by setting attributes. You would possibly want to regulate GPU utilization based mostly on the scale of each the draft and goal fashions to suit your occasion for inference.

optimized_model = builder.optimize(
    instance_type="ml.g5.12xlarge",
    accept_eula=True,
    speculative_decoding_config={
        "ModelProvider": "sagemaker"
    }
    # speculative_decoding_config={
        # "ModelProvider": "{custom}",
        # use S3 URI or HuggingFace mannequin ID for {custom} draft mannequin
        # observe: utilizing HuggingFace mannequin ID as draft mannequin requires HF_TOKEN in setting variables
        # "ModelSource": "s3://custom-bucket/draft-model", 
    # }
)

# deploy the optimized mannequin to a SageMaker endpoint
predictor = optimized_model.deploy(accept_eula=True)

# use pattern enter payload to check the deployed endpoint
predictor.predict(sample_input)

Conclusion

Optimizing generative AI fashions for inference efficiency is essential to delivering cost-effective and responsive generative AI options. With the discharge of the Inference Optimization Toolkit, now you can optimize your generative AI fashions utilizing fashionable methods equivalent to speculative decoding, compilation, and quantization to attain as much as 2X greater throughput and scale back prices by as much as 50%. This lets you obtain the perfect price-performance stability to your particular use case with just some clicks in SageMaker JumpStart or a number of strains of code utilizing the SageMaker Python SDK. The Inference Optimization Toolkit considerably simplifies the mannequin optimization course of, serving to enterprises speed up their adoption of generative AI and capitalize on extra alternatives to enhance enterprise outcomes.

For extra data, see Optimizing Mannequin Inference with Amazon SageMaker and Obtain as much as 2X Greater Throughput and Scale back Prices by as much as 50% with Generative AI Inference in Amazon SageMaker Utilizing the New Inference Optimization Toolkit – Half 1.

Concerning the Creator

James Wu Senior AI/ML Specialist Options Architect
Saurabh Trikhande Senior Product Supervisor
Rishab Ray Chowdhury Senior Product Supervisor
Kumara Swami Bora I am a front-end engineer.
Alwyn (Chiyun) Chao Senior Software program Improvement Engineer
Seiran Senior SDE