Speed up NLP inference with ONNX runtime on AWS Graviton processors

ONNX is an open supply machine studying (ML) framework that gives interoperability throughout a variety of frameworks, working techniques, and {hardware} platforms. ONNX runtime Runtime engine used for mannequin inference and coaching with ONNX.

AWS Graviton3 processors are optimized for ML workloads, together with help for bfloat16, Scalable Vector Extension (SVE), and Matrix Multiplication (MMLA) directions. ONNX’s Bfloat16-accelerated SGEMM kernel and int8 MMLA-accelerated quantized GEMM (QGEMM) kernel enhance inference efficiency for some pure language processing (NLP) fashions on AWS Graviton3 by as much as 65% for fp32 inference and as much as 65% for int8 quantized inference 30% enchancment. A base Amazon Elastic Compute Cloud (Amazon EC2) occasion.beginning model v1.17.0ONNX runtime helps these optimized kernels.

This submit reveals the right way to run ONNX runtime inference on an AWS Graviton3-based EC2 occasion and the right way to configure the occasion to make use of an optimized GEMM kernel. We additionally reveal the ensuing speedup by way of benchmarks.

Optimized GEMM kernel

The ONNX runtime helps the Microsoft Linear Algebra Subroutines (MLAS) backend because the default execution supplier (EP) for deep studying operators. AWS Graviton3-based EC2 cases (c7g, m7g, r7g, c7gn, and Hpc7g cases) help bfloat16 format and MMLA directions for deep studying operator acceleration. These directions enhance SIMD {hardware} utilization and scale back end-to-end inference latency by as much as 1.65x in comparison with instruction-based kernels in armv8 DOT merchandise.

The AWS group applied an MLAS kernel for bfloat16 quick operations and int8 quantized normal matrix multiplication (GEMM) utilizing the BFMMLA, SMMLA, and UMMLA directions. These directions have increased throughput for matrix multiplication in comparison with DOT directions. Assist for bfloat16 means that you can effectively deploy fashions skilled utilizing bfloat16, fp32, and computerized combined precision (AMP) with out the necessity for quantization. The optimized his GEMM kernel is built-in into the ONNX runtime CPU EP as a MLAS kernel, as proven within the following determine.

The primary diagram reveals the ONNX software program stack, highlighting parts (in orange) which are optimized for improved inference efficiency on the AWS Graviton3 platform.

The next diagram reveals the ONNX runtime EP circulate, highlighting parts (in orange) which are optimized for improved inference efficiency on the AWS Graviton3 platform.

Allow optimization

Optimization is a part of ONNX runtime 1.17.0 has been launched and is on the market since onnxruntime-1.17.0 Python wheel and conda-1.17.0 package. The optimized int8 kernel is enabled by default and robotically chosen for AWS Graviton3 processors. However, the Bfloat16 quick math kernel shouldn’t be enabled by default, so to allow it he wants the next session possibility within the ONNX runtime:

# For C++ functions

SessionOptions so; 
so.config_options.AddConfigEntry( kOrtSessionOptionsMlasGemmFastMathArm64Bfloat16, "1");

# For Python functions

sess_options = onnxruntime.SessionOptions()
sess_options.add_session_config_entry("mlas.enable_gemm_fastmath_arm64_bfloat16", "1")

Benchmark outcomes

First, we measured the inference throughput (queries per second) of the unoptimized fp32 mannequin (utilizing ONNX runtime 1.16.0). That is marked as 1.0 with a pink dotted line within the following graph. We then in contrast the enhancements within the bfloat16 quick math kernel in ONNX runtime 1.17.1 for a similar fp32 mannequin inference. The normalized outcomes are plotted on a graph. We see throughput enhancements of as much as 65% for the BERT, RoBERTa, and GPT2 fashions. Related enhancements may be seen in inference latency.

Much like the fp32 inference comparability graph above, we began by measuring the inference throughput in queries per second for the unoptimized int8 quantized mannequin (utilizing ONNX runtime 1.16.0). That is marked 1.0 with a pink dot. That is the road within the following graph. We then in contrast the enhancements of the optimized MMLA kernel in ONNX runtime 1.17.1 for a similar mannequin inference. The normalized outcomes are plotted on a graph. We see throughput enhancements of as much as 30% for the BERT, RoBERTa, and GPT2 fashions. Related enhancements may be seen in inference latency.

Benchmark setup

We used an AWS Graviton3-based c7g.4xl EC2 occasion and an Ubuntu 22.04-based AMI to reveal the efficiency beneficial properties from the ONNX runtime’s optimized GEMM kernel. Extra details about cases and AMIs is supplied within the following snippet.

Occasion: c7g.4xl occasion
Area: us-west-2
AMI: ami-0a24e6e101933d294 (Ubuntu 22.04/Jammy with 6.5.0-1014-aws kernel)

The ONNX runtime repository supplies inference benchmark scripts for transformer-based language fashions. Scripts help a variety of fashions, frameworks, and codecs. We chosen PyTorch-based BERT, RoBERTa, and GPT fashions to cowl widespread language duties reminiscent of textual content classification, sentiment evaluation, and masked phrase prediction. This mannequin covers each encoder and decoder transformer architectures.

The next code demonstrates the steps to carry out inference on an fp32 mannequin in bfloat16 quick arithmetic mode and int8 quantization mode utilizing the ONNX runtime benchmark script. This script downloads the mannequin, exports it to ONNX format, quantizes it to int8 for int8 inference, and runs the inference with varied sequence lengths and batch sizes. If the script completes efficiently, it can print the inference throughput (QPS) in queries per second (QPS) and latency (ms) together with the system configuration. Please check with. ONNX Runtime Benchmark Script For extra info.

# Set up Python
sudo apt-get replace
sudo apt-get set up -y python3 python3-pip

# Improve pip3 to the most recent model
python3 -m pip set up --upgrade pip

# Set up onnx and onnx runtime
# NOTE: We used 1.17.1 as a substitute of 1.17.0 because it was the most recent
# model accessible whereas amassing information for this submit
python3 -m pip set up onnx==1.15.0 onnxruntime==1.17.1

# Set up the dependencies
python3 -m pip set up transformers==4.38.1 torch==2.2.1 psutil==5.9.8

# Clone onnxruntime repo to get the benchmarking scripts
git clone --recursive https://github.com/microsoft/onnxruntime.git
cd onnxruntime
git checkout 430a086f22684ad0020819dc3e7712f36fe9f016
cd onnxruntime/python/instruments/transformers

# To run bert-large fp32 inference with bfloat16 quick math mode
python3 benchmark.py -m bert-large-uncased -p fp32 --enable_arm64_bfloat16_fastmath_mlas_gemm

# To run bert-base  fp32 inference with bfloat16 quick math mode
python3 benchmark.py -m bert-base-cased -p fp32 --enable_arm64_bfloat16_fastmath_mlas_gemm

# To run roberta-base  fp32 inference with bfloat16 quick math mode
python3 benchmark.py -m roberta-base -p fp32 --enable_arm64_bfloat16_fastmath_mlas_gemm

# To run gpt2  fp32 inference with bfloat16 quick math mode
python3 benchmark.py -m gpt2 -p fp32 --enable_arm64_bfloat16_fastmath_mlas_gemm

# To run bert-large int8 quantized inference
python3 benchmark.py -m bert-large-uncased -p int8

# To run bert-base int8 quantized inference
python3 benchmark.py -m bert-base-cased -p int8

# To run roberta-base int8 quantized inference
python3 benchmark.py -m roberta-base -p int8

# To run gpt2 int8 quantized inference
python3 benchmark.py -m gpt2 -p int8

conclusion

On this submit, you realized the right way to run ONNX runtime inference on an AWS Graviton3-based EC2 occasion and the right way to configure the occasion to make use of an optimized GEMM kernel. We additionally demonstrated that it resulted in quicker speeds. Please strive.

When you discover a use case the place you do not observe related efficiency enhancements with AWS Graviton, please open a difficulty with the AWS Graviton Technical Information. GitHub to tell us about it.

Concerning the writer

Sunita Nadanpalli I am a software program improvement supervisor at AWS. She leads her Graviton software program’s efficiency optimization for machine studying and her HPC workloads. She is obsessed with open supply software program improvement and delivering high-performance, sustainable software program options utilizing Arm SoCs.