Tencent Hunyuan releases HPC-Ops: Excessive-performance LLM inference operator library

Tencent Hunyuan Open Sources HPC operationsa production-grade operator library for large-scale language mannequin inference structure gadgets. HPC-Ops focuses on low-level CUDA kernels for core operators corresponding to Consideration, Grouped GEMM, and Fused MoE, and exposes them by way of compact-C and Python APIs for integration into present inference stacks.

HPC-Ops runs on large-scale inner providers. These deployments ship roughly 30 % enchancment in queries per minute for Tencent-HY fashions and roughly 17 % enchancment for DeepSeek fashions on mainstream inference playing cards. These advantages are reported on the service stage, in order that they mirror the cumulative impact of quick kernels throughout the precise inference pipeline.

HPC-Ops scope and design

HPC-Ops is a production-grade, high-performance, easy-to-use operator library for LLM inference developed by the Tencent Hunyuan AI Infra group. This venture isn’t supposed to switch any service supply framework. As a substitute, it gives a kernel and a clear API that may be referred to as from methods that already deal with scheduling, KV cache administration, batch processing, and transport.

This API is designed for use seamlessly inside in style inference frameworks corresponding to vLLM and SGLang. This implies framework groups can substitute HPC-Ops kernels behind their very own abstractions with out altering the exterior habits of the server.

HPC-Ops makes use of C++ and CUDA, CuTe and CUTLASS as constructing blocks. The kernel is created as a comparatively small instance that additionally serves as a contemporary CUDA tutorial.

Kernel efficiency traits

This venture will publish the utmost speedup numbers noticed for every operator in comparison with established baselines. These are microbenchmarks, and the analysis group emphasizes that efficiency varies relying on geometry and workload, however represents an higher sure for optimization.

For bf16 consideration, HPC Ops stories as much as 1.33x speedup for prefill and as much as 2.22x speedup for decoding in comparison with FlashInfer, FlashAttendant 2, FlashAttendee 3, and TensorRT LLM. For fp8, we report as much as 1.12x extra consideration on prefill and as much as 2.0x extra on decoding in comparison with FlashInfer, FlashAttendee 3, and TensorRT LLM.

For FusedMoE fp8, the utmost speedup noticed is ~1.49x for prefill and ~1.14x for decoding in comparison with TensorRT LLM and vLLM. GroupGEMM fp8 stories features of as much as 1.1x in prefill and as much as 1.88x in decoding in comparison with DeepGEMM.

These numbers are necessary as a result of in autoregressive technology, decoding is often the latency bottleneck, decreasing batch dimension and dominating reminiscence visitors. The truth that Attendance and GroupGEMM present the biggest relative enchancment in decoding means that HPC-Ops focuses on the components of the pipeline that the majority customers care about.

Supported kernels and precision

Within the present launch, the performance is grouped into three operator households.

• The eye kernel covers each prefill and decoding and consists of assist for paged consideration. Web page consideration is a reminiscence structure that frameworks corresponding to vLLM use to rearrange key and worth cache blocks into web page constructions, bettering reminiscence reuse over lengthy sequences.
• The grouped GEMM is carried out as a quantized GroupGEMM with fp8 weights. HPC-Ops helps block-wise and tensor-wise scaling, permitting groups to commerce off quantization granularity with parameter storage and calibration prices.
• Fused-MoE combines skilled routing and skilled computation in a single quantized operator. It additionally makes use of fp8 skilled weights and helps block-wise and tensor-wise scaling methods.

Throughout these kernels, HPC-Ops gives native assist for bf16 and fp8 knowledge varieties. That is in line with present manufacturing tendencies of transferring inference to lower-precision codecs that protect precision whereas decreasing reminiscence bandwidth and bettering tensor core utilization.

Essential factors

• Tencent Hunyuan has open sourced HPC-Ops as a production-grade operator library for LLM inference on NVIDIA SM90 GPUs, together with H20, with C++ and CUDA kernels constructed on CuTe and CUTLASS.
• In manufacturing deployments, HPC-Ops stories QPM enhancements of roughly 30 % for Tencent-HY fashions and roughly 17 % for DeepSeek fashions on mainstream inference playing cards.
• Operator microbenchmarks present as much as 2.22x bf16 consideration decoding, as much as 2.0x fp8 consideration decoding, as much as 1.49x fp8 FusedMoE prefill, and as much as 1.88x fp8 GroupGEMM decoding in comparison with robust baselines corresponding to FlashInfer, FlashAttendant, TensorRT LLM, and DeepGEMM. A most speedup of 2x is proven.
• The library focuses on three households of operators: consideration with paged consideration assist, quantized GroupGEMM with fp8 weights, and quantized Fused MoE with fp8 skilled weights, with per-block and per-tensor scaling and assist for native bf16 plus fp8 precision.
• HPC-Ops is designed as an operator layer that’s built-in into present inference frameworks corresponding to vLLM and SGLang, and the roadmap targets sparse consideration to lengthy context LLM, enhanced quantization together with 4-bit and 8-bit methods, and kernels that enhance computational overlap with multi-GPU communication.

Please examine Click here for the report. Additionally, be at liberty to observe us Twitter Remember to hitch us 100,000+ ML subreddits and subscribe our newsletter. dangle on! Are you on telegram? You can now also participate by telegram.