What changed

AMD has released FlashInfer on ROCm 10.1, bringing its API to AMD Instinct GPUs. This release includes new features such as 8-bit floating-point (FP8) prefill, POD-Attention, fused mixture-of-experts (MoE), and AI Tensor Engine for ROCm (AITER) decode under CUDA graph capture. These features are available on CDNA3 (gfx942: MI300X and MI325X) and CDNA4 (gfx950: MI350X and MI355X) GPUs.

What this means in practice

The FlashInfer API on ROCm provides the same wrappers and plan() and run() functions as on CUDA, allowing developers to use the same code without changes. The decode, prefill, and MLA wrappers maintain the same argument order as upstream FlashInfer v0.6.18. The default backend="auto" makes routing decisions based on AMD's testing and benchmarking on both CDNA3 and CDNA4 GPUs. This means that the API and the code calling it remain consistent, with the backend choice handled automatically.

Limitations and availability

The in-tree HIP kernels are not only a fallback for machines without AITER but also serve default operations like RMSNorm, activation, RoPE, and KV-append. These kernels are optimized for CDNA3 and CDNA4 GPUs. The RoPE kernels and activation kernels for SiLU and GELU were retuned on CDNA3 (MI300X), and the norm and prefill kernels size their shared-memory use based on the local data share (LDS) capacity of the GPU. The example provided shows how the in-tree prefill kernel uses CDNA4's transposing LDS read instruction to improve performance. The support matrix in the repository README lists every operation with its backends, per architecture.

Source facts for AMD Releases FlashInfer on ROCm 10.1 with New Features: FlashInfer on ROCm™ brings its API to AMD Instinct™ GPUs, and this release continues from the prefill-attention release. Cascade attention, multi-head latent attention (MLA), and the standalone rotary position embedding (RoPE) ops, all in progress then, have now landed. All of it runs on CDNA3 (gfx942: MI300X and MI325X)2 and CDNA4 (gfx950: MI350X and MI355X)3 GPUs.
Original AINewsTrend explainer based on the cited source. This graphic summarizes source statements; it is not an independent benchmark.

Sources and further reading

  1. amd: FlashInfer on ROCm 10.1: The FlashInfer API with Automatic Kernel Routing