AMD Instinct MI325X Ships: 288GB HBM3e Challenges NVIDIA H200 for LLM Inference

AMD has begun shipping the Instinct MI325X accelerator, featuring 288GB HBM3e memory—36% more than NVIDIA H200's 141GB. The additional memory allows serving larger LLM batches without quantization, potentially improving inference quality. AMD's ROCm 6.2 software stack now supports all major ML frameworks (PyTorch, JAX, TensorFlow) with competitive performance to CUDA. Microsoft Azure has deployed MI325X clusters for Azure AI services, and Oracle Cloud has announced MI325X availability. AMD claims MI325X achieves 92% of H200 performance at 80% of the cost for inference workloads.