AMD has entered direct competition with Nvidia in the high-performance AI accelerator market with the new Instinct MI350P PCIe card, boasting 144GB of HBM3E memory. Designed as a drop-in upgrade for existing servers, the MI350P delivers roughly 40% faster FP16 and FP8 theoretical compute performance compared to Nvidia’s H200 NVL. The card achieves this with 128 compute units and a peak theoretical performance of 2,299-4,600 TFLOPs using MXFP4, while operating within a 600W power envelope, configurable down to 450W for broader system compatibility. AMD states the MI350P is the fastest AI accelerator card on the market that fits in a traditional PCIe slot, targeting inference and retrieval-augmented generation pipelines.
MI350P: CDNA4 Architecture and 3nm/6nm FinFET Process
The AMD Instinct MI350P accelerator features 144GB of HBM3E memory, indicating a commitment to handling the immense data demands of modern artificial intelligence applications. This high-bandwidth memory, delivering 4TB/s, enables more complex models and larger datasets to reside directly on the card, minimizing latency and maximizing throughput. Constructed utilizing TSMC’s advanced 3nm and 6nm FinFET processes, the MI350P integrates 128 Compute Units and has a 128MB last-level cache. The card’s specifications are comparable to half of what AMD’s high-end MI350X and MI355X AI GPUs offer. This performance improvement is particularly notable given the increasing reliance on reduced-precision formats like FP8 for accelerating large language models and other AI workloads.
The card’s architecture, based on AMD’s CDNA4, also incorporates native support for lower-precision MXFP6 and MXFP4 formats, optimizing performance for these demanding tasks. Industry analysis highlights the significance of the MI350P as a competitor to Nvidia’s fastest PCIe AI accelerator, the H200 NVL. A key element of the MI350P’s design is its power efficiency and flexibility. While boasting considerable performance, the card operates within a 600W power envelope, a figure that is manageable within typical server infrastructure. AMD has engineered the MI350P to be configurable down to 450W, broadening its compatibility with thermally or power-constrained systems.
This adaptability is a practical consideration for data centers seeking to integrate the accelerator without requiring extensive infrastructure upgrades. The card’s physical form factor, a 10.5″ dual-slot design with a fanless cooling solution, further simplifies integration, relying on existing chassis fans for thermal management. The MI350P’s capabilities extend to scalability, with support for pairing up to eight cards within a single system, allowing for performance increases proportional to the number of installed accelerators. AMD asserts the GPU is capable of an estimated 2,299 TFLOPs and 4,600 peak TFLOPs of performance using MXFP4, positioning it as a powerful solution for a range of AI applications, from inference to retrieval-augmented generation (RAG) pipelines. Nvidia has not announced a PCIe version of its latest B200 Blackwell GPUs running HBM memory, so AMD currently has the most advanced AI accelerator that fits in a PCIe form factor. It remains to be seen how widely adopted AMD’s new card will be, given Nvidia’s hold on the market with CUDA, but AMD is working to improve its competing ROCm software stack.
144GB HBM3E Memory & 4TB/s Bandwidth Specifications
The demand for ever-increasing memory capacity and bandwidth within accelerator cards continues to escalate, driven by the expanding complexity of artificial intelligence models and data-intensive computing tasks. Current high-performance systems typically rely on configurations featuring substantial, though varied, amounts of high bandwidth memory; however, AMD’s recent MI350P introduces a new benchmark with its specification of 144GB of HBM3E memory. This substantial memory pool isn’t simply about size; the choice of HBM3E technology itself signifies a commitment to maximizing data throughput, a critical factor in accelerating AI workloads. HBM3E stacks memory dies vertically, enabling a significantly wider interface than traditional memory configurations and, consequently, a much higher bandwidth. The MI350P delivers an impressive 4TB/s of memory bandwidth.
This figure is particularly noteworthy when considering the architecture’s ability to move data between the processing units and memory at an unprecedented rate, directly impacting the speed at which complex calculations can be performed. This level of bandwidth is essential for handling the massive datasets used in training and deploying large language models and other advanced AI applications. The specifications are comparable to those of AMD’s high-end MI350X and MI355X AI GPUs. Beyond raw speed, AMD has engineered the MI350P with flexibility in mind, allowing data centers to deploy the MI350P without requiring extensive modifications to their power and cooling systems. This is a practical consideration often overlooked in the pursuit of peak performance, and it demonstrates AMD’s attention to real-world deployment challenges.
The GPU is paired to 144GB of HBM3E memory with 4TB/s of bandwidth, and a 128MB last-level cache, further optimizing data access and reducing latency. The introduction of lower-precision MXFP6 and MXFP4 formats further enhances the MI350P’s capabilities, accelerating large language models. Up to eight MI350P cards can be paired together in a single system, allowing data centers to scale performance based on how many cards are used.
FP8/FP16/FP64 Performance: MI350P vs. H200 NVL
Researchers at TSMC are currently focused on optimizing fabrication processes for advanced chip designs, a necessity highlighted by AMD’s recent unveiling of the Instinct MI350P accelerator card. This new card, built on TSMC’s 3nm and 6nm FinFET processes, immediately positions itself as a direct competitor to Nvidia’s H200 NVL in the rapidly evolving artificial intelligence hardware market. Beyond simply entering the competition, AMD is challenging established performance benchmarks with a card boasting 144GB of HBM3E memory, a substantial capacity designed to handle the demands of increasingly complex AI models. The MI350P is designed to be a drop-in upgrade solution for existing air-cooled servers and comes with 128 Compute Units, paired with a 128MB last-level cache. However, the most striking performance claims center around its floating-point precision capabilities.
This allows for accelerated processing of large language models, a key area of development in the field. AMD states that the MI350P is geared towards small, medium, and large AI workloads surrounding inference and RAG pipelines, signaling a broad target market beyond just large-scale training. Notably, AMD has engineered the MI350P with power efficiency in mind. This flexibility is particularly important given the thermal and power constraints often found in existing data center deployments. AMD claims the GPU has an estimated 2,299 TFLOPs and 4,600 peak TFLOPs of performance using MXFP4. The MI350P features 20% better FP64, 40% better FP16, and 39% better FP8 theoretical compute performance.
Scalable Multi-GPU Systems & Inference Workload Focus
The demand for accelerated computing is rapidly shifting, with a growing emphasis on deploying artificial intelligence models for real-world applications rather than solely focusing on training. AMD’s introduction of the Instinct MI350P addresses this evolving need by prioritizing scalability and efficient inference, a critical step where trained AI models process new data. Unlike some high-end GPUs geared towards massive training runs, the MI350P is designed to function effectively within the constraints of standard data center infrastructure, supporting up to eight cards in a single system to increase throughput for demanding workloads. This capability is particularly relevant for resource-intensive tasks like retrieval-augmented generation (RAG) pipelines, where AI systems must quickly access and process vast amounts of information. Central to the MI350P’s design is its substantial memory capacity; the card features 144GB of HBM3E memory, enabling it to handle larger models and datasets without performance bottlenecks.
This high-bandwidth memory, delivering 4TB/s, is crucial for inference tasks where rapid data access is paramount. Performance benchmarks reveal a direct challenge to established competitors. The MI350P offers an estimated 2,299 TFLOPs and 4,600 peak TFLOPs of performance using MXFP4. The MI350P is geared towards small, medium, and large AI workloads surrounding inference and RAG pipelines. The MI350P features 20% better FP64, 40% better FP16, and 39% better FP8 theoretical compute performance compared to the H200 NVL. The card’s specifications relate to the MI350X and MI355X, but are not exactly half of those GPUs. The MI350P offers native support for lower-precision MXFP6 and MXFP4. This isn’t merely an incremental improvement; it signifies a substantial leap in processing speed for key AI calculations. The MI350P distinguishes itself through its power efficiency and operational flexibility. This adaptability is a significant advantage, as many existing data centers lack the infrastructure to support extremely high-power GPUs.
