Scientists are addressing the immense computational demands of large language models (LLMs) with a novel processing-in-memory (PIM) accelerator called PRIMAL. Yue Jiet Chong, Yimin Wang, and Zhen Wu from the National University of Singapore, together with Xuanyao Fong et al., designed PRIMAL to enhance both the speed and energy efficiency of LLM inference through low-rank adaptation (LoRA). The system integrates heterogeneous PIM processing elements (PEs) with a SRAM reprogramming and power gating (SRPG) scheme, enabling pipelined LoRA updates and dramatically reducing power consumption. By optimizing dataflow and minimizing communication overhead, PRIMAL demonstrably outperforms the Nvidia H100 GPU with LoRA rank 8 on the Llama-13B model, offering a promising pathway toward more sustainable and accessible AI applications.
PRIMAL achieves a 5× increase in throughput and a 25× improvement in energy efficiency compared to the Nvidia H100 when processing LoRA rank 8 (Q, V) on Llama-13B. The architecture features heterogeneous PEs interconnected via a 2D-mesh inter-PE computational network (IPCN), facilitating efficient parallel processing and reduced data movement. This innovative design represents a significant step forward in hardware acceleration for LLMs, addressing the growing need for faster and energy-efficient AI computation.
Chiplet Architecture and Operational Dataflow Details
At the core of PRIMAL is a chiplet-based architecture composed of multiple compute tiles (CTs). Each CT houses the 2D-mesh IPCN, which dynamically orchestrates dataflow and executes data multiply-accumulate (DMAC) operations critical for attention score computations. These are complemented by static weight multiply-accumulate (SMAC) operations performed by the PEs, which leverage both non-volatile RRAM analog compute-in-memory (RRAM-ACIM) and volatile SRAM digital compute-in-memory (SRAM-DCIM) macros. The RRAM-ACIM stores large pre-trained weights, while the SRAM-DCIM efficiently handles smaller, frequently updated LoRA matrices, creating a synergistic compute system.
A key innovation is the SRAM reprogramming and power gating (SRPG) scheme, which enables pipelined LoRA updates and achieves sub-linear power scaling by intelligently overlapping reconfiguration with computation and selectively gating idle resources. This reduces reconfiguration overhead, maximizes resource utilization, and significantly boosts both performance and energy efficiency. Spatial mapping and dataflow orchestration were meticulously optimized to further minimize communication overhead, ensuring efficient data movement between PEs and across the IPCN.
Additionally, the researchers implemented a sophisticated mapping strategy, co-locating weight matrices with their corresponding intermediate data on the PE crossbar arrays. This approach minimizes data transfer distances and maximizes computational throughput. PRIMAL operates with 64-bit width, 1 GHz frequency, 32×32 IPCN dimension, 1024 PEs per compute tile, and 256×256 RRAM-ACIM arrays paired with 256×64 SRAM-DCIM arrays. This work paves the way for deploying LLMs on edge devices and in data centers with significantly reduced power consumption and enhanced performance.
PRIMAL PIM architecture for LoRA LLM inference significantly
Scientists developed PRIMAL, a processing-in-memory (PIM) accelerator for large language model (LLM) inference leveraging low-rank adaptation (LoRA). The architecture integrates heterogeneous PIM processing elements (PEs) connected via a 2D-mesh inter-PE computational network (IPCN), enabling parallel execution of LLM layers. A novel SRAM reprogramming and power gating (SRPG) scheme overlaps LoRA updates with computation and selectively gates idle resources, achieving sub-linear power scaling and pipelined LoRA updates that significantly reduce energy consumption during model adaptation.
The team applied a hardware-software co-verification methodology, designing digital components in Verilog HDL, verifying them rigorously, and performing synthesis with Synopsys Design Compiler followed by place-and-route with Cadence Innovus. Power and area metrics for scratchpad memory macros were obtained via CACTI, while other macros were modelled in software, enabling a comprehensive evaluation. Inference was emulated using a cycle-accurate, instruction-level simulator based on the IPCN instruction set and spatial mapping scheme for precise performance analysis.
Experimental results show that PRIMAL achieves 1.5× throughput and 25× energy efficiency (9.85 tokens/J versus 0.4 tokens/J) compared to the Nvidia H100 for LoRA rank 8 (Q, V) on Llama-13B with a 2048×2048 batch. The SRPG scheme actively power-gates IPCN and RRAM-ACIM macros in idle compute tiles, while keeping SRAMs and scratchpad memory powered to preserve LoRA weights and context data for KV caching, enabling continuous operation. SRPG implementation provides up to 80% power savings versus a baseline without power gating, demonstrating scalability for larger LLMs.
Hardware analysis reveals that the RRAM-ACIM macro dominates area due to integrated analog SMAC operations, while the SRAM-DCIM macro consumes more power because of increased digital switching activity. Overall, PRIMAL establishes a highly scalable, energy-efficient architecture capable of supporting large LLMs with LoRA, leveraging CIM energy efficiency and LoRA adaptability for optimized inference performance.
