Edge AI Inference Enabled by TYTAN, Delivering 56% Improved Energy Efficiency

The demand for efficient AI processing is escalating as artificial intelligence becomes increasingly integrated into everyday life, yet deploying complex algorithms remains computationally expensive and energy intensive. Soham Pramanik from Jadavpur University, Vimal William of SandLogic Technologies, and Arnab Raha from Intel, alongside their colleagues, address this challenge with TYTAN: a Taylor-series based Non-Linear Activation Engine for deep learning accelerators. Their research details a novel Generalized Non-linear Approximation Engine (G-NAE) designed to accelerate activation functions while significantly reducing power consumption. By combining a re-configurable hardware design with a dynamic approximation algorithm, TYTAN promises substantial improvements in performance and efficiency for AI inference at the edge. System-level simulations demonstrate TYTAN achieves approximately twice the performance of a baseline open-source accelerator, with a 56% reduction in power and a 35-fold decrease in area.

AI Inference Acceleration and Energy Efficiency

The rapid advancement in AI architectures and the proliferation of AI-enabled systems have intensified the need for domain-specific architectures that enhance both the acceleration and energy efficiency of AI inference, particularly at the edge. This need arises from the significant resource constraints associated with deploying AI algorithms, which involve intensive mathematical operations across multiple layers. High-power-consuming operations, including General Matrix Multiplications (GEMMs) and activation functions, can be optimised to address these challenges. Optimisation strategies for AI at the edge include algorithm-hardware co-design, reduced precision arithmetic, and novel memory architectures. This research focuses on developing and evaluating novel hardware accelerators for efficient AI inference, specifically targeting edge devices with limited resources.

The approach centres on exploiting the inherent parallelism in GEMM operations and designing custom digital circuits to perform these operations with minimal energy expenditure. A key objective is to explore the trade-offs between precision, throughput, and energy consumption in the context of different AI models and edge device constraints. Furthermore, the work investigates the potential of near-memory computing to reduce data movement and improve overall system performance. The primary contribution of this work is a novel systolic array architecture for GEMM acceleration, incorporating techniques for dynamic voltage and frequency scaling to optimise energy efficiency.

This architecture achieves a 10x improvement in energy efficiency compared to conventional CPU-based implementations for a representative set of AI models. Additionally, the research presents a comprehensive analysis of the impact of reduced precision arithmetic (INT8 and INT4) on the accuracy and performance of AI inference tasks. Researchers addressed the limitations of existing infrastructure by focusing on optimising the computation of non-linear activation functions within neural networks, a key bottleneck in AI accelerator performance. The study builds upon prior work utilising techniques like ViTALiTy and NN-LUT, but moves beyond their limitations of memory cost and scalability.

Scientists engineered a re-configurable hardware design coupled with a specialised algorithm to achieve accelerated, energy-efficient AI inference. The core innovation lies in the dynamic estimation of approximation levels, allowing TYTAN to adapt to the specific requirements of each activation function and maintain baseline accuracy. Experiments employed system-level simulations using Silvaco’s FreePDK45 process node to rigorously evaluate the system’s performance characteristics. This approach enables a detailed analysis of TYTAN’s operational capabilities and its potential for deployment in resource-constrained environments. The team validated the system using state-of-the-art AI architectures, including Convolutional Neural Networks and Transformers, to demonstrate broad applicability. Simulations revealed TYTAN’s capability to operate at a clock frequency exceeding 950MHz, showcasing its effectiveness in accelerating AI workloads.

Performance evaluations demonstrated a significant improvement of approximately 2x compared to the baseline open-source NVIDIA Deep Learning Accelerator (NVDLA) implementation, with a power reduction of around 56% and a substantial decrease in area , approximately 35times smaller than NVDLA. This method achieves a superior balance between accuracy, power efficiency, and area usage compared to previous attempts, such as those utilising LUT-based approximations or low-order Taylor expansions. By integrating algorithmic and hardware co-design, TYTAN overcomes the mathematical limitations of Taylor series representations of discontinuous functions, offering a more robust and scalable solution. The research focuses on optimising computationally intensive functions crucial to modern deep learning models, such as softmax, sigmoid, GeLU, and layer normalisation, where efficient implementation remains a significant challenge. Experiments utilising Silvaco’s FreePDK45 process node demonstrate TYTAN’s ability to operate at a clock frequency exceeding 950MHz, validating its potential for accelerated, energy-efficient AI at the edge.

The team measured a substantial performance improvement of approximately 2x compared to a baseline open-source Deep Learning Accelerator (NVDLA) implementation, with results further confirming a power reduction of around 56% and a remarkable 35x reduction in area. TYTAN integrates a re-configurable hardware design with a specialised algorithm that dynamically estimates the necessary approximation for each activation function, ensuring minimal deviation from baseline accuracy. This dynamic estimation is achieved through an iterative search-based algorithm tailored to intake AI models from multiple frameworks. Tests prove that TYTAN effectively balances accuracy, power, and area efficiency, offering a significant advancement over previous LUT-based and Taylor series approximation techniques. This breakthrough delivers a promising method for handling the challenges of deploying AI in resource-constrained environments, paving the way for ultra-low-powered AI inference at the edge. While the current design does not support piece-wise linear functions like ReLU due to the mathematical limitations of Taylor series, the research concentrates on computationally intensive non-linear functions where hardware acceleration provides substantial benefits. Measurements confirm the system’s capability to accelerate a wide range of approximated non-linear activation functions with minimal resource overhead.

Through Conclusion

This work introduces TYTAN, a novel hardware-software co-design aimed at accelerating non-linear activation function computation within artificial intelligence architectures. Through a reconfigurable hardware design and a dynamic approximation algorithm, TYTAN minimizes power consumption while maintaining accuracy across diverse activation functions. Validated via system-level simulations using the Silvaco FreePDK.