Tiny Transformers on FPGAs Enable Low-Power Time-Series Analysis.

Researchers developed an automated framework deploying compact, integer-only Transformer models on embedded Field-Programmable Gate Arrays (FPGAs) for time-series tasks. Quantization-aware training and hardware-aware optimisation achieved low-power inference – as little as 0.033 mJ per inference – with millisecond latency on Spartan-7 FPGAs.

The analysis of sequential data, commonly known as time-series analysis, underpins applications ranging from financial forecasting to predictive maintenance and environmental monitoring. Deploying complex analytical models on low-power, embedded systems presents a significant engineering challenge, particularly given the computational demands of contemporary machine learning. Researchers at the University of Duisburg-Essen – Tianheng Ling, Chao Qian, Lukas Johannes Haßler, and Gregor Schiele – address this issue in their paper, ‘Automating Versatile Time-Series Analysis with Tiny Transformers on Embedded FPGAs’. They present a fully automated framework for deploying compact Transformer models – a type of neural network – onto Field-Programmable Gate Arrays (FPGAs), achieving low-energy inference across a range of time-series tasks.

Enabling Compact Transformers on Resource-Constrained FPGAs

The deployment of computationally intensive Transformer models onto embedded systems with limited resources remains a significant challenge. This work presents a fully automated framework that successfully implements compact Transformer models on Field Programmable Gate Arrays (FPGAs) with constrained resources. The research achieves this through a combination of quantization-aware training and automated hardware mapping, resulting in integer-only, task-specific accelerators that demonstrate low energy consumption and millisecond-level latency on Spartan-7 FPGAs. This framework streamlines the adaptation of Transformer models to resource-constrained environments, facilitating wider adoption in edge computing applications.

The increasing demand for on-device intelligence necessitates efficient hardware implementations of deep learning models. Transformer networks, while excelling in tasks such as natural language processing and computer vision, demand substantial computational resources. Traditional deployment strategies often rely on high-density FPGAs or Application-Specific Integrated Circuits (ASICs), which can be expensive and power-hungry. This research addresses these limitations by developing a comprehensive automation flow that optimises Transformer models for deployment on low-cost, low-power FPGAs, enabling a new generation of intelligent edge devices. The framework automates the entire process, encompassing quantization-aware training, hardware-aware hyperparameter optimisation using Optuna, and automatic VHDL generation, significantly reducing design effort and time-to-market.

The core of the framework lies in its ability to effectively quantize Transformer models without substantial accuracy degradation. Quantization reduces the numerical precision of model parameters and activations – typically from 32-bit floating-point to 8-bit integer or lower – thereby reducing memory footprint and computational complexity. However, aggressive quantization can lead to accuracy loss, necessitating careful calibration and optimisation. The framework employs quantization-aware training, which simulates the effects of quantization during the training process, allowing the model to adapt and maintain accuracy even with reduced precision.

The framework leverages Optuna, an automated hyperparameter optimisation software framework, to fine-tune the quantization process and identify optimal configurations for the target FPGA. Optuna systematically explores the design space, searching for the best combination of quantization parameters to maximise accuracy while minimising resource utilisation. This automated optimisation eliminates the need for manual tuning, which is time-consuming and requires significant expertise.

Automatic VHDL generation forms a critical component of the framework, streamlining the hardware implementation process and reducing the need for manual coding. VHDL, or Very High Speed Integrated Circuit Hardware Description Language, is a standard language used to describe the logic of digital circuits that will be implemented on the FPGA. The framework automatically translates the optimised quantized model into a VHDL description, which can then be synthesised, placed, and routed – the steps required to create a physical implementation on the FPGA.

The relationship between bit-width and performance is crucial for practical implementation. While higher precision generally yields better accuracy, it also increases hardware resource utilisation and energy consumption. Conversely, aggressive quantization to 4-bit representation significantly reduces resource requirements but can lead to substantial accuracy degradation. The framework allows users to explore this design space and identify optimal configurations based on their specific application requirements.

Experimental results demonstrate the framework’s effectiveness across a variety of datasets and FPGA platforms. The framework’s ability to effectively balance accuracy and energy consumption makes it well-suited for a wide range of edge computing applications.

This automated framework represents an advance over existing FPGA-based Transformer deployments, which often rely on manual configuration and high-density platforms. Manual configuration is time-consuming and requires significant expertise in both machine learning and FPGA design. Furthermore, high-density FPGAs are often expensive and consume more power, limiting their suitability for edge devices with limited resources.

Future work will focus on exploring more advanced quantization techniques, such as mixed-precision quantization – where different layers of the network use different bit-widths – and dynamic quantization – where the quantization range adapts during inference. The framework will also be extended to support a wider range of FPGA platforms and deep learning architectures. Additionally, the research will investigate the use of hardware-aware Neural Architecture Search to automatically design Transformer models that are optimised for deployment on resource-constrained FPGAs.