Streaming Algorithm and Hardware Accelerator Achieve 0.94 Accuracy for Top-K Flow Detection with 1.96% Error

Identifying the most frequent data streams, or top-K flows, within network traffic is crucial for optimising network performance and bolstering security, yet accurately tracking these flows presents a significant challenge given ever-increasing data speeds and volumes. Carolina Gallardo-Pavesi, Yaime Fernández, Javier E. Soto, and colleagues at Universidad de Concepción address this problem with a novel algorithm and dedicated hardware accelerator. Their approach combines a modified TowerSketch technique with a priority queue, enabling precise identification of top-K flows, up to 32,768, with over 94% precision and a remarkably low average relative error of under 2%. The team further demonstrates the power of their innovation by implementing it on a state-of-the-art FPGA, achieving a line rate exceeding 200 Gbps and processing one packet per cycle at 392MHz, representing a substantial advance in real-time network monitoring capabilities.

Hardware Acceleration of Network Flow Estimation

This research introduces a new method for efficiently identifying the most frequent network flows, known as top-k flows, in high-speed networks. The team developed a dedicated hardware accelerator based on TowerSketch, a probabilistic data structure that accurately identifies these flows while minimizing memory usage. This approach addresses the challenge of keeping pace with the increasing speeds of modern networks, where traditional methods struggle to provide timely analysis. The core of the solution is a hardware implementation of the TowerSketch algorithm, enabling efficient storage and processing of flow data.

A dedicated hardware accelerator offloads computationally intensive tasks from the central processing unit, significantly improving performance. The architecture is optimized for high throughput and minimizes latency, making it suitable for real-time network monitoring. The design was implemented on a Field-Programmable Gate Array, providing flexibility and reconfigurability. Performance evaluations, using both real and synthetic network traffic, demonstrate significant improvements in throughput and latency compared to software-based implementations. This work contributes to the field of network monitoring and security by providing a high-performance and scalable solution for analyzing network traffic.

High Precision Top-K Flow Identification Achieved

This research presents a novel algorithm for identifying the largest K flows within network traffic, a crucial task for enhancing both network efficiency and security. By combining a modified TowerSketch with a priority queue array, scientists have achieved high precision, exceeding 0. 94, in identifying these top-K flows, even when dealing with extremely large datasets containing up to 32,768 flows. Furthermore, the algorithm accurately estimates the frequency of these flows with an average relative error of under 1. 96%, demonstrating a significant improvement over existing sketch-based methods and priority queue architectures. To validate their approach, the team designed and implemented a hardware accelerator on a Virtex UltraScale+ FPGA, capable of processing one packet per cycle at 392MHz, achieving a minimum line rate exceeding 200 Gbps. This accelerator utilizes a small proportion of the device resources, suggesting it can be readily integrated into existing network processing hardware.

High Precision Top Flow Identification Achieved

Researchers developed a system capable of identifying up to 32,768 top flows with a precision exceeding 0. 94, while estimating their frequency with an average relative error under 1. 96%. This achievement stems from a combination of a modified TowerSketch data structure and a priority queue array, designed to overcome limitations of existing sketch-based algorithms when dealing with skewed network traffic distributions. The implemented TowerSketch utilizes six rows of counters with varying bit widths, ranging from 8 to 32 bits, to efficiently capture frequency information.

Incoming packets trigger updates to these counters, with the algorithm carefully managing potential overflows to maintain accuracy. The team designed the TowerSketch architecture to read counter values based on the incoming packet’s flow identifier, enabling rapid frequency estimation. To extract the top-K flows, the researchers introduced a priority queue array, a data structure that approximates a traditional priority queue but with significantly improved hardware performance. This design avoids the performance bottlenecks associated with sorting operations in conventional priority queues, especially when dealing with large values of K.

The team implemented this algorithm on a VirtexU280 UltraScale+ FPGA, achieving a processing rate of one packet per cycle at 392MHz. This translates to a minimum line rate exceeding 200 Gbps, demonstrating the accelerator’s ability to handle high-speed network traffic. The hardware architecture is optimized for parallel processing, with dedicated modules for the TowerSketch updates and priority queue insertions. The system efficiently processes packet headers to extract flow identifiers and update the data structures, enabling real-time top-K flow identification.