VectorCDC Accelerates Hashless Data Deduplication Throughput by Up To 26.2x

Data deduplication, a crucial technique for efficient storage, relies heavily on algorithms that identify and eliminate redundant data, but these processes can be surprisingly slow and create performance bottlenecks. Sreeharsha Udayashankar, Abdelrahman Baba, and Samer Al-Kiswany, all from the University of Waterloo, address this challenge with a new method called VectorCDC, which significantly accelerates content-defined chunking, the core of many deduplication systems. The team leverages vector CPU instructions commonly found in modern processors to scan files much more quickly, achieving substantial gains in throughput. Results demonstrate that VectorCDC outperforms existing accelerated techniques by a factor of 8. 35 to 26. 2, all while maintaining the same level of storage space savings, representing a considerable advancement for data storage efficiency.

Researchers have developed a method to accelerate data chunking, improving performance without compromising deduplication ratios, by leveraging vector instructions to speed up the process and enhance efficiency. This work addresses the limitations of traditional CDC approaches when handling large datasets, which demand substantial computational resources and time.

VectorCDC accelerates hashless change data capture (CDC) algorithms by utilising vector CPU instructions, such as SSE/AVX. Evaluation demonstrates that VectorCDC achieves 8. 35 to 26. 2 times higher throughput than existing vector-accelerated techniques without affecting deduplication space savings. The method exploits single instruction, multiple data (SIMD) capabilities to process multiple data elements concurrently, significantly boosting performance and benefiting content-defined chunking, a technique used in deduplication systems to identify and eliminate redundant data.

Faster, Scalable, Secure Data Deduplication Techniques

Research in data storage increasingly focuses on data deduplication, the process of eliminating redundant copies of data to save storage space. This is a central theme, with efforts directed towards making deduplication faster, more scalable, more secure, and more efficient in various environments.

A significant portion of the research focuses on how to break down data into chunks for deduplication, optimising for speed and accuracy. Content-defined Chunking (CDC) is a dominant theme, with papers exploring algorithms that leverage data locality. Similarity-based chunking algorithms consider the similarity of data when creating chunks, while bimodal chunking combines different chunking approaches. Parallelism and scalability are also crucial, with researchers utilising multithreading and designing distributed systems to handle large datasets and high workloads.

Efficient indexing is vital for quickly finding duplicate chunks, leading to exploration of various hash table implementations optimised for performance and memory usage. Sparse indexing techniques index only the unique chunks. Delta compression, which stores only the differences between files, is often combined with deduplication, with approaches like LoopDelta and stream-informed delta compression further optimising the process. Hardware acceleration, utilising instructions like AVX-512 CD for faster comparison, also plays a key role.

Research extends to leveraging non-volatile memory (NVM) to improve deduplication performance and designing file systems specifically for deduplication, such as MinervaFS. Addressing performance optimisation, handling dynamic data, and ensuring security and privacy are constant challenges. Researchers are exploring secure deduplication schemes, including encrypted deduplication, and addressing vulnerabilities like side-channel attacks. Optimising deduplication for Docker images, backup and archival systems, and cloud storage are also key areas of focus, alongside applications in big data analytics and utilising I/O patterns to improve performance.

Overall trends indicate a focus on practical implementations, combining multiple techniques, prioritising security, and demanding scalability to handle ever-growing data volumes.

VectorCDC Accelerates Hashless Data Deduplication Significantly

VectorCDC introduces a new methodology for accelerating content-defined chunking, a critical process in data deduplication systems. By leveraging vector instructions available on modern CPUs, the technique significantly improves throughput compared to existing approaches, achieving speed increases of 8. 35 to 26. 2 times faster than current vector-accelerated techniques. This acceleration is achieved through novel tree-based search and packed scanning methods applied to hashless content-defined chunking algorithms.

The research demonstrates that VectorCDC is effective across a range of CPU architectures, including Intel, ARM, and IBM processors, and is compatible with existing encryption schemes used in data storage. The team has publicly released their code integrated with DedupBench and made a dataset available on Kaggle, promoting reproducibility and further research. Future work may extend the technique to handle even larger datasets and explore its application in various storage systems.