In today’s data-driven world, industries across various sectors are grappling with the enormous influx of data being generated. The sheer volume of data poses significant challenges in terms of infrastructure costs, data transfer speeds, and storage requirements. As a result, many industries are reevaluating their approach to cloud-based storage solutions. One crucial solution to mitigate the issues associated with excessive storage usage, data integrity, transfer efficiency, and improved streaming speeds is lossless data compression. Numerous compressors currently available employ prediction-based compression techniques and utilize learned models to address these challenges effectively.To tackle this problem, our research focuses on exploring and developing strategies for compressing sequential data using neural network predictors. Neural networks are well-known as universal function approximators capable of learning complex mappings, and they have demonstrated impressive performance in prediction tasks. We leverage this capability by combining an arithmetic coder with recurrent neural network predictions, enabling us to achieve lossless compression for a wide range of datasets including synthetic, text, and genomic data [1]. Through a series of experiments using both simulated and actual text and genomic datasets, we evaluate the effectiveness of our approach. The results and observations from these experiments provide valuable insights for future research and advancements in the field of lossless data compression.
Paper
Full text
Adaptive Arithmetic Coding using Neural Network
Semantic Scholar · Computer Science · 2023
Abstract
In today’s data-driven world, industries across various sectors are grappling with the enormous influx of data being generated. The sheer volume of data poses significant challenges in terms of infrastructure costs, data transfer speeds, and storage requirements. As a result, many industries are reevaluating their approach to cloud-based storage solutions. One crucial solution to mitigate the issues associated with excessive storage usage, data integrity, transfer efficiency, and improved streaming speeds is lossless data compression. Numerous compressors currently available employ prediction-based compression techniques and utilize learned models to address these challenges effectively.To tackle this problem, our research focuses on exploring and developing strategies for compressing sequential data using neural network predictors. Neural networks are well-known as universal function approximators capable of learning complex mappings, and they have demonstrated impressive performance in prediction tasks. We leverage this capability by combining an arithmetic coder with recurrent neural network predictions, enabling us to achieve lossless compression for a wide range of datasets including synthetic, text, and genomic data [1]. Through a series of experiments using both simulated and actual text and genomic datasets, we evaluate the effectiveness of our approach. The results and observations from these experiments provide valuable insights for future research and advancements in the field of lossless data compression.