Bayesian Attention Networks for Data Compression

Paper

Bayesian Attention Networks for Data Compression

Published Mar 29, 2021 · Michael Tetelman

ArXiv

UNKNOWN SJR score

0

Citations

0

Influential Citations

Semantic Scholar

Abstract

The lossless data compression algorithm based on Bayesian Attention Networks is derived from first principles. Bayesian Attention Networks are defined by introducing an attention factor per a training sample loss as a function of two sample inputs, from training sample and prediction sample. By using a sharpened Jensen's inequality we show that the attention factor is completely defined by a correlation function of the two samples w.r.t. the model weights. Due to the attention factor the solution for a prediction sample is mostly defined by a few training samples that are correlated with the prediction sample. Finding a specific solution per prediction sample couples together the training and the prediction. To make the approach practical we introduce a latent space to map each prediction sample to a latent space and learn all possible solutions as a function of the latent space along with learning attention as a function of the latent space and a training sample. The latent space plays a role of the context representation with a prediction sample defining a context and a learned context dependent solution used for the prediction.

Preprint

Study Snapshot

Key takeawayBayesian Attention Networks for data compression can effectively compress large data sets without sacrificing quality, with a latent space playing a crucial role in context representation and solution learning.

PopulationOlder adults (50-71 years)

Sample size24

MethodsObservational

OutcomesBody Mass Index projections

ResultsSocial networks mitigate obesity in older groups.

Sign up to use Study Snapshot

Consensus is limited without an account. Create an account or sign in to get more searches and use the Study Snapshot.

Create an account

Paper

Bayesian Attention Networks for Data Compression

Sign up to use Study Snapshot

References

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Pure transformers applied directly to image patches can perform well on image classification tasks, outperforming state-of-the-art convolutional networks while requiring fewer computational resources to train.

On Compression Principle and Bayesian Optimization for Neural Networks

The paper proposes a compression principle for neural networks, which optimizes predictions by minimizing total message length while maintaining decodability, and demonstrates its effectiveness using Bayesian Stochastic Gradient Descent (BSGD).

Bit-Swap: Recursive Bits-Back Coding for Lossless Compression with Hierarchical Latent Variables

Bit-Swap is a new compression scheme that achieves better lossless compression rates for hierarchical latent variable models with Markov chain structure compared to existing techniques.

Attention is All you Need

The Transformer, based solely on attention mechanisms, achieves superior quality in machine translation tasks while being more parallelizable and requiring less time to train.

Data Compression: The Complete Reference

Data Compression provides a comprehensive reference for various compression methods, covering run length encoding, statistical methods, dictionary-based methods, image compression, audio compression, and video compression.

Citations