How Can We Be So Dense? The Benefits of Using Highly Sparse Representations

Paper

How Can We Be So Dense? The Benefits of Using Highly Sparse Representations

Published Mar 27, 2019 · Subutai Ahmad, Luiz Scheinkman

ArXiv

UNKNOWN SJR score

92

Citations

8

Influential Citations

Semantic Scholar

Abstract

Most artificial networks today rely on dense representations, whereas biological networks rely on sparse representations. In this paper we show how sparse representations can be more robust to noise and interference, as long as the underlying dimensionality is sufficiently high. A key intuition that we develop is that the ratio of the operable volume around a sparse vector divided by the volume of the representational space decreases exponentially with dimensionality. We then analyze computationally efficient sparse networks containing both sparse weights and activations. Simulations on MNIST and the Google Speech Command Dataset show that such networks demonstrate significantly improved robustness and stability compared to dense networks, while maintaining competitive accuracy. We discuss the potential benefits of sparsity on accuracy, noise robustness, hyperparameter tuning, learning speed, computational efficiency, and power requirements.

Highly Cited

Preprint

Study Snapshot

Key takeawaySparse representations in artificial networks can improve robustness and stability while maintaining competitive accuracy compared to dense networks.

PopulationOlder adults (50-71 years)

Sample size24

MethodsObservational

OutcomesBody Mass Index projections

ResultsSocial networks mitigate obesity in older groups.

Sign up to use Study Snapshot

Consensus is limited without an account. Create an account or sign in to get more searches and use the Study Snapshot.

Create an account

Paper

How Can We Be So Dense? The Benefits of Using Highly Sparse Representations

Sign up to use Study Snapshot

References

Sparse DNNs with Improved Adversarial Robustness

Sparse deep neural networks (DNNs) with appropriate sparsity improve their adversarial robustness, while over-sparsification can make them more vulnerable to attacks.

SNIP: Single-shot Network Pruning based on Connection Sensitivity

Our single-shot network pruning method, based on connection sensitivity, effectively reduces space and time complexity in neural networks while maintaining performance on various classification tasks.

The Sparse Manifold Transform

The Sparse Manifold Transform effectively models sparse discreteness and low-dimensional manifold structure in natural scenes, while maintaining approximate invertibility.

Sparse Unsupervised Capsules Generalize Better

Sparse unsupervised capsules networks generalize better and improve classification accuracy compared to supervised masking, potentially enabling deeper capsules networks.

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks

The lottery ticket hypothesis suggests that randomly-initialized neural networks contain subnetworks with initializations that make training effective, leading to comparable test accuracy and reduced storage requirements without compromising accuracy.

Citations

A Spatiotemporal Information-Driven Cross-Attention Model With Sparse Representation for GNSS NLOS Signal Classification

The proposed spatiotemporal information-driven cross-attention model improves GNSS NLOS signal classification accuracy by capturing environmental characteristics and signal temporal information simultaneously, achieving 98% accuracy across multiple locations.

Exploring The Neural Burden In Pruned Models: An Insight Inspired By Neuroscience

The neural burden phenomenon exists in artificial neural network models, and our method effectively mitigates the decline in model performance when pruning during training.

Understanding Inhibition Through Maximally Tense Images

Maximally tense images (MTIs) can help understand feature inhibition in vision models by simultaneously excite and inhibit a given feature, using novel visualization techniques like +/- attribution inversions and attribution atlas.

A Hyperdimensional Computing Architecture with 4.4x Energy Efficiency in 65nm CMOS

Binary sparse representations in hyperdimensional computing architectures can significantly reduce energy consumption and area in 65nm CMOS processes, improving classification accuracy and efficiency.

Hard ASH: Sparsity and the right optimizer make a continual learner

Sparse activation functions and adaptive learning rate optimizers can enhance learning retention in neural networks, as demonstrated by the Split-MNIST task.

Improving Robustness to Model Inversion Attacks via Sparse Coding Architectures

Sparse coding architectures in neural networks improve robustness against model inversion attacks, maintaining comparable or higher classification accuracy while degrading training data reconstruction quality by factors of 1.1 to 18.3.

Generalizing Projected Gradient Descent for Deep-Learning-Aided Massive MIMO Detection

The generalized PGD (GPGD) method improves massive MIMO detection performance by implementing projection after multiple gradient descent steps, and combining it with deep neural networks for self-corrected auto-detection.