Bayesian Optimized Continual Learning with Attention Mechanism

Paper

Bayesian Optimized Continual Learning with Attention Mechanism

Published May 10, 2019 · Ju Xu, Jin Ma, Zhanxing Zhu

ArXiv

UNKNOWN SJR score

6

Citations

0

Influential Citations

Semantic Scholar

Abstract

Though neural networks have achieved much progress in various applications, it is still highly challenging for them to learn from a continuous stream of tasks without forgetting. Continual learning, a new learning paradigm, aims to solve this issue. In this work, we propose a new model for continual learning, called Bayesian Optimized Continual Learning with Attention Mechanism (BOCL) that dynamically expands the network capacity upon the arrival of new tasks by Bayesian optimization and selectively utilizes previous knowledge (e.g. feature maps of previous tasks) via attention mechanism. Our experiments on variants of MNIST and CIFAR-100 demonstrate that our methods outperform the state-of-the-art in preventing catastrophic forgetting and fitting new tasks better.

Preprint

Study Snapshot

Key takeawayBayesian Optimized Continual Learning with Attention Mechanism (BOCL) effectively prevents catastrophic forgetting and adapts better to new tasks in neural networks.

PopulationOlder adults (50-71 years)

Sample size24

MethodsObservational

OutcomesBody Mass Index projections

ResultsSocial networks mitigate obesity in older groups.

Sign up to use Study Snapshot

Consensus is limited without an account. Create an account or sign in to get more searches and use the Study Snapshot.

Create an account

Paper

Bayesian Optimized Continual Learning with Attention Mechanism

Sign up to use Study Snapshot

References

Progress & Compress: A scalable framework for continual learning

Our progress & compress framework accelerates learning progress in continual learning domains, preserving performance on previously encountered tasks while accelerating progress on subsequent problems.

Reinforced Continual Learning

Reinforced Continual Learning improves AI models' ability to solve new tasks faster and prevent catastrophic forgetting, outperforming existing alternatives for deep networks.

Overcoming catastrophic forgetting with hard attention to the task

The proposed task-based hard attention mechanism effectively reduces catastrophic forgetting in neural networks by 45 to 80%, offering robustness and monitoring capabilities for online learning and network compression applications.

FearNet: Brain-Inspired Model for Incremental Learning

FearNet is a memory-efficient model for incremental class learning, achieving state-of-the-art performance on image and audio classification benchmarks without storing previous examples.

Lifelong Learning with Dynamically Expandable Networks

Dynamically Expandable Network (DEN) significantly outperforms existing lifelong learning methods for deep networks, achieving the same performance with fewer parameters and achieving optimal network structure even when all tasks are available.

Gradient Episodic Memory for Continual Learning

Gradient Episodic Memory (GEM) effectively reduces forgetting and allows beneficial knowledge transfer across tasks in continual learning models, outperforming state-of-the-art methods on MNIST and CIFAR-100 datasets.

Overcoming Catastrophic Forgetting by Incremental Moment Matching

Incremental moment matching (IMM) effectively overcomes catastrophic forgetting in neural networks by balancing information between an old and a new network, achieving state-of-the-art performance on various datasets.

Continual Learning Through Synaptic Intelligence

Intelligent synapses in artificial neural networks significantly reduce forgetting and improve continual learning in classification tasks while maintaining computational efficiency.

Citations

Similarity-Based Adaptation for Task-Aware and Task-Free Continual Learning

This paper introduces a continuous learning framework that adapts to the task at hand by focusing on relevant pieces of previous knowledge, improving performance on various benchmarks.

Adaptive attention principal component analysis with continual learning ability for multimode process monitoring

Adaptive attention principal component analysis with continual learning capability effectively monitors multimode nonlinear nonstationary processes, consolidating learned knowledge and assimilating new significant features.

Continual Learning with Sparse Progressive Neural Networks

Sparse progressive neural networks effectively integrate prior knowledge into new skills, preventing catastrophic forgetting and reducing the number of learned task-specific parameters on novel tasks.

Continual Learning with Adaptive Weights (CLAW)

CLAW, a continual learning approach based on probabilistic modeling and variational inference, achieves state-of-the-art performance on six benchmarks in classification accuracy and addressing catastrophic forgetting.

Selective Feature Learning with Filtering Out Noisy Objects in Background Images

The Single Shot multibox Detection (SSD) algorithm effectively eliminates irrelevant objects in target images, improving performance in image classification tasks.