Visual interpretability for deep learning: a survey

doi:10.1631/FITEE.1700808

Paper

Visual interpretability for deep learning: a survey

Published Jan 1, 2018 · Quanshi Zhang, Song-Chun Zhu

Frontiers of Information Technology & Electronic Engineering

Q2 SJR score

760

Citations

17

Influential Citations

Full text Semantic Scholar

Abstract

This paper reviews recent studies in understanding neural-network representations and learning neural networks with interpretable/disentangled middle-layer representations. Although deep neural networks have exhibited superior performance in various tasks, interpretability is always Achilles’ heel of deep neural networks. At present, deep neural networks obtain high discrimination power at the cost of a low interpretability of their black-box representations. We believe that high model interpretability may help people break several bottlenecks of deep learning, e.g., learning from a few annotations, learning via human–computer communications at the semantic level, and semantically debugging network representations. We focus on convolutional neural networks (CNNs), and revisit the visualization of CNN representations, methods of diagnosing representations of pre-trained CNNs, approaches for disentangling pre-trained CNN representations, learning of CNNs with disentangled representations, and middle-to-end learning based on model interpretability. Finally, we discuss prospective trends in explainable artificial intelligence.

Highly Cited

Study Snapshot

Key takeawayHigh model interpretability in deep learning can break bottlenecks and improve the understanding of neural-network representations, potentially leading to explainable artificial intelligence.

PopulationOlder adults (50-71 years)

Sample size24

MethodsObservational

OutcomesBody Mass Index projections

ResultsSocial networks mitigate obesity in older groups.

Paper

Visual interpretability for deep learning: a survey

References

Interpreting CNNs via Decision Trees

Decision trees effectively explain the rationales of each CNN prediction at the semantic level, providing valuable insights beyond traditional pixel-level analysis.

Towards Interpretable R-CNN by Unfolding Latent Structures

Unfolding latent structures in object detection using R-CNN improves model interpretability without affecting performance on the PASCAL VOC 2007 and 2012 datasets.

Interpretable R-CNN

The proposed method, using an AOGParsing operator, effectively combines top-down hierarchical and compositional grammar models with bottom-up deep neural networks for object detection, outperforming state-of-the-art methods on the PASCAL VOC 2007 and 2012 datasets.

Dynamic Routing Between Capsules

A multi-layer capsule system achieves state-of-the-art performance on MNIST and is significantly better than convolutional nets at recognizing highly overlapping digits using an iterative routing-by-agreement mechanism.

One Pixel Attack for Fooling Deep Neural Networks

Our one-pixel attack using differential evolution can fool 67.97% of natural images in Kaggle CIFAR-10 and 16.04% of ImageNet datasets, demonstrating the vulnerability of deep neural networks to low-dimensional attacks.

Citations

XKT: Toward Explainable Knowledge Tracing Model With Cognitive Learning Theories for Questions of Multiple Knowledge Concepts

The XKT model, based on cognitive learning theories and multidimensional item response theory, provides high precision and interpretability in predicting student exercise performance in educational applications.

Universal scaling laws in quantum-probabilistic machine learning by tensor network towards interpreting representation and generalization powers

Universal scaling laws in quantum-probabilistic machine learning can help interpret representation and generalization powers of tensor networks, potentially leading to white-box ML schemes within the quantum probabilistic framework.

A New Semi-Iterative Well-Test Matching Method Based on Featured Points for Homogeneous Reservoir

The semi-iterative model (SIM) is an accurate and efficient method for matching well-test curves in homogeneous reservoirs, reducing manual fitting time by 90% or more.

Concept-Based Explanations in Computer Vision: Where Are We and Where Could We Go?

Concept-based XAI approaches in computer vision offer intuitive explanations and can be improved by considering concept representation, control techniques, and knowledge representation and learning.

Benchmarking Protein Language Models for Protein Crystallization

LightGBM classifiers using ESM2 model embeddings outperform other methods in predicting protein crystallization outcomes, identifying 5 potentially crystallizable proteins.

Enhancing Sports Injury Risk Assessment in Soccer Through Machine Learning and Training Load Analysis.

Machine learning algorithms can effectively assess non-contact muscle injury risk in soccer players, highlighting the importance of physiological overload as a contributor to injuries.