The Analysis of Deep Neural Networks by Information Theory: From Explainability to Generalization

Despite their great success in many artificial intelligence tasks, deep neural networks (DNNs) still suffer from a few limitations, such as poor generalization behavior for out-of-distribution (OOD) data and the "black-box" nature. Information theory offers fresh insights to solve these challenges. In this short paper, we briefly review the recent developments in this area, and highlight our contributions.

Neural Networks and Applications Anomaly Detection Techniques and Applications Computational Physics and Python Applications Computer Science

Paper

Full text

The Analysis of Deep Neural Networks by Information Theory: From Explainability to Generalization

OpenAlex · Neural Networks and Applications · 2023

Abstract

Despite their great success in many artificial intelligence tasks, deep neural networks (DNNs) still suffer from a few limitations, such as poor generalization behavior for out-of-distribution (OOD) data and the "black-box" nature. Information theory offers fresh insights to solve these challenges. In this short paper, we briefly review the recent developments in this area, and highlight our contributions.

Source of record · OpenAlex

References (8)

01Principle of Relevant Information for Graph SparsificationShujian Yu, F. Alesiani, Wenzhe Yin et al.2022 · Conference on Uncertainty in Artificial Intelligence · 17 citations In Library

02Optimal Randomized Approximations for Matrix-Based Rényi’s EntropyYuxin Dong, Tieliang Gong, Shujian Yu et al.2022 · IEEE Transactions on Information Theory · 12 citations In Library

03Measuring Dependence with Matrix-based Entropy FunctionalShujian Yu, F. Alesiani, Xi Yu et al.2021 · AAAI Conference on Artificial Intelligence · 40 citations In Library

04Mutual Information Estimation using LSH SamplingDeng Pan, Xiangrui Li, X. Li et al.2020 · International Joint Conference on Artificial Intelligence · 37 citations

05Understanding Autoencoders with Information Theoretic ConceptsShujian Yu, J. Príncipe2018 · Neural Networks · 149 citations In Library

06Mutual Information Neural EstimationMohamed Ishmael Belghazi, A. Baratin, Sai Rajeswar et al.2018 · International Conference on Machine Learning · 1.7k citations In Library

07Opening the Black Box of Deep Neural Networks via InformationRavid Shwartz-Ziv, Naftali Tishby2017 · arXiv.org · 1.6k citations In Library

08Multivariate Extension of Matrix-Based Rényi's $\alpha$α-Order Entropy FunctionalShujian Yu, L. S. Giraldo, R. Jenssen et al.2020 · IEEE Transactions on Pattern Analysis and Machine Intelligence · 80 citations

Similar papers

Theoretical Insights into Neural Networks and Deep Learning: Advancing Understanding, Interpretability, and GeneralizationUsman Ahmad Usmani, Mohammed Umar Usmani2023 In Library

Information Flow in Deep Neural NetworksRavid Shwartz-Ziv2022 · arXiv (Cornell University) · 25 citations In Library

Opening the Black Box of Deep Neural Networks via InformationRavid Shwartz-Ziv, Naftali Tishby2017 · arXiv.org · 1.6k citations In Library

Information-Theoretic Generalization Bounds for Deep Neural NetworksHaiyun He, Ziv Goldfeld2025 · IEEE Transactions on Information Theory In Library

Information-Theoretic Generalization Bounds for Deep Neural NetworksHaiyun He, Ziv Goldfeld2024 · arXiv (Cornell University) · 15 citations In Library

Deep Networks as Logical Circuits: Generalization and InterpretationChristopher Snyder, Sriram Vishwanath2020 · arXiv (Cornell University) In Library

Examining the Causal Structures of Deep Neural Networks Using Information TheoryScythia Marrow, Eric J. Michaud, Erik Hoel2020 · Entropy In Library

Technical Report: Quantifying and Analyzing the Generalization Power of a DNNYuxuan He, Junpeng Zhang, Lei Cheng et al.2025 · arXiv (Cornell University) In Library

On information captured by neural networks: connections with memorization and generalizationHrayr Harutyunyan2023 · arXiv (Cornell University) · 0 citations In Library

Going Deeper, Generalizing Better: An Information-Theoretic View for Deep LearningJingwei Zhang, Tongliang Liu, Dacheng Tao2023 · IEEE Transactions on Neural Networks and Learning Systems · 10 citations In Library

Source of record

OpenAlex

44B

Welcome.

This panel sticks with you. Pick anything below and it opens right beside it, so you can dig through 60,000-plus records without ever losing your spot here.

Travel 44B

LibraryPapers, policy, standards, statute LabsEvery organization building AI ModelsIntelligence, price, and speed BenchmarksThe evaluation catalog IncidentsAI failures and harms SearchOne field across everything DashboardThe state of AI accountability in NY 44B RegistryThe Art. 44-B compliance portal