A note on the unbiased estimation of mutual information

Estimators for mutual information are typically biased. However, in the case of the Kozachenko-Leonenko estimator for metric spaces, a type of nearest neighbour estimator, it is possible to calculate the bias explicitly.

Bayesian Methods and Mixture Models Statistical Methods and Inference Advanced Statistical Methods and Models

View graph Source

Paper

Similar papers

Analysis of k-Nearest Neighbor Distances with Application to Entropy EstimationShashank Singh, B. Póczos2016 · arXiv.org · 39 citations In Library

Demystifying Fixed k-Nearest Neighbor Information EstimatorsWeihao Gao, Sewoong Oh, P. Viswanath2016 · International Symposium on Information Theory · 150 citations In Library

A Bayesian Nonparametric Estimation of Mutual InformationLuai Al‐Labadi, Forough Fazeli-Asl, Zahra Saberi2021 · arXiv (Cornell University) In Library

A test for independence via Bayesian nonparametric estimation of mutual informationLuai Al‐Labadi, Forough Fazeli Asl, Zahra Saberi2021 · Canadian Journal of Statistics In Library

The Nearest Neighbor Information Estimator is Adaptively Near Minimax Rate-OptimalJiantao Jiao, Weihao Gao, Yanjun Han2017 · Neural Information Processing Systems · 48 citations In Library

Bias Reduction and Metric Learning for Nearest-Neighbor Estimation of Kullback-Leibler DivergenceYung-Kyun Noh, Masashi Sugiyama, Song Liu et al.2018 · Neural Computation · 34 citations In Library

Beyond Normal: On the Evaluation of Mutual Information EstimatorsPaweł Czyż, Frederic Grabowski, Julia E. Vogt et al.2023 · arXiv (Cornell University) · 65 citations In Library

Improving Numerical Stability of Normalized Mutual Information Estimator on High DimensionsMarko Tuononen, Ville Hautamäki2025 · IEEE Signal Processing Letters In Library

Efficient Estimation of Mutual Information for Strongly Dependent VariablesShuyang Gao, G. V. Steeg, A. Galstyan2014 · International Conference on Artificial Intelligence and Statistics · 219 citations In Library

Estimating Mutual Information by Local Gaussian ApproximationShuyang Gao, G. V. Steeg, A. Galstyan2015 · Conference on Uncertainty in Artificial Intelligence · 32 citations In Library

Source of record

OpenAlex

44B

Welcome.

This panel sticks with you. Pick anything below and it opens right beside it, so you can dig through 60,000-plus records without ever losing your spot here.

Travel 44B

LibraryPapers, policy, standards, statute LabsEvery organization building AI ModelsIntelligence, price, and speed BenchmarksThe evaluation catalog IncidentsAI failures and harms SearchOne field across everything DashboardThe state of AI accountability in NY 44B RegistryThe Art. 44-B compliance portal