The Information Bottleneck Problem and Its Applications in Machine Learning

Inference capabilities of machine learning (ML) systems skyrocketed in recent years, now playing a pivotal role in various aspect of society. The goal in statistical learning is to use data to obtain simple algorithms for predicting a random variable <inline-formula> <tex-math notation="LaTeX">$Y$ </tex-math></inline-formula> from a correlated observation <inline-formula> <tex-math notation="LaTeX">$X$ </tex-math></inline-formula>. Since the dimension of <inline-formula> <tex-math notation="LaTeX">$X$ </tex-math></inline-formula> is typically huge, computationally feasible solutions should summarize it into a lower-dimensional feature vector <inline-formula> <tex-math notation="LaTeX">$T$ </tex-math></inline-formula>, from which <inline-formula> <tex-math notation="LaTeX">$Y$ </tex-math></inline-formula> is predicted. The algorithm will successfully make the prediction if <inline-formula> <tex-math notation="LaTeX">$T$ </tex-math></inline-formula> is a good proxy of <inline-formula> <tex-math notation="LaTeX">$Y$ </tex-math></inline-formula>, despite the said dimensionality-reduction. A myriad of ML algorithms (mostly employing deep learning (DL)) for finding such representations <inline-formula> <tex-math notation="LaTeX">$T$ </tex-math></inline-formula> based on real-world data are now available. While these methods are effective in practice, their success is hindered by the lack of a comprehensive theory to explain it. The information bottleneck (IB) theory recently emerged as a bold information-theoretic paradigm for analyzing DL systems. Adopting mutual information as the figure of merit, it suggests that the best representation <inline-formula> <tex-math notation="LaTeX">$T$ </tex-math></inline-formula> should be maximally informative about <inline-formula> <tex-math notation="LaTeX">$Y$ </tex-math></inline-formula> while minimizing the mutual information with <inline-formula> <tex-math notation="LaTeX">$X$ </tex-math></inline-formula>. In this tutorial we survey the information-theoretic origins of this abstract principle, and its recent impact on DL. For the latter, we cover implications of the IB problem on DL theory, as well as practical algorithms inspired by it. Our goal is to provide a unified and cohesive description. A clear view of current knowledge is important for further leveraging IB and other information-theoretic ideas to study DL models.