TherML: Thermodynamics of Machine Learning

In this work we offer a framework for reasoning about a wide class of existing objectives in machine learning. We develop a formal correspondence between this work and thermodynamics and discuss its implications.

cs.LG cond-mat.stat-mech stat.ML Computer Science Physics Mathematics

Source

Paper

References (39)

01Representation Learning with Contrastive Predictive CodingAäron van den Oord, Yazhe Li, O. Vinyals2018 · arXiv.org · 14k citations In Library

02Thermodynamics from InformationM. N. Bera, A. Winter, M. Lewenstein2018 · 424 citations

03Mathematical Theory of Bayesian StatisticsSumio Watanabe2018 · 99 citations

04Energy–entropy competition and the effectiveness of stochastic gradient descent in machine learningYao Zhang, Andrew M. Saxe, Madhu S. Advani et al.2018 · Molecular Physics · 62 citations In Library

05MINE: Mutual Information Neural EstimationIshmael Belghazi, Sai Rajeswar, A. Baratin et al.2018 · arXiv.org · 749 citations

06Fixing a Broken ELBOAlexander A. Alemi, Ben Poole, Ian S. Fischer et al.2017 · International Conference on Machine Learning · 555 citations

07Stochastic Gradient Descent Performs Variational Inference, Converges to Limit Cycles for Deep NetworksP. Chaudhari, Stefano Soatto2017 · Information Theory and Applications Workshop · 341 citations In Library

08A Bayesian Perspective on Generalization and Stochastic Gradient DescentSamuel L. Smith, Quoc V. Le2017 · International Conference on Learning Representations · 310 citations In Library

09Thermodynamic cost and benefit of data representationsSusanne Still2017 · 15 citations

10Stochastic Gradient Descent as Approximate Bayesian InferenceStephan Mandt, M. Hoffman, D. Blei2017 · Journal of machine learning research · 766 citations In Library

11Snapshot Ensembles: Train 1, get M for freeGao Huang, Yixuan Li, Geoff Pleiss et al.2017 · International Conference on Learning Representations · 1.1k citations In Library

12Variational Lossy AutoencoderXi Chen, Diederik P. Kingma, Tim Salimans et al.2016 · International Conference on Learning Representations · 714 citations In Library

Scroll for more · 27 remaining

Similar papers

On the Temperature of Machine Learning SystemsDong Zhang2024 · arXiv (Cornell University) · 2 citations In Library

Understanding Machine Learning Paradigms through the Lens of Statistical Thermodynamics: A tutorialStar, Wanquan Liu2024 · arXiv (Cornell University) · 0 citations In Library

Thermodynamic Machine Learning through Maximum Work ProductionA. B. Boyd, J. Crutchfield, M. Gu2020 · New Journal of Physics · 20 citations In Library

Machine Learning for Harnessing Thermal Energy: From Materials Discovery to System OptimizationMan Li, Lingyun Dai, Yongjie Hu2022 · ACS Energy Letters · 68 citations In Library

Thermodynamics of Reinforcement Learning CurriculaJacob Adamczyk, J. Rojas, Rahul V. Kulkarni2026 · arXiv.org · 0 citations In Library

Machine learning for molecular thermodynamicsJiaqi Ding, Nan Xu, Manh Tien Nguyen et al.2021 · Chinese Journal of Chemical Engineering · 40 citations In Library

Status, Challenges, and Potential for Machine Learning in Understanding and Applying Heat Transfer PhenomenaMatthew L. Hughes, G. Kini, S. Garimella2021 · Journal of heat transfer · 86 citations In Library

Learning Thermodynamics with Boltzmann MachinesG. Torlai, R. Melko2016 · arXiv.org · 211 citations In Library

Machine learning to identify variables in thermodynamically small systemsD. Ford, Aditya Dendukuri, Gulce Kalyoncu et al.2020 · Computers and Chemical Engineering · 4 citations In Library

Thermodynamic Prediction Enabled by Automatic Dataset Building and Machine LearningJuejing Liu, Haydn Anderson, Noah I. Waxman et al.2025 · arXiv.org · 1 citation In Library

Patents near this paper

MACHINE LEARNING OPTIMIZATION FRAMEWORKUS 11,853,017 · INTERNATIONAL BUSINESS MACHINES CORPORATION · 2023 Combining Machine Learning With Domain Knowledge And First Principles For Modeling In The Process IndustriesUS 11,853,032 · ASPEN TECHNOLOGY, INC. · 2023 MACHINE LEARNING DEVICE, CONTROL SYSTEM, AND MACHINE LEARNING METHODUS 11,680,916 · FANUC CORPORATION · 2023 MACHINE LEARNING DEVICE FOR MACHINE TOOL AND THERMAL DISPLACEMENT COMPENSATION DEVICEUS 10,401,829 · FANUC CORPORATION · 2019 MACHINE LEARNING MODEL INTERPRETATIONUS 11,645,541 · ADOBE SYSTEMS INCORPORATED · 2023

Source of record

arXiv