On the Entropy Formula for Real, Complex, and Quaternionic Deep Linear Networks

We extend the entropy formula of Menon and Yu for the real Deep Linear Network (DLN) to its complex and quaternionic analogues, obtaining a unified formula for DLNs over $\mathbb{R}$, $\mathbb{C}$, and $\mathbb{H}$.

ML methods cs.LG math-ph math.DG

Source

Paper

References (10)

01Entropic Regularization in the Deep Linear NetworkAlan Chen, Tejas Kotwal, Govind Menon2025 · arXiv.org · 3 citations In Library

02An entropy formula for the Deep Linear NetworkGovind Menon, Tianmin Yu2025 · arXiv.org · 6 citations In Library

03The geometry of the deep linear networkGovind Menon2024 · arXiv.org · 11 citations In Library

04Deep Linear Networks for Matrix Completion - An Infinite Depth LimitNadav Cohen, Govind Menon, Zsolt Veraszto2022 · SIAM Journal on Applied Dynamical Systems · 13 citations In Library

05Motion by mean curvature and Dyson Brownian MotionChing-Peng Huang, Dominik Inauen, Govind Menon2022 · Electronic Communications in Probability · 12 citations

06Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizersB. Bah, Holger Rauhut, Ulrich Terstiege et al.2019 · Information and Inference A Journal of the IMA · 86 citations In Library

07Random MatricesM. Stephanov, J. Verbaarschot, T. Wettig2005 · 4.6k citations

08Matrix models for beta ensemblesIoana Dumitriu, A. Edelman2002 · 726 citations

09Quaternions and matrices of quaternionsFuzhen Zhang1997 · 1.3k citations

10Nacional 2508, Col. San Pedro Zacatenco, 07360 Ciudad de México, México Emailaddress

The expressivity of classical and quantum neural networks on entanglement entropyChih-Hung Wu, Ching-Che Yen2023 · The European Physical Journal C · 3 citations In Library

Efficient and Compact Representations of Deep Neural Networks via Entropy CodingGiosuè Cataldo Marinò, Flavio Furia, D. Malchiodi et al.2023 · IEEE Access · 4 citations In Library

Understanding Neural Networks with Logarithm Determinant Entropy EstimatorZhanghao Zhouyin, Ding Liu2021 · arXiv (Cornell University) · 13 citations In Library

Metric Entropy Limits on Recurrent Neural Network Learning of Linear Dynamical SystemsClemens Hutter, Recep Gül, Helmut Bölcskei2021 · arXiv (Cornell University) · 10 citations In Library

Metric entropy of causal, discrete-time LTI systemsClemens Hutter, Thomas Allard, Helmut Bölcskei2022 · arXiv (Cornell University) In Library

Approximation and Estimation for High-Dimensional Deep Learning NetworksA. Barron, Jason M. Klusowski2018 · arXiv.org · 63 citations In Library

Understanding Neural Networks withLogarithm Determinant Entropy EstimatorZhanghao Zhouyin, Ding Liu2023 · Research Square In Library

Entropy-Constrained Training of Deep Neural NetworksSimon Wiedemann, Arturo Marbán, Klaus-Robert Müller et al.2018 · IEEE International Joint Conference on Neural Network · 31 citations In Library

Deep learning in high dimension: Neural network expression rates for generalized polynomial chaos expansions in UQC. Schwab, J. Zech2018 · Analysis and Applications · 190 citations In Library

Patents near this paper

Patent US 11,263,526US 11,263,526 METHOD AND APPARATUS FOR NEURAL NETWORK QUANTIZATIONUS 11,321,609 · SAMSUNG ELECTRONICS CO., LTD. · 2022 Patent US 11,593,643US 11,593,643 CONTINUOUS RELAXATION OF QUANTIZATION FOR DISCRETIZED DEEP NEURAL NETWORKSUS 11,562,208 · QUALCOMM INCORPORATED · 2023 Patent US 11,521,060US 11,521,060

Source of record

arXiv

On the Entropy Formula for Real, Complex, and Quaternionic Deep Linear Networks

Paper

References (10)

Similar papers

Patents near this paper

Welcome.

Patents near this paper