Summary
The authors propose a new technique for handling gradient instability (vanishing/exploding) in neural network models, especially sequence models such as RNNs. Specifically, by leveraging results from recent works that establish a connection between Lyapunov growth exponents (from dynamical systems) and the singular values of the model-Jacobian itself, the authors propose a (regularization) approach that they coin ‘gradient flossing’ to control the evolution of these singular values desirably. They demonstrate that, as a result, the gradient signal remains stable, thereby allowing for successful training of the model itself.
Additionally, the authors provide insights on the steps (schedule, pre/during training) to perform such a regularization leading to optimal gradient stabilization & computational benefits.
Strengths
The authors visit a longstanding problem in training NN models, esp. sequence models with long temporal dependencies. By leveraging recently established connections between Lyapunov exponents and the singular values of the (long-term) Jacobian, the authors propose a new technique in the form of:
- Defining a loss function out of these Lyapunov exponents
- Determine a clever strategy to evaluate this loss function through a QR decomposition
to constrain these exponents directly to stable target values. This, in turn, allows control over the evolution of the Jacobian spectra itself.
The authors provide experimental evidence supporting this technique on two specific tasks that, by design, require long-time gradient signals to propagate efficiently.
Furthermore, the authors provide practical insights into the (expensive) computational needs of the exact implementation of such a scheme and the place (pre/during training) of the same. They then propose a modified, intermittent regularization routine with preserved benefits yet having lower compute necessities.
The authors do an overall good job of satisfactorily articulating the problem statement, the proposed solution, observed benefits, and implementation limitations.
Weaknesses
While I believe this is a good contribution, I feel the work can undergo a fair amount of extra polishing in terms of clarity. Please find below (Questions) my comments/questions on areas where this work can be further improved.
Questions
- Eq. (2) does not necessarily need to refer to the ‘long-term Jacobian’ unless $\Delta{t} \equiv t - \tau \to \infty$. That should be clarified.
- Is the loss function from Eq. (4) used as an additional or the only loss to train?
- Eq. (4) does not seem to contain a reference to the target $\lambda$ values that the authors eventually aim to constrain them to
- While the analytical $t\to\infty$ limit is understandable, can you provide an estimate of a reasonable sequence length for *practical data* that can serve as a finite size equivalent of this limit?
- Is Eq. (5) subject to the assumption that parameter updates are gradient-based?
- I believe the ’s’-index in Eq. (6) is supposed to run from $\tau$ to be exact (?)
- Clearly, in arriving at Eq. (6), the $Q$’s from *subsequent* time-steps $s$ and $s+1$ act on each other to reduce to unity ($Q_{s+1}^T Q_{s}$), leaving with the $R$’s only. Can the authors explain why that might be the case in general? Is there an assumption about a small learning rate here?
- Since the authors claim the QR decomposition need not be applied at every time step, will the equivalent Eq. (6) thus arrived at still remain to be factually correct? The expression of $T_t$ contains an iterative product of $ D_s$, of which if one chooses to QR decompose a handful only, then that does not seem to yield $\prod_s R^s$ for some period $t_{ons}$.
- In Lines 140-141, you refer to using vanilla gradient descent for the experiments of Fig. 1 when you specify in Lines 132-133 that experiments were performed using ADAM.
- Isn’t a target $\lambda$ value of -1 indicative of vanishing gradients as per Eq. (3)? How is the model still reliably attaining that value regardless, then?
- In Fig. 2 (A), the ‘theory’ and ‘simulation’ legends seem to be interchanged.
- In Fig. 2 (A) & (C), the numerical and theoretical results should overlap in the long time-step regime, yet as time increases, the two appear to diverge in the former and maintain a uniform scaling (~10^2) in the latter. Can you explain why?
- Can you explain what you mean by ‘flossing before training’? That does not seem to be well clarified in the text.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
To the best of my knowledge, the authors act commendably in attributing to the various limitations of their approach.