> How does the proof of universal in-context learning of linear RNNs guide the practical development of these models? While I am not opposed to theoretical papers, I'm not convinced that this proof will significantly benefit the development of the field. As a researcher who leans more towards empirical work, I cannot recommend acceptance for this paper unless it clearly guides practice or explains some successes or flaws in existing empirical phenomena. However, I would not oppose others in the theory community who may find this work valuable. I believe my perspective is typical among empirical researchers; theory should either guide empirical study or explain existing empirical phenomena.
Our paper has the following contributions that are directly applicable to empirical research:
1. **LSRL is a tool with practical applications in itself.** We develop LSRL, a new programming language that is isomorphic to a large class of recurrent models. Using this language and the compiler we have developed, one can directly program behaviours that can be then incorporated into RNNs, LSTMs, GRUs and Hawk/Griffin models. Due to this isomorphism, conversely, given a model with these architectures, we can “decompile” it as an LSRL program. This would be quite useful for people doing interpretability, explainability and safety research for models with these architectures.
2. **We explain the successes of architectures with multiplicative gating.** Recurrent architectures with multiplicative gating such as Mamba and Hawk have outperformed architectures without it. With our experiments, we show that multiplicative gating is much more numerically stable when one wants the model to follow an algorithm in-context precisely, which is how a lot of the evaluation and benchmarks are set up. We also show that multiplicative gating enables a much more compact implementation of concrete algorithms, and hence can explain why smaller models with multiplicative gates are sometimes comparable in performance to larger models that do not have them. Therefore, our work **explains one aspect that can drive the success** of gating architectures and **guides the practice to use multiplicative gating**.
We believe that beyond serving as a basis for engineering and applied research, the foundation of science is a desire to understand the world around us for the joy of discovery and knowledge. The development of machine learning is not restricted to building new models, but also includes understanding, characterizing and organizing the properties of the methods, tools and mathematical models we already have. We are positive that a large part of the NeurIPS audience feels the same way, hence why “Theory” is one of the areas explicitly mentioned in this year’s Call for Papers. In this way, our work fills key gaps in the characterization of the universal approximation properties of neural network models, an area with a long history, while also serving as a guide for design decisions for recurrent architectures and explaining some successes of multiplicative gating.
> Regarding gated RNNs, gating the recurrence remains linear if no activation is involved in the middle of the recurrence. For example, current linear models like Griffin, HGRN, Mamba, GLA, and RWKV6 all incorporate such gating in the recurrence and can leverage either parallel scan [1] or chunkwise form [2] for parallel training. Gated linear recurrence should definitely reference these linear RNN models with data-dependent forget gates or decays. The definition used in this work seems questionable to me.
Perhaps we are using two different definitions of “linear recurrence”. We mean it in the sense of a “linear time-invariant system” (LTI). As such, models like Mamba and Griffin would not satisfy this definition. They would be something like a “linear time-variant system” (LTV). However, as every time-invariant system is a trivial time-variant system, the class of LTV systems is a proper superset of the class of LTI systems. Therefore, if there is an instance of an LTI system with a certain property (in our case, universal in-context approximation), then it immediately follows that there is an instance for an LTV system with that property.
In our work we show that the more restricted setting already is a universal in-context approximator, automatically making the more general setting you are referring to a universal in-context approximator as well. We show this connection explicitly and formally in Appendix D, where we show that the Hawk/Griffin architecture is also a universal in-context approximator. Hence, the definition we use is not limiting our results in any way whatsoever, it is just the most general (or, if one wants, _abstract_) definition that serves the goal of our work.