Summary
The paper proposes to use transformers to learn to optimise. The proposed architecture relies on the performer architecture and leverage attention to encode long-term dependencies. The authors discuss two variants, a coordinate-wise and tensor-wise approach, to account for the dependencies between the parameters to optimise. They also propose to use a hierarchal pooling strategy to bypass the linear attention of performers. The authors compare their method to classical optimizers, such as ADAM, and LSTM-based optimisers. The paper also provides a theoretical analysis of the compact associative memory used by the method.
Strengths
The authors propose a transformer-based optimiser by combining a topological and a temporal encorder. The resulting transformer is outperforming LSTM-based counterparts on the experiments conducted by they authors. The method is also competitive with widely used optimisers such as ADAM. The overall results are encouraging and the reported results cover both NLP and CV applications.
Weaknesses
While the authors propose a transformer-based architecture to learn an optimiser, their choices are mainly driven by practical considerations. The authors do not motivate their choices beyond these and do not discuss trade-offs and/or other possible choices. The methodological contributions of this work are modest beyond combining existing components and the use of a hierarchical pooling strategy. While I have no doubt that making these components work well together is non trivial, the challenges to make them work together are not discussed. Hence, it appears that the paper does not do justice to work that has been done in the background. More importantly, the lack of details would make it difficult for other researchers to reproduce the results.
The authors provide a theoretical analysis of the compact associative memory used, but the importance and implications of this result is not discussed.
While the experiments cover the NLP and CV domain, reported results were underwhelming. The authors conducted first small scale experiments where they compared to a wide range of off-the shelve optimizers. However, in the NLP and CV domain, they only optimised a subset of the parameters with the proposed approach, cast doubts on the broad applicability, its scaling properties and its robustness.
Questions
I would like to ask the authors the following questions:
1/ Can you provide a summary of the meta-training procedure and a discussion of the choices made? How robust is Mnemosyne to the specific procedure used and the set-up?
2/ Can you explain why long-term dependencies are important when learning an optimizers? Can you provide evidence that the proposed architecture is able to capture these and leverage them?
3/ Can the authors articulate the importance of the theoretical analysis?
4/ Why did you consider the VeLo architecture in the experiments? Did you consider other options?
5/ Can you provide details of the configuration of Mnemosyne in the experiments? How many of these did you try out and how did the results vary?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
I would have liked the authors to be more vocally self-critical about the pros and cons of their proposed method. The paper reads like the approach is a sliver bullet, which obviously is not the case. I would have liked to see a discussion of the strengths and weaknesses of the method, an analysis of the robustness, an empirical validation of the claims (like importance of long-term dependencies), etc.