Summary
The paper proposes a new model from the Dense Associative Memory family that uses Sparsemax function for its energy. This model is studied and compared to the model with the softmax-based energy from both theoretical and empirical perspectives. The work proposes that the sparse model outperforms the model with the softmax (dense model) in terms of the memory retrieval bound.
Strengths
The work derives a novel model from the modern HN family and analyses its energy and dynamics using convex conjugate of the sparse entropic regularizer. As far as I can tell this is a sophisticated result, which meaningfully extends the family of models that has been previously studied.
The work studies theoretically several properties of the convergence dynamics in this new model. In general, results pertaining to continues models from this family are scarce, which makes this submission even more valuable.
The authors also discuss possible ways of integrating their model into existing Hopfield-like frameworks, and propose Sparse Hopfield, SparseHopfieldPooling, SparseHopfieldLayer layers based on their energy function.
Empirical results look promising.
The proofs in the appendices look convincing, although I have not checked them carefully.
Weaknesses
It would be nice if the authors could summarize in a concise and crisp way what is the theoretical advantage of their model with the Sparsemax compared to previously studied modern HN models. I can see that the new model uses a very different language for its formulation (which is great), but I am struggling to understand its computational benefits compared to previously studied models. The improvements in empirical performance, which the authors present are great, but they are not too significant to claim superiority just based on them. I would appreciate a clear theoretical proposition here.
The statement in lines 578-579 is somewhat confusing. The model studied in Ramsauer 2020 uses softmax activation function, the model studied in Demircigil uses exponential activation function. Several papers (incorrectly) state that these two models are identical. This is wrong, since they have mathematically distinct energy functions and update equations.
Some wording in Appendix C might be somewhat confusing. The appendix makes it sound that everyone before Ramsauer 2020 studied binary Hopfield networks, but Ramsauer 2020 introduced the continuous networks. This is not quite correct. For instance https://www.pnas.org/doi/10.1073/pnas.81.10.3088 introduced continues sparse (as opposed to dense) Hopfield networks in 1984. Krotov & Hopfield 2016 introduced continuous dense Hopfield networks (see equation 10 in their paper and most of the empirical results on MNIST), etc. Ramsauer 2020 focused on studying a specific model from that family (with softmax activation) and calculated the capacity of that specific model. But the continuous networks (both sparse and dense) were introduced in prior work.
Questions
1. Could the authors please explain the computational benefits of their model compared to previously studied dense associative memories?
2. I am not too familiar with Gini entropy and its conjugate, but formulas for the relationship between $F(\mathbf{p})$ and $\Psi(\mathbf{p})$ (e.g. above equation 2.4) look very similar to Legendre transformation to me. Krotov & Hopfield 2020 also use Legendge transformation to compute the energy function using Lagrangians. Is there a precise relationship between the Lagrangian language of Krotov and Hopfield and the formalism developed in this submission?
3. The authors present quantitative metrics to empirically evaluate the performance of their model. This is great, but in order to get more intuition about the new model it would be helpful to show a few images for experiments presented in section 4.1 that the authors used in their experiments. For example pairs of initial states and final retrieved images. It would be interesting to see visually what kinds of mistakes the new network makes compared to previously studied models.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Limitations
This is a theoretical work. The authors address societal impacts in the last section “Broader Impact” and appendix A.