Self-Supervised Learning of Representations for Space Generates Multi-Modular Grid Cells

To solve the spatial problems of mapping, localization and navigation, the mammalian lineage has developed striking spatial representations. One important spatial representation is the Nobel-prize winning grid cells: neurons that represent self-location, a local and aperiodic quantity, with seemingly bizarre non-local and spatially periodic activity patterns of a few discrete periods. Why has the mammalian lineage learnt this peculiar grid representation? Mathematical analysis suggests that this multi-periodic representation has excellent properties as an algebraic code with high capacity and intrinsic error-correction, but to date, there is no satisfactory synthesis of core principles that lead to multi-modular grid cells in deep recurrent neural networks. In this work, we begin by identifying key insights from four families of approaches to answering the grid cell question: coding theory, dynamical systems, function optimization and supervised deep learning. We then leverage our insights to propose a new approach that combines the strengths of all four approaches. Our approach is a self-supervised learning (SSL) framework - including data, data augmentations, loss functions and a network architecture - motivated from a normative perspective, without access to supervised position information or engineering of particular readout representations as needed in previous approaches. We show that multiple grid cell modules can emerge in networks trained on our SSL framework and that the networks and emergent representations generalize well outside their training distribution. This work contains insights for neuroscientists interested in the origins of grid cells as well as machine learning researchers interested in novel SSL frameworks.

Paper

Similar papers

Peer review

Reviewer 1Nr37/10 · confidence 3/52023-07-03

Summary

Shows how grid modules of grid cells can emerge as solutions to a self-supervised learning framework, implemented as a recurrent neural network. They take insights from Continuous Attractor models (velocity dependent weights), Dorrell et al (2023) representation theory (path invariance), and ideas about efficient coding (Sreenivasan & Fiete), and put them all within a single SSL framework, based on three loss functions for RNNs - maximising separation of distinct locations, path invariance and high capacity for encoding locations. The simulation results show modules of grid cells are formed.

Strengths

The single theoretical framework for explaining various aspects of the grid code is a strength

Weaknesses

The advance on previous work is not so clear, given that each aspect has been presented previously, with representation theory covered by Dorrell et al., nice analysis of the emergence of grid codes in RNNs in Sorscher et al., and coding efficiency in Sreenivasan and Fiete, and using the basline continuous attractor model of RNNs with velocity dependent weights for path integration.

Questions

Given the eventual aim to understand what is necessary for the emergence of grids in RNNs, should Sorscher et al have been cited earlier? Or the aim to understand the effect of environmental manipulations such as rewards, should Nayebi et al (NeurIPS 2021) be cited? For the eventual aim of pushing this framework towards machine learning in general domains such as vision, audition etc, what will correspond to velocity, given the reliance on a framework based on path integration?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

yes

Reviewer cRXg8/10 · confidence 3/52023-07-05

Summary

The paper proposes a computational framework for the emergence of grid cells in the mammalian cortex through self-supervised learning. The learning objective is formulated combining requirements of path independence for location code, error-correcting coding, efficient coding. Validity of the approach is demonstrated through numerical experiments on a recurrent neural network. Resulting cells reproduce important properties of grid cells observed in biology: cells organized into modules with common spatial frequency and orientation; modules exist for a range of frequencies; cells in a single module regularly tile the space.

Strengths

The paper formulates a principled and biologically meaningful optimization problem and arrives at a representation that manifests important properties of grid cells. The role of each component of the objective function is studied experimentally. Rich future directions outlined in Discussion section.

Weaknesses

Authors do not discuss relation of their work to the literature arguing for grid cells role in predictive representation (e.g., Stachenfeld et al, 2017; Momannejad 2020). The biological plausibility of proposed learning procedure is also not discussed.

Questions

Do all learning batches start from the same implied position x? Otherwise it's hard to imagine coordination of position codes between batches. Reader needs at least some elaboration on updates of W. What's a ratemap? Is it a reception field? I could not decipher figure 5a, left. line 199-200: do you mean arena is larger in testing than in training? how can the value for the low spatial frequency be found in testing?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

3 good

Contribution

4 excellent

Limitations

See weaknesses

Reviewer dHUQ6/10 · confidence 4/52023-07-05

Summary

This work reviews some of the issues with existing models of grid cells (cells in mammalian brains that fire when the animal is located at the vertices of a hexagonal grid) and suggests a new model based on recurrent neural networks (RNNs). The model is self-supervised, eliminating the worry that the structure of the readout in a supervised task could influence the outcome. Besides leading generically to grid-like firing, the model also exhibits multiple grid scales organized in modules such that the scale is the same within each module but the phase varies.

Strengths

*Originality:* The paper suggests a way for grid cells to emerge from a self-supervised learning (SSL) paradigm, in contrast to previous work which works mostly in a supervised regime. *Quality:* The paper does a good job of surveying prior work and includes a fair amount of simulations. *Clarity:* The presentation is generally clear. *Significance:* Grid cells are of tremendous interest in neuroscience and there is a considerable volume of work attempting to explain their properties and the reasons behind their existence. This paper provides a novel model for how grid cell may emerge – as a self-supervised means of keeping track of an animal's location in space – and is thus of great interest to neuroscientists.

Weaknesses

1. The authors present this work as a significant advance over methods based on supervised learning because the latter depend on specific design choices. However, the same seems to be true in the new approach: for instance, while the separation and path-invariance loss are pretty natural, the capacity loss is counter-intuitive, as mentioned even by the authors in the Discussion. Excluding the capacity loss eliminates the multi-scale nature of the solution. Moreover, the emergence of grid cells is sensitive to the parameters used in the loss, as shown in Figure 7. It is thus not immediately obvious that the proposed method requires any less fine tuning to lead to grid cells than prior models. 2. Ideally the code used to run all the simulations would have been included with the supplementary material. It was promised for after acceptance, but this seems hard to justify, since the code exists already, and it could be useful for a thorough evaluation of the paper. 3. Most of the figure panels should be significantly larger – Figure 7 is a particularly bad example. I understand that space is a limiting factor, but some progress can be made by including fewer ratemaps (don't see the need for more than 3x3 or 4x2 examples of each kind). Also ensure that font sizes don't go below 7 or 8 – when they do, it may be better to just remove the text because it is very inconvenient to read.

Questions

1. The permutation invariance discussed around eq. (2) assumes Euclidean space, where the order of the steps does not change the outcome of a path. This seems overly restrictive since a typical animal's habitat is unlikely to feature a flat 2d plane, but instead is more likely to contain obstacles, hills, tunnels, etc. Can the authors comment on how their method might adapt to such cases? 2. The motivation for the capacity loss in eq. (9) seems a bit obscure. This was touched upon briefly in the Discussion, but it would be useful for the authors to indicate how they came up with this particular formulation. Also, were other capacity loss functions attempted, and how did they work? Minor comments: * line 127: $f$ is used here with a different meaning from $f$ in, e.g., eqns. (2) or (3), which is a bit confusing * Figure 6, panel c: what non-linear dimensionality reduction technique was used?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

The authors have adequately discussed limitations of their work.

Reviewer keKQ7/10 · confidence 3/52023-07-05

Summary

The paper shows that recurrent networks trained with a "self-supervised" loss leads to units of the internal representations that organize as grid cells. In particular, the paper defines a loss that promotes separations between neural representations encoding different spatial locations, encourages a representation to be invariant to different possible paths taken leading to it's representation, and maximizing the capacity of the representation, and the authors use paired velocities and neural representations as their dataset.

Strengths

- The paper was very clearly written and organized, with nice visuals that supported the text. Moreover, the paper provided a helpful background of previous research that was relevant to the formulation used in the paper. - The paper defines a loss that is nicely linked to existing theories for the emergence of grid cells, and shows that this loss, optimized using gradient descent leads emergence of grid cells in artificial recurrent networks. - The authors performed ablations of the hyperparameters in their loss to show the dependence of their results on the different terms

Weaknesses

- How robust are the results to other hyperparameters, like batch-size, learning rate, etc? - While the authors criticize previous work that identified the emergence of grid cells using supervised RNN to specifics of the target function (line 55), the authors do not seem to properly explain how their setup differs, and does not lead to similar implicit assumptions. For example, what is the difference between the velocities being used as a supervised signal (which the authors criticize), versus incoporating implicitly into their dataset and self-supervised loss? Are there similar assumptions with respect to the creation of the dataset (e.g having a sufficient number of examples with overlapping positions) - Further, the authors assert that "SSL mitigates the need for large scale supervised data", but it is unclear to me how different it is to incorporate the velocities as a paired dataset rather than a target variable for a supervised objective. - (Minor) The claim in the discussion "how might ... SSL principles be applied to drive computational neuroscience forward" seems too general. - (Minor) Difficult to read text in Figure 7 - (Minor) extra italics on t in line 138

Questions

- Do all the internal units have grid cell properties, or only a fraction? - Related, but rather than show example units that look like grid cells, are there any metrics that quantify the extent to which a cell is a grid cell? - Are there any predictions that can be made, for example, what might happen for animals that explore 3d space? - Can the authors comment on the relationship between the toroid's and the encoding of space to make the paper self-contained (e.g. schematics from Fig. 2)?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

Yes

Reviewer dHUQ2023-08-11

Thanks for the explanation! Regarding sharing the code, a couple of points: first, I don't think my score would have been different had you included it. That does not mean the lack of the code is not a weakness. Second, had NeurIPS *required* the code to be submitted, the paper wouldn't have just been penalized for not including it, but it would have been rejected. That of course is not the case. It seems reasonable to me to be penalized for failing to do something you "are strongly encouraged" to do (to quote from the submission guidelines)—indeed, it would seem like a rather empty suggestion if it had no incentive to push authors to follow it. Finally, I sincerely doubt any work would be rejected for unintentionally sharing your identities in shared code. Most of the submissions I've reviewed have had code included and everything went fine.

Authorsrebuttal2023-08-12

Author response to reviewer dHUQ

Thank you. For future submissions, we will ensure we have code with the submission. Here, we would like to take the opportunity to reiterate the difference between a supervised and self-supervised approach: In the supervised learning approaches (e.g. Banino et al, 2018; Sorscher et al, 2019), for each time point, the loss function has access to the absolute position of the agent in the environment. It’s unrealistic to assume that biological agents have access to their absolute spatial position at all times. In the self-supervised case, for each pair of positions, the loss function has 1 coarse-grained bit of information: Whether the current position is within $\sigma_x$ of the previously considered position. This is the key difference between the 2 approaches and the fact that our self-supervised networks learn multiple modules of grids despite having access to only this impoverished version of spatial information is the significant, non-trivial advance. We have varied hyperparameters of the loss (specifically the coding scale $\sigma_g$) to show the different representations that can emerge - unlike supervised approaches that claim generality far beyond their empirical results. If there are no further scientific questions, please consider revising your score.

Reviewer dHUQ2023-08-13

Thank you for explaining the advantages of the self-supervised approach again. I will update the score from 5 to 6.

Authorsrebuttal2023-08-18

Author response

We appreciate the discussion and engagement with our work and the increase of your score from 5 to 6. We'd like to emphasize two central aspects of our work's novelty. We kindly ask you to further increase your score if these insights and perspectives resonate with you: 1) Capacity: One of the primary contributions of the work lies in the introduction of capacity as a loss term in our self-supervised learning approach. We believe that capacity is an essential factor to consider, especially when aiming to model biological neural representations where efficient representations are crucial. Capacity was considered only by grid cell theorists and by explicitly constructing our capacity loss term, we have demonstrated how to incorporate it as a broadly applicable normative principle for neuroAI. To the best of our knowledge, this is a novel concept that hasn’t been explored in previous works. Furthermore, the potential applicability of this concept extends beyond just our current work; it might be highly relevant to other modalities within machine learning and neural modeling where efficiency and capacity are of paramount importance. 2) Self-Supervised Learning for Spatial Navigation: Another significant aspect of our study is the application of self-supervised learning to spatial navigation. Spatial navigation, as a cognitive task, has its roots deeply intertwined with abstract reasoning in the brain. By applying self-supervised learning techniques to this domain, we hope to bridge the gap between embodied computational models and biological systems, potentially offering insights into how animals, including humans, make consistent models of their environment. We believe that our work is the first to offer a self-supervised approach in this domain, thereby contributing a unique perspective to this problem. We trust that our contributions, when viewed in light of the above perspective, underscore the novelty and potential impact of our research.

Reviewer dHUQ2023-08-18

Thanks for the additional explanations. I will keep the score as-is.

Authorsrebuttal2023-08-19

We have now provided anonymized code

Thank you for your reply. We have now anonymized our code and made it available at this anonymous google drive link: https://drive.google.com/drive/folders/1JNmdeTpJhktOoFJ-slC1l2AIqRSw3sAk?usp=drive_link Let us know if this code helps in your assessment of the robustness of our results. Thank you, Authors.

Reviewer cRXg2023-08-11

Thanks for detailed feedback. I have no more issues regarding biological plausibility or W updates. Ratemap: a brief definition of the term in the main text would help the uninitiated reader. On testing: My question is about the size of the arena, which is not the same thing as the trajectory length, right? Larger arena would have areas unexplored during training, and that is my confusion.

Authorsrebuttal2023-08-12

Author response to reviewer cRXg

Thank you for your specific question. Generalization to arenas larger than the training arena is a key advance of our work (relative to supervised learning approaches) and is in line with previous theoretical work on grid cell coding. The sizes of the arenas in Fig. 5 are larger than the trajectory length, as you have correctly pointed out - this is generalization by extrapolation. Grid modules provide unique positional representations upto a scale that is exponential in the number of modules - see Fiete et al (2008) [1] for theoretical arguments supporting this (also, see below for our intuitive explanation of this paper). Once a multimodular representation has been learnt on the training trajectories, the network can generalize to much larger arenas. The capacity loss provides the key top down inductive bias for this generalization capability (or else the network overfits to the training trajectories, learning a single grid scale which is roughly the size of the training trajectories, as we have shown in our ablation experiments in Fig 7a). This is also why we refer to the capacity loss as "capacity": it allows the network to represent many more locations uniquely. In light of these arguments and explanations,if there are no further scientific questions, we would request the review to please consider revising your score. We will add a paragraph better contextualizing these results to our paper along with the relevant citation. [1] What Grid Cells Convey about Rat Location. Ila Fiete, Yoram Burak and Ted Brookings, Journal of Neuroscience 2 July 2008, 28 (27) 6858-6871 To intuitively explain this result, multiple modules is akin to decimal or binary: with each additional digit or bit, the number of unique numbers grows exponentially. Grid cells are a little different for two reasons: (1) the scales of each grid module can be of the same order of magnitude, and (2) each digit/bit can be updated in parallel by path integration rather than sequentially (i.e. there’s no need to carryover when updating). Another intuitive perspective is a classic number theory argument: the chinese remainder theorem (https://en.wikipedia.org/wiki/Chinese_remainder_theorem): For example, to uniquely represent numbers up to 23, one can follow two schemes: Scheme 1: Have one box for each number, totalling 23 boxes. Scheme 2: Represent the number with remainders after dividing by 3, 5 and 7: This totals 3+5+7=15 boxes. For larger numbers, ratio of boxes between scheme 1 and scheme 2 increases exponentially. For the given example of 3, 5 and 7, one can actually represent numbers up to 3$\times$5$\times$7 = 105 uniquely. You can think of each remainder as one single grid module.

Authorsrebuttal2023-08-14

Author response

Dear Reviewer, Since the author-reviewer discussion period is coming to an end this week, we request the reviewer to consider increasing their score if we have addressed the scientific concerns raised in their review. Specifically, we have expanded on the difference between a supervised and self-supervised approach in the Global Response. Here, we reiterate the key difference: In the supervised learning approaches (e.g. Banino et al, 2018; Sorscher et al, 2019), for each time point, the loss function has access to the absolute position of the agent in the environment. It’s unrealistic to assume that biological agents have access to their absolute spatial position at all times. In the self-supervised case, for each pair of positions, the loss function has 1 coarse-grained bit of information: Whether the current position is within $\sigma_x$ of the previously considered position. This is the key difference between the 2 approaches and the fact that our self-supervised networks learn multiple modules of grids despite having access to only this impoverished version of spatial information is the significant, non-trivial advance.

Authorsrebuttal2023-08-14

Author Response

Dear Reviewer, Since the author-reviewer discussion period is coming to an end this week, we request the reviewer to consider increasing their score if we have addressed the scientific concerns raised in their review. Specifically, we have devoted our entire global response to better motivate our work and compare it to the previous approaches to grid cell emergence that the reviewer has correctly pointed out. In particular, we clearly state the limitations of each previous approach and show how we have combined insights from all these approaches to overcome every limitation.

Reviewer keKQ2023-08-18

I thank the authors for their reply. I remain concerned regarding the robustness of the results, which seems to arise from the lack of quantitative metric characterizing their grid cells; and instead only providing some example units from the different runs and ablations (without quantifying the fraction of cells with "grid-like" properties). I also suspect this may make characterizing the effect of the machine learning hyperparameters more difficult as well, once those runs complete. Given the lack of quantitative demonstration of the robustness of the results, it makes it difficult to gauge the significance of the findings.

Authorsrebuttal2023-08-19

We have now provided anonymized code

Thank you for your reply. We have now anonymized our code and made it available at this anonymous google drive link: https://drive.google.com/drive/folders/1JNmdeTpJhktOoFJ-slC1l2AIqRSw3sAk?usp=drive_link Let us know if this code helps in your assessment of the robustness of our results. Thank you, Authors.

Program Chairsdecision2023-09-21

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC