TopoNets: High Performing Vision and Language Models with Brain-Like Topography

Response 1 to Reviewer FUA2

> Modeling cortical topography is an important line of research for both neuroscience and AI… **We thank the reviewer for their constructive feedback and for recognizing the challenges that previous topographic models have faced in maintaining task performance.** Your comments and insights help underscore the importance of addressing this balance, and we hope you will find our results a meaningful progress in this direction. We would like to begin addressing some of the questions raised and share our planned analyses and changes. We will upload an updated text in the next few days and a summary of all the changes. Please let us know if you'd like any other questions/specific changes. Thank you for your engaging with us! **Q1. Are TDANNs and TopoNets directly comparable?** **Response:** This is a very important point. We would first like to clarify the key differences between TDANNs and TopoNets which highlights the challenges in making direct comparisons. 1. TDANNs were trained on SimCLR and Toponets on Imagenet categorization. The model's training regimen will of course change the model's overall performance 2. TDANN models induce topography on the outputs of every ResNet sub-block. In TopoNets we induced topography in *every* convolutional layer. We will update the text to highlight the differences in the model training and layers on which to induce topography. **Given these differences, how do we compare (benchmark) models?** **Option 1**: We take a leaf out of **integrative benchmarking** (from BrainScore, Schrimpf et al., 2020). That is, we compare models on engineering measures (Imagenet) and neural measures (V1 and VTC) irrespective on what they were trained on. *We agree* with the reviewer that the comparison between TDANN and TopoNets does not seem fair by this measure (though note that LLCNN and TopoNets can be compared). However, this a somewhat accepted measure in vision neuroscience endorsed even by authors of the TDANN study (senior authors were contributors in the original BrainScore paper, Schrimpf, Kubilius 2018). **Option 2:** We report the **difference in model-performance** from baseline (non-topographic versions). For TDANN, the reported difference in categorization performance between the baseline model (trained on SimCLR) and TDANN-SimCLR is 5%. The difference between our baseline model (Imagenet trained) and TopoNet-Imagenet is 7% (for model with all convolutional layers topographic). We also now trained a model with topography on the same layers as Margalit et al (for an even more fair comparison) and the difference in model performance is 3%. We will now report this as well for yet another perspective on task performance. **We will update the paper to highlight the challenges in comparing models and these new results** That said, we would very much like the comparison between TDANN and TopoNets on task performance *not* to get in the way of the core contributions. Which is that TopoNets signify **a fundamentally new method for training topographic neural network models that scale well across model architectures and domains (vision and language)** and is strongly supported by all our current (and new results). **Q2. Figure 6 - I would prefer seeing the same plots replicated for the baseline vision and language models—(a) without being trained jointly on TopoLoss, (b) unoptimized (control) on both the task and topographic losses.** **Response:** We are happy to provide these. We do not observe clustering of units by category selectivity (vision models) or temporal integration windows (language) in baseline models or unoptimized (control models). We will update the figures to highlight these changes. **Changes to the paper:** Updated plots with baseline and unoptimized models alongside the results from topographic models. **Q3. Line 85: “…diminished capacity” (a) are you implying neural predictivity (and if so, V1? VTC?), image-by-image human behavior prediction, or something else?** **Response:** Sorry that this was vague. We imply neural predictivity in V1 and IT (from BrainScore) as well as neuron-to-voxel match from NSD (see below). We will also present these data in a table to highlight the performance improvements better. **Q4. (b) Margalit et al. (2024) show that TDANNs have a higher correlations... under one-to-one matching. Can the authors please clarify?** Response: Yes we also performed the same voxel-to-unit one-to-one on NSD from the VTC for all 8 subjects. We now clarify this better in the text and include a table with these numbers for both TDANN and TopoNets **Q5. Line 161 - “8 distinct ResNet-18 models …”...What is the 8th model?** **Response:** Thanks for catching this. This was originally a typo. However, based on the reviewer's feedback we trained an additional ResNet-18 model with topography induced on the same layers as TDANN (see above). So we now do have 8 distinct ResNet18 models.

Response 2 to Reviewer FUA2

**Thank you for highlighting changes where we could improve the writing and for not letting these things affect your scores** > In-text citations that are parenthetical and not narrative We have fixed these now. > Line 182 - there should be at least a single sentence explanation of what FFCV is. We expand on FFCV in the text further > “Resnet" -> “ResNet" Done > Lines 213+ - the authors explain "L1 unstructured pruning” and “downsampling” as policies but not how they are being used as metrics. This should be made clear at this point in the manuscript. We agree. We have changed the text to make this point clearer. > The first paragraph of Result 3.1 talks extensively about "model performance” without being explicit about what the task is (I am presuming object categorization) and what dataset it is being evaluated on (i.e., ImageNet). It is only in the figure on the next page that this is made apparent. I would encourage the authors to be more direct in their writing. We agree. We have changed the paper and are now very direct about the task and performance > Double quotes Fixed > There should be a paragraph in the introduction talking about what topographic signatures look like for both vision and language Excellent point! We introduce topographic signatures right in the introduction now > Line 450 - "B. Topographic" (typo) Thank you! Fixed > Line 483 - the authors claim that they have created “a broad suite of topographic AI models”. The use of the term “broad suite” is, in my opinion, an overstatement, given that the only model classes that were evaluated were residual networks and GPT. I would rephrase this claim. We feel that "broad suite" is a somewhat fair characterization given the sheer number and variety of TopoNets we introduce in this paper. We have in fact gone significantly over and beyond all previous studies (that typically looked at a single model architecture or domain). But we will now rephrase this to a large number of residual models, vision transformers and GPTs". This adjustment ensure that the description is factually accurate and allows the readers to draw their own conclusions about the breadth of our results and analyses. **We thank the reviewer again for these suggestions. These changes have made our study more readable and significantly stronger**

Followup 1 to Reviewer FUA2

We would like to thank reviewer FUA2 for engaging with us! Below we address some of the outstanding points and comments. **1. Statement about model performance odd, given that one of the *core* contributions…** **Response:** We are happy to clarify the intention behind our statement: it was certainly not meant to gloss over specific features or comparisons but rather to highlight other meaningful comparisons (e.g., between TopoNets and LLCNNs) and additional metrics. The performance of TopoNets is indeed a core contribution. We wouldn’t be writing this paper and the reviewer not taking our approach seriously if our models did not learn anything meaningful. We have discussed the challenges associated with comparing models trained on different objectives. But performance gains relative to LLCNN are significant and undeniable. We do not want to misrepresent the claims though. You will find that we are more careful about saying supervised models in several places in the updated manuscript. **2. Why one would prefer using spatial constraints introduced through TopoNets?** **Response:** This is an excellent question, and we have both a concise and an extended response. *The concise answer* (from the BrainScore perspective): TopoNets perform better than TDANN on the on neural metrics from BrainScore (updated paper). Prediction accuracies are higher for every brain region we evaluated. *The more extended answer*: TopoNets offer several advantages. (1) Versatility: TopoNets can be applied easily to conv-nets and transformers (2) Ease of implementation: The method is straightforward, requiring only 3 additional lines of code and quick to run (3) Broader applicability: We are making TopoNets available across a number of architectures and domains: ResNet-18s, ResNet-50s, ViT-based in vision, NanoGPTs and GPT-Neo-125M in language. **3. ...a fundamentally new method…** **Response** We want to emphasize the distinction between the **observation**—smoothness of unit responses across the cortical sheet—and the **implementation** of this property in neural network models. In TopoNets, smoothness emerges via the downsampling mechanism. while in TDANNs, it is achieved via explicit matching with the topographic functional profile and swapping of unit locations. TopoNet is certainly a fundamentally new approach (the implentation) for inducing topography. Both TDANN and TopoNets produce equally smooth unit responses (the observation), as has been quantified and compared in our study. Now there could be infinitely many algorithmic implementations for inducing smoothness of nearby model units. Based on the reviewer’s responses, I feel we are in agreement that the litmus test for a new topographic model must be with regard to the representations learnt because of this additional spatial constraint. Whether it is on Imagenet (for broader adoption in the AI community), on brain prediction metrics (eg. BrainScore), or how well both models recapitulate the qualitative patterns observed in brains. We tried to cover all of these aspects within our study.

Followup2 to Reviewer FUA2

**4. I do not think that is true....** **Response:** We were quite struck by the reviewer's comment. Our response was factually accurate and is easily verifiable. We took this opportunity to double and triple check this information again in multiple ways. But in order to be sure, we are including additional links and details below. We want to underscore that it is of utmost importance to us that we do not misrepresent anything. **Please note that we did not say that TDANN does not incorporate spatial loss in every layer.** In the TDANN source code, we can see that there are two objects that are returned from the forward pass: the logits (`flat_outputs`) and the spatial outputs. The spatial outputs is a dictionary containing outputs from the sub-blocks in ResNet18 with names `layer2.0` , `layer2.1` etc. This information can be publicly accessed via this link: https://github.com/neuroailab/TDANN/blob/80c585df69dcb831d1d6802e0776a8cf25d8cfef/spacetorch/models/trunks/resnet.py#L70 In the definition of the model training step, we find that that the spatial loss is applied on these spatial outputs i.e in 8 different positions within the model. This information can be publicly accessed via this link: https://github.com/neuroailab/TDANN/blob/80c585df69dcb831d1d6802e0776a8cf25d8cfef/spacetorch/train_steps/custom_train_step.py#L75 On the contrary, TopoNet-ResNet18 models have topography induced in *every conv layer* (19 in total) within the residual blocks. In order to match the TDANN strategy, we also trained ResNet18 models by applying TopoLoss only to the `conv2` module in each sub-block (8 in total). As reported before, we observe a smaller trade-off in performance compared to our original approach (3% drop from baseline, compared to 5% reported for TDANN). **5. I appreciate the additional experiments the authors are performing, and I would be inclined to raise my score when neural predictivity scores are added to the paper.** **Response:** Thank you for acknowledging our efforts! We strive to be thorough and careful, supporting each claim with as many lines of evidence as possible. We really appreciate the reviewer's time and consideration for increasing the score for our paper. This discussion has been very constructive and reflects a shared commitment to making the paper stronger and better.

Response1 to reviewer 3kin

We thank the reviewer for their thoughtful feedback. We are reaching out now to begin addressing some of the questions raised and to share our planned analyses, which we will update in the next couple of days. Thank you for your engagement! **Q1: Motivation for choosing 2D versus other possible topographic maps.** **Response:** This is an important point. The choice of a 2D cortical column on which to induce topography was grounded in the understanding of topography from neuroscience. *First*, the selectivity of neurons in sensory cortex is known to change systematically in 2D space. The selectivity reflects the organization of feature preferences (e.g., orientation, spatial frequency) across the cortical surface. The pyramidal neurons along the depth (third) dimension within a cortical column share similar selectivity (but reflect input/output relationships). Therefore topographic organization here specifically refers to systematic changes in selectivity as a function of distance across the 2D cortical sheet. *Second*, the topographic maps we describe here are directly based on comparisons to human fMRI maps of the cortical surface. Surface maps unfold the 3D folded brain into a 2D sheet, and capture the spatial organization of neural activity in a biologically realistic manner. While other topographical structures may hold interest, 2D maps enable us to compare our method with previous approaches (like TDANN and LLCNN) and are the biologically grounded starting point. **Revision to paper:** We will expand on the motivation for 2D maps and explicitly state the limitations of 2D cortical sheets in the updated manuscript. **Q2. Is there a specific rationale behind comparing this model against LLCNN-G, given the architectural differences?** **Response:** Each prior approach (LLCNN, TDANN) begins with a common base model architecture (ResNet18) and applies an algorithmic modification to induce topography. While this modification could be seen as a change in architecture, the critical question remains: how do these changes impact **task performance** (e.g., categorization) compared to the unmodified base architecture?  LLCNN, especially, was trained in a supervised manner on ImageNet (similar to TopoNet-ResNet18s). Prior studies, including the LLCNN preprint, have highlighted the trade-off between inducing topography and categorization performance, with large reductions in task performance being a recurring concern (despite training of significantly higher number of epochs). It is within this context, our results are meaningful and significant. **TopoNets** represent an algorithmic approach to inducing topography that achieves substantially higher task performance while maintaining equivalent (or higher) levels of topography. This increase performance is critical because it helps a theoretical debate regarding the relationships between dimensionality, topography, and task performance (Section 3.2 and Figure 4). **Proposed Change:** 1. We will clearly highlight the challenges in comparing TopoNets with LLCNN and TDANN, particularly emphasizing the architectural changes and task performance differences, to provide greater transparency in the comparisons 2. We will explicitly report the **change in model performance** (after inducing topography) relative to the base model, emphasizing how TopoNets outperform both the unmodified architecture and previous topographic models, making the improvements undeniable. 3. We will emphasize how TopoNets’ performance dismantles lingering doubts about the trade-off between topography, dimensionality, and task performance. **Q3: Claims are primarily substantiated in vision models without equivalent evidence in NLP. Are there any models to compare in NLP?** **Response:** The reviewer is correct that some of the claims are substantiated in vision models. This is largely because topography in language models is in its infance. At the time of submission, there were no alternative approaches for introducing topography into language models during model training. Note that we did note performance of Topoformers (Figure 3). This makes our work foundational in this area. That said, we would like to highlight some of the core claims of our study that do generalize across vision and language. **Claim 1:** That TopoLoss can be integrated into vision and language models (this is the first paper that applies and compares a topographic loss across conv-nets and transformers in a single study) **Claim 2:** That TopoNets are high-performing across vision and language models (Figure 3). We are now training NanoGPT models to further strengthen the claim in the language domain. **Claim 3:** That topography **not** task performance explains reductions in effective dimensionality of the learned representations in TopoNet models (Figure 4). **Claim 4:** That TopoNets recapitulate brain-like signatures (category-selectivity maps in vision and temporal integration windows in language)

Response 1 to Reviewer gnkR

We thank the reviewer for their thoughtful feedback. We are reaching out now to begin addressing some of the questions raised and to share our planned analyses, which we will update in the next couple of days. **Q1. The weaknesses include a lack of extensive benchmarking against state-of-the-art models beyond those already tested, which may limit the generalizability of the findings.** The model backbones used in this study—ResNet-18 for vision and GPT-Neo-125M for language—were selected to enable direct comparisons with previous studies on topographic models in human neuroscience. Please note we did try to move beyond architectures by enabling topographic versions of Resnet50 and GPT-Neo-125M which are more expressive model architectures. In this sense, TopoLoss is a novel algorithm specifically designed to introduce brain-like topography. The key comparison in this paper is between previous methods for inducing topography and our proposed TopoLoss, not SOTA models. We kindly request the reviewer to evaluate the paper within this intended context, particularly its application to address theoretical questions and its relevance to neuroscience. TopoLoss signifies an advance in brain-inspired topographic modeling across architectures and domains. **Q2. While TopoLoss is shown to improve performance, the paper does not sufficiently address how it scales with larger models or more complex tasks, raising questions about its applicability in high-dimensional settings.** **Response:** Scalability is in fact a key strength of TopoLoss which in retrospect, we should have emphasized more and presented further evidence for. As we noted above, we needed to provide evidence that our algorithm works by comparing it with prior methods that exclusively used ResNet-18 architectures. We thank the reviewer for encouraging us to showcase this aspect further. To address scalability explicitly, we are now training: 1. Vision Transformers (`vit_b_32`) on ImageNet 2. NanoGPTs We will share the updated results in the manuscript and hope these additional tests to more complex architectures (Vision transformers) and tasks (NanoGPTs on 10B tokens) will allay the reviewer’s concern regarding the scalability of TopoNets in high dimensional domains. That said, we kindly request the reviewer to also reconsider the neuroscience context of our work. TopoNets are designed not only for scalability but to provide insights into brain-inspired principles and their computational consequences, bridging gaps between neuroscience and AI. We hope these updates address the reviewer’s concerns fully. Thank you! **Changes to the paper.** We will update the paper with more scaled up models (as above). **Q3. Inspiration from the brain's vision and language processing** **Response:** We are tremendously inspired by the human brain. This study is squarely in the space of developing novel algorithms that endow current models with more brain-like principles to understand (1) _how_ to incorporate brain-like mechanisms into AI systems, (2) _what_ the representational consequences of brain-like constraints are, and (3) _why_ the brain is designed the way it is. Our study directly addresses these foundational questions in the following way. 1. _How_ to incorporate topography in neural networks? TopoLoss is a fundamentally novel approach to inducing topography rooted in neuroscientific principles (and Turing patterns, for afficionados). Beyond resulting in high-performing TopoNets, our algorithm is an inherently versatile framework that can be applied across model architectures (conv-nets and transformers) and domains. 2. _What_ is the representational consequence of brain-like topography? Topography drives representations to be lower dimensional (Section 3.2) compared to baseline models. Consequently results in more brain-like (Section 3.4) representations. This manifests in two ways: (1) by improving the ability to predict neural data in monkey and human brains , and (2) in recapitulating signatures of brain-like processing in the visual and language cortices (like category-selectivity maps and temporal integration windows). 3. _Why_ is the brain’s design topographic: TopoNets demonstrates parameter efficiency through lesioning (L1 pruning) and downsampling. This provides a new perspective on the role of topography in a surrogate computational system (ANNs), offering insights into its functional significance and evolutionary advantages. **Changes to the paper.** We will now refactor the Discussion to directly articulate the neuroscience inspiration and the relevance of our work with these points.

Response 1 to bq7j

**Q1: The selected backbone is outdated, and more meaningful choices like ViT or LLaMA should be considered.** **Response:** The model backbones used in this study—ResNet-18 for vision and GPT-Neo-125M for language—were carefully selected to enable direct comparisons with previous studies on topographic models in human neuroscience. Specifically, ResNet-18 has been used in prior work as a standard architecture for inducing topography in vision models (LLCNN and TDANN). Similarly, GPT-Neo-125M has demonstrated strong alignment with human brain data in language processing tasks, as evidenced by results on benchmarks like BrainScore-Language. We interpret the reviewer’s question as probing whether TopoNets can scale to larger, more complex architectures. **They absolutely can!** To demonstrate the scalability of TopoNets, we have begun training on larger and more contemporary architectures, including ViT (as suggested by the reviewer) and nanoGPT (on 10B tokens from fineweb-edu). Early results already confirm that TopoNets scale effectively, maintaining their performance and topographic properties. **Planned Experiments:** We are training Topographic ViTs and nanoGPTs to further demonstrate the scalability, generalizability, and relevance of TopoNets for modern, larger-scale architectures. Results from these experiments will be included in the revised manuscript. **Q2: The method does not demonstrate performance improvement or discuss efficiency or interpretability.** **Response:** The notion of performance improvement in our work is within the context of incorporating _brain-like principles_ into standard neural networks. In this sense, TopoLoss is a novel and generalizable algorithm specifically designed to introduce brain-like topography. The key comparison in this paper is between previous methods for inducing topography and our proposed TopoLoss, not SOTA models. We kindly request the reviewer to evaluate the paper within this intended context, particularly its application to address theoretical questions and its relevance to neuroscience. TopoLoss signifies an advance in brain-inspired topographic modeling across architectures and domains. As we demonstrate, TopoNets outperform previous topographic models on engineering performance metrics (esp LLCNNs where the comparisons are more directly meaningful, see Reviewer 4) and even at predicting responses in the brain (compared to TDANNs). Regarding performance improvements with baseline models on neuroscience inspired benchmarks 1. We show that TopoNets are lower dimensional (like brains) compared to baseline (non-topographic) models despite their high task performance (Section 3.2) 2. TopoNets are more efficient in terms of parameter usage and weight sparseness compared to baseline models, demonstrating representational economy. (Section 3.3) 3. Toponet feature maps are more directly interpretable and brain-like compared to baseline models (Section 3.4 and updated Figure 6 with category-selective and temporal integration window maps for baseline and untrained models) These findings underscore the dual achievement of maintaining task performance while incorporating biologically inspired principles into neural nets. Thank you for the opportunity to clarify this! **Q3. Consider incorporating ViT and LLaMA as backbones for the framework to enhance relevance and performance.** **Response:** We have started training two models with TopoLoss: 1. ViTs on Imagenet 2. NanoGPTs on the FineWeb-Edu dataset We will update the paper with these results soon.

Response 2 to bq7j

**Q4. Compare the application of topological loss on the transformer's attention matrix instead of the fc matrix.** **Response:** We appreciate the reviewer’s suggestion and would like to respectfully clarify why applying topographic loss to the transformer’s attention matrix may not align with the intended goals of our approach. The purpose of topographic loss is to shape spatially organized, persistent feature representations, and in transformers, this is most directly achieved by targeting the `mlp.c_fc` layer in each transformer block. Prior studies (Geva 2020, Geva 2022) indicate that it is the `mlp.c_fc`module that encodes “world knowledge” (analogous to what we imagine as the language system in human brains). This makes it the natural and theoretically grounded target for inducing topographic organization. The attention layers, by contrast, maps the interactions along the sequence dimension (i.e between tokens). It does not store persistent feature representations, which limits its suitability for the kind of spatial organization we aim to achieve. That said, we greatly value the reviewer’s insights and wonder if the suggestion may be intended to explore a different or complementary aspect. If so, we would be more than happy to address this point further with additional context or clarification. Thank you for raising this thought-provoking question! **Changes to the paper:** We will better motivate the reason for inducing topography in `mlp.c_fc` module as opposed to the attention matrix.

Response to remaining concern raised by Reviewer bq7J

We really appreciate the reviewer's time and consideration for increasing the score for our paper! **We do wish to address the reviewer's remaining concern.** **Alignment of TopoLoss with LoRA in large-scale models** We greatly appreciate the reviewer’s thoughtful feedback and the additional context regarding the interaction of large-scale models and LoRA fine-tuning. This is a deeply insightful point and highlights an important aspect of TopoLoss that we would like to clarify. *LoRA* is designed to enable efficient task adaptation through fine-tuning, while *TopoLoss* is focused on imposing interpretability and localized representations in the model during pre-training. These objectives are *complementary* rather than *conflicting*. Investigating the intersection of LoRA’s low-rank adaptations with TopoLoss would require extending the study to task-specific adaptations. I hope the reviewer concurs that doing so is beyond the scope of the current paper. **But we want to provide evidence that TopoLoss causes no issues with LoRA fine-tuning.** Large-scale transformer models trained with TopoLoss can still be fine-tuned via LoRA on the attention matrix. This division of focus allows us to leverage the strengths of both methods: **LoRA:** For efficient and flexible task-specific adaptations. **TopoLoss:** For interpretability, parameter efficiency, and localized representations during pre-training. To further support this claim, we are now running an additional experiment demonstrating that TopoLoss and LoRA can be used together. Specifically, TopoLoss can be imposed on the mlp.c_fc module during pre-training, while LoRA can be applied to the attention matrix during fine-tuning. We will include the results in the revised manuscript to further clarify this point. Thank you for raising this important concern, and we look forward to presenting these findings asap.

Compatibility of TopoLoss with LoRA fine-tuning

We have now updated the paper once more to specifically address Reviewer bq7J’s remaining concern. **Q1. TopoLoss on the mlp.c_fc module of the transformer may not align well with large-scale transformer models, as these models typically employ LoRA fine-tuning exclusively on the attention matrix.** **Response:** We have now updated the paper with the results from the additional experiments (we proposed yesterday) demonstrating that **models trained with TopoLoss align seamlessly with LoRA fine-tuning** on the attention matrix. Specifically, we evaluated our TopoNet-NanoGPT models, fine-tuned using LoRA on two datasets and across three LoRA ranks (to ensure robustness and generalization of our findings). The results from this experiment are now included in **Appendix A.11** and in an **additional paragraph in the Discussion**. We find that the validation loss for fine-tuning curves show a characteristic and consistent drop following LoRA-based fine-tuning (as expected). Notably, TopoNet-NanoGPTs consistently achieve a larger improvement in model performance compared to the baseline model (likely because of ceiling effects). These findings, summarized in Figure 12, provide important empirical support of our previously theoretical claim: that **TopoLoss integrates seamlessly with LoRA fine-tuning.** We hope these additional experiments that we’ve now conducted will address reviewer’s remaining concern. We kindly request a renewed consideration of our study in light of these additional experiments and updated results. Thank you again for raising this question and for the opportunity to address the remaining concerns.