Convolutional neural networks (CNNs) have been shown to be state-of-the-art models for visual cortical neurons. Cortical neurons in the primary visual cortex are sensitive to contextual information mediated by extensive horizontal and feedback connections. Standard CNNs integrate global contextual information to model contextual modulation via two mechanisms: successive convolutions and a fully connected readout layer. In this paper, we find that self-attention (SA), an implementation of non-local network mechanisms, can improve neural response predictions over parameter-matched CNNs in two key metrics: tuning curve correlation and peak tuning. We introduce peak tuning as a metric to evaluate a model's ability to capture a neuron's top feature preference. We factorize networks to assess each context mechanism, revealing that information in the local receptive field is most important for modeling overall tuning, but surround information is critically necessary for characterizing the tuning peak. We find that self-attention can replace posterior spatial-integration convolutions when learned incrementally, and is further enhanced in the presence of a fully connected readout layer, suggesting that the two context mechanisms are complementary. Finally, we find that decomposing receptive field learning and contextual modulation learning in an incremental manner may be an effective and robust mechanism for learning surround-center interactions.
Paper
References (40)
Scroll for more · 28 remaining
Similar papers
Peer review
Summary
This paper explores the role of self-attention mechanisms in modeling contextual modulation in primary visual cortex (V1) neurons of macaque monkeys. Key findings and contributions include: * Adding a self-attention layer to convolutional neural networks (CNNs) improves neural response prediction of macaque V1 neurons in terms of overall tuning curve correlation and peak tuning (PT metrics). This suggests self-attention provides additional flexibility and benefits complementary to CNNs' inherent contextual modulation mechanisms (successive convolutions and fully connected layers). * The classical receptive field is the primary driver of a neuron's overall response, as models focusing on receptive field information achieve the highest correlation. However, contextual modulation, especially via self-attention, is crucial for strong and robust peak tuning. * Incremental learning, where the receptive fields are learned first followed by self-attention and fully connected layers, allows the model to properly learn the contributions of the receptive field and contextual modulation. This incremental approach leads to a receptive field-centric model more aligned with neurophysiological evidence. * Interpretable contextual modulation effects, such as association fields, emerge in the self-attention module of models that capture peak tuning well. * Self-attention augmented CNNs are more data-efficient compared to baseline CNNs. Overall, the paper provides insights into the computational role of self-attention (single head) in modeling horizontal connectivity and contextual modulation in the visual cortex, and proposes an incremental learning approach to best train networks with self-attention.
Strengths
The paper is overall well written and the results are quite convincing. Here are the points I believe are strengthening the submission: * The paper introduces a new perspective on modeling contextual modulation in V1 neurons using self-attention mechanisms, which has not been extensively explored before. * The authors systematically investigate the contributions of different contextual modulation mechanisms (convolutions, self-attention, and fully connected layers) and their interactions, providing insights into their roles in modeling neural responses. * The incremental learning approach, where receptive fields are learned before contextual modulation, is inspired by the developmental process in the visual cortex, adding biological plausibility to the model. * New evaluation metric: the peak tuning index is introduced to assess a model's ability to capture a neuron's feature preference, addressing the limitations of standard metrics like Pearson correlation. * The authors demonstrate that models with strong peak tuning display interpretable contextual modulation effects, such as association fields, in the self-attention module. I actually believe that this last point should be included in the main paper instead of the Appendix, as it brings added value by bridging previous research at the interface of neuroscience, perception, and machine learning.
Weaknesses
**Major Weaknesses** * While the paper compares the proposed models to established CNN architectures, it does not include comparisons with more recent state-of-the-art models, such as transformer-based models, which have shown promising results in modeling mouse V1 neurons. More precisely, while Li et al. (2023) is mentioned in the introduction, no comparison with this model is provided in the paper. Although it's true that multiple attention heads may intuitively seem unnecessarily complex for the task outlined in the paper, Li et al. (2023) demonstrates very good results in predicting neural responses in mouse V1 neurons. **Minor Details** * Figure 1(b) is not readable: The axes labels, ticks, and different images are unclear. Consider resizing or reorganizing the figure differently.
Questions
Just a minor question: In section 3.4, I read that the loss is MSE. Many (if not most) papers in the field use a Poisson NLL loss, even with two-photon data (where, I agree that assumptions of e.g., electrophysiological recordings and Poisson counting aren't met). Could you elaborate on the use of MSE? Have you tried both and found better results with MSE? Overall, I'm happy to maintain the assigned score (I believe the paper is more of a 7 in its current state, but 7 is not among the allowed ratings) if a more extensive comparison with Li et al. (2023) is provided.
Rating
8
Confidence
4
Soundness
3
Presentation
3
Contribution
3
We thank the reviewer for the kind comments and constructive feedback. Here we prepare the following responses to address the reviewer’s questions: **Comparisons to State of the Art Models.** Li et al. (2023) use a multi-layer, multi-attention-head ViT to model mouse V1 neurons, achieving state-of-the-art results, which highlights the value of the self-attention mechanism. Our work has a different focus. Although we build upon their work, we began our study prior to the release of their paper. We aim to understand which aspects of contextual modulation are necessitated by self-attention, how different contextual modulations interact and complement each other, and whether the non-local third order interaction mechanism (Fei & Pitkow, 2023) unique to self-attention can contribute to predicting V1 neuron responses. We expect that V1T will perform very well, likely better than our attention based CNNs, as it leverages a powerful—but difficult to interpret—black box transformer with many attention layers and heads. Therefore, a comparison to the ViT model by Li et al. is not meaningful, as our focus is understanding the key mechanism of the state-of-the-art model, rather than beating it. **Figure 1b.** Thank you for alerting us to the lack of readability and clarity in Fig 1b. We will revise it accordingly. **Loss Function.** We have tried both Poisson loss and MSE. We present the results with Poisson loss in section A.3, Figure 8E. We note that empirically, MSE provided the best performance.
Summary
The paper demonstrates that a self-attention layer inserted in a convolutional network can improve its ability to predict neural responses to natural stimuli from macaque area V1. The neural responses were collected with two-photon imaging in two separate monkeys. Different variants of a feed-forward CNN are compared (with or without self-attention, with or without fully-connected spatial integration, with end-to-end learning or incremental pretraining and fine-tuning of specific layers). The authors conclude that self-attention can improve neural prediction accuracy, especially for explaining “peak tuning” (i.e., the top 1% of highest responses).
Strengths
* Fitting neural network models to brain data is an important approach to determine the biological plausibility of artificial systems, and to understand computational brain mechanisms * The methods are generally sound and the compared models and ablations allow us to draw conclusions about the investigated mechanisms
Weaknesses
* The paper does not clearly state its primary objective, and the reader is left to choose between two options. The main objective could be to prove that recurrent and horizontal interactions help explain extra-classical receptive field properties in V1, with self-attention as one specific way to implement these interactions. Or, it could be to argue that self-attention (as implemented in modern neural networks with Q, K and V projections) is a good model for recurrent/horizontal interactions in area V1. I believe the former interpretation is better supported by the data, although less novel (since earlier papers have already reported that recurrent layers can improve neural predictions). However, I fear that the community would more readily jump to the latter conclusion, which is unfortunately not well supported. To make this interpretation, the self-attention models should be directly contrasted with other baselines including one (or more) recurrent layer(s) that implement horizontal interactions without self-attention. In the absence of these baselines, only the first conclusion can be supported, but this hinders the novelty of the paper’s message. In any event, the authors should clearly state which of these two goals they are targeting.
Questions
* There is no clear description of the training objective for the different models compared. I assume it to be a regression of single-neuron activation from each training image, using an MSE loss, but this should be described much more explicitly. * The notions of “overall tuning” and “tuning peak” are used in the abstract and introduction, but these are not standard terms, and cannot be understood without some amount of explanation (e.g. “the tuning peak corresponds to the top 1% of highest responses”). * The Zhang (2022) reference does not include information about a journal, preprint server or conference venue.
Rating
3
Confidence
4
Soundness
2
Presentation
2
Contribution
2
Ethics concerns
The paper describes data collection in two macaque monkeys without reporting legal or ethical approval for these experiments.
We thank the reviewer for the constructive feedback and insightful comments. We carefully prepare the following response to address the reviewer’s concerns: **Primary Objective.** The reviewer questions whether we are pursuing: (1) proving that recurrent and horizontal connections help explain extra-classical receptive field properties in V1, with self-attention as a specific way to implement these interactions; or (2) showing that self-attention (QKV projection) with multiplication of Q and K is a good model for recurrent horizontal interaction in area V1. We aim to pursue both goals with caveats. While prior work (e.g., Zhang et al., 2018) shows that adding recurrent circuits can improve neural response prediction—especially when data size is small—our work provides some new insights: (1) Self-attention can complement other contextual modulation mechanisms already present in standard feedforward networks, improving prediction of overall tuning and peak tuning. (2) The introduction of non-local third order interactions (Fei & Pitkow, 2023) through self-attention, where attention weights implicate the involvement of the far surround context, suggests that peak generation may involve non-local excitatory cortical circuits, potentially gated by top-down feedback. In summary, we believe self-attention is not simply just another form of recurrent interaction; it captures an unique third order interaction, distinct from other surround contextual modulation mechanisms. Our work goes beyond showing that contextual modulation explains classical receptive fields; it demonstrates that non-local networks with third order interactions are essential for characterizing both overall tuning and peak tuning (selectivity) of neurons. Regarding the second objective, we do believe that self-attention approximates the third order interaction or flexible gating of local circuits. We do not think the cortical mechanisms are necessarily computing Q, K, and V, and the attention interaction matrix in the same way as in self-attention. But at a functional or computational level, we believe that the third order interactions in self-attention embodies a key element of cortical computation and contextual modulation. We will clarify our goals with the reviewer’s suggestions in mind for our next revision. **Training Objective.** The training objective is MSE loss. We also experimented with other loss functions, such as Poisson loss, but found no significant differences in performance (see section A.3 Figure 8E). We will revise section 3.4 to explicitly describe the training objectives for the different models. **Explanation of Evaluation Metrics.** We thank the reviewer for pointing this out. We will revise our paper to provide a brief explanation of these metrics when they are first introduced. **Zhang et al. (2022).** We apologize for the oversight. Zhang et al. (2022) is an archived paper (https://arxiv.org/abs/2110.00825). We will make sure to cite it properly in the next revision of our paper. **Ethics Statement.** We apologize for not including the following statement, which we should have: "All procedures involving animals for generating the data of this paper were in accordance with the Guide for the Institutional Animal Care and Use Committee (IACUC) of the experimenters' institutions." We will be sure to include this in the next revision of our paper.
Thank you for your response. You are now insisting on the fact that QKV attention is a unique form of recurrent modulation, because it implements third-order interactions; therefore, you suggest that your results go beyond previous demonstrations that recurrent layers improve neural predictions. The main problem remains exactly as in my initial review: "To make this interpretation, the self-attention models should be directly contrasted with other baselines including one (or more) recurrent layer(s) that implement horizontal interactions without self-attention". I thus maintain my original score. In honesty, I believe that this paper in its current framing will do more harm than good to the community. Fellow scientists and the media will be quick to jump to the conclusion that self-attention is a "better" model of V1 processing than previously proposed alternatives (including recurrent layers, GRUs and LSTMs). Yet there is no direct evidence here to support this claim.
We thank the reviewer for their prompt and thoughtful reply. While we are sorry to hear this perspective, we understand and respect where it is coming from. **1.** Our goal is to explore various contextual modulation mechanisms. Our paper demonstrates that the self-attention mechanism, which implements third-order interactions, complements other contextual mechanisms in specific ways. We believe our work contributes a valuable data point and perspective to the field. We disagree with the notion that this could be a disservice to the community. Given the widespread use of transformers, we do not believe this work will lead to uncritical bandwagon adoption of self-attention but rather provide nuanced insights for researchers to consider. **2.** The ViT model (Li et al., 2023), which leverages self-attention within the transformer layer, has already been shown to be a state-of-the-art model in predicting mouse V1 responses (as discussed in our responses to other reviewers). As such, we believe it is important to dissect and evaluate the self-attention mechanism in a controlled framework. We believe that conducting a careful study to understand the merits of self-attention relative to other contextual modulation mechanisms contributes valuable insights to the field. **3.** It is entirely possible that GRU and LSTM architectures could also realize sophisticated contextual modulation and perhaps complement the three mechanisms we evaluated here. However, these models typically require significantly more parameters, and to our knowledge, they have not yet been systematically evaluated for V1 neural response prediction. Our claims are deliberately limited—we do not assert that self-attention is inherently superior to LSTM or GRU. We believe it is unfair to dismiss our work solely because we have not exhaustively tested every possible alternative mechanism that remains unexplored in this domain. **4.** Since the start of this discussion period, we have implemented CNN models augmented with one and two recurrent layers as used in Zhang et al. (2018) and are currently running tests for direct comparison (as mentioned in our responses to other reviewers). So far, in experiments with a subset of neurons from M2S1 (see table below), we have not observed significant improvement over the baseline feedforward model in overall tuning (correlation) or response peak measures, despite performing a coarse hyperparameter grid search. It is possible that our experimentation with recurrent models has been insufficient or that recurrent networks are primarily beneficial with smaller datasets, as Zhang et al. (2018) demonstrated greater data efficiency and improved performance under limited data conditions. Our dataset sizes (34K images for monkey 1 and 50K images for monkey 2) are significantly larger—over 10 times greater than the long presentation data (2250 images) and over 4 times greater than the short presentation data (8000 images) used in Zhang et al.’s study. We are now testing models trained on subsets of our data to verify whether recurrent models offer advantages with smaller datasets. It is also important to note that a CNN with a recurrent layer can be unfolded into a deeper feedforward model. To test this, we experimented with adding two extra convolutional layers (with kernel size 3) and four extra convolutional layers (with kernel size 2) to the feedforward model but found no improvement, suggesting that our baseline feedforward model may already be saturated in performance. This could also explain the lack of improvement observed in the recurrent models we implemented. | Model | CORR. | PT_S | PT_J | |--------------------------------|-------|--------|--------| | ff-CNN | 0.474 | 17.333 | 8.667 | | ff+SA-CNN | 0.498 | 19.667 | 11.000 | | Recurrent CNN (1 Recurrent Layer, 7 iterations) | 0.435 | 5.667 | 3.333 | | Recurrent CNN (2 Recurrent Layers, 7 iterations) | 0.450 | 1.000 | 0.660 | | Deeper CNN (2 Additional 3x3 Convolutions) | 0.474 | 14.667 | 8.333 | | Deeper CNN (4 additional 2x2 Convolutions) | 0.472 | 11.667 | 6.667 |
We follow up here on our previous hypothesis that recurrent networks may be more beneficial when working with smaller datasets. We implemented a CNN with recurrent layer augmentation, following the scheme described in Spoerer et al. (2020) (see [Spoerer et al. (2020) Recurrent neural networks can explain flexible trading of speed and accuracy in biological vision]). Additionally, we referenced Zhang et al. (2021) (see [Zhang et al. (2021) Recurrent networks improve neural response prediction and provide insights into underlying cortical circuits]) for implementation details. See Figure 3 in https://anonymous.4open.science/r/sacnn/iclr_2025_disscussion_figs.pdf for the following results. Our results indicate that when the full training dataset (100% data) is utilized, the feedforward CNN augmented with self-attention (ff+SA-CNN) outperforms the recurrently augmented CNN across both overall tuning metrics and peak tuning metrics. However, consistent with the findings of Zhang et al., the recurrent augmented CNN exhibits superior data efficiency. Specifically, when the training dataset size is reduced to 25% or less, the recurrent augmented CNN surpasses both ff-CNN and the ff+sa-CNN in performance.
Summary
This paper presents a method for predicting the neural activity of neurons in the primary visual cortex of primates upon presentation of natural images. This method is based on the use of a hierarchical neural network comprising several convolutional layers and learned in a supervised manner with the aim of predicting neuronal activity. An innovation of this algorithm lies in the inclusion of a self-attention mechanism in the processing sequence. This attention mechanism could then, on the one hand, help improve prediction capabilities and, on the other, help understand the modulation mechanisms present in a biological neural network which consist in facilitating or depressing the response of neurons according to a context.
Strengths
The strength of this paper lies in the rigorous introduction of this model and the comparison of different architectures, enabling the reader to disentangle the role of different network components. This method is validated by computational simulations that quantify the correlation between observed and predicted activity. This methodology then enables predictions to be made that confirm or refute certain biological theories, such as the modulation of normal activity according to the context of neighboring neurons.
Weaknesses
I've noted a number of weaknesses in the document that affect its comprehensibility and potential impact. Foremost, the presentation needs to be simplified. First, it is difficult to understand from the abstract that your aim is to predict neural activity. In terms of the structure of the paper, there are inconsistencies in the sequence of sections, making section 3.5 and the model evaluation seem to conclude the paper. Some parts of the methods imply knowledge of intermediate results, and a major restructuring is needed. Secondly, the model is missing some essential elements for a proper understanding. If you describe it well, the input image given to the network is not the one you want to produce. From the figure, I deduce that it's the fluorescence contrast of the image, but this needs to be written more explicitly. Thirdly, the figures are not always clear: figure four, for example, neither shows nor explains what is shown here. I'm guessing that the images are sorted according to their response, and that you superimpose the prediction that is made. Figure five shows results, but with fonts too small to read. This figure can also be simplified, as many elements seem redundant. Finally, the results discussed in the discussion are interesting, as they allow biological facts to be deduced from the study of a neural network, and in particular from the inclusion of an attention layer. However, this conclusion may also be the consequence of the limitations of the various models. To be validated, the conclusion should be causally related to the mechanisms described. In particular, I'm surprised that you don't look more closely at the activity in the attention layer: what are the strength and distance of the different modulations? This would enable you to draw neurological conclusions.
Questions
Assuming that the limitations detailed above will be corrected, I have many questions about this work: You use a correlation measure to compare your predictions with neurobiological observations. Can you justify this approach in relation to the observed distribution of neuronal activity? Indeed, it seems that this distribution is sparse and that another measure might be more appropriate. You have placed the attention layer after a features layer that is supposed to represent simulated activity in a neuron in the primary visual area. As such, can you interpret the various Q, K, and V features you've inferred during learning? It seems you're trying to predict activity on a per-frame basis, but neural activity is dynamic. Do you think you can apply this type of model, and in particular the attentional layer, to the temporal domain? Finally, you mention a homology between the neural network that learns to predict neural activity and the network of biological neurons underlying the production of this activity. This major point needs to be better introduced.
Rating
6
Confidence
5
Soundness
3
Presentation
2
Contribution
4
We thank the reviewer for the insightful advice and questions. We prepare the following responses to the reviewer’s questions: **Presentation.** We mention in the abstract that the task in this work is "neural response prediction," but we recognize that we should clarify this further. Specifically, we are developing a deep learning model to predict a neuron's response to arbitrary novel natural images. We will also rename section 3.5 from "Model Evaluation" to "Model Evaluation Metrics" for clarity. These revisions will be incorporated into our next revision. We recognize that part of our presentation may seem awkward due to the iterative nature of our work. The approach in section 3.3 is driven or motivated by the results presented in section 4.2. In our current structure, we group all the methods together in one section, as is customary, but we did attempt to provide some motivation for the method in 3.3 with a brief reference to the results in 4.2. We believe this is a reasonable compromise. We are open to suggestions regarding restructuring. **Neural Response Modeling Details.** The problem of neural response prediction is well-established, and the reviewer’s inference is correct. We will work on revising the paper to make this more explicit and easier to understand. Additionally, the appendix provides more detailed descriptions of the experimental procedures. **Figure Clarity.** Thank you for pointing out these issues. The reviewer is correct in the way that we sort images and superimpose predictions in our figures. We will revise our figures according to these suggestions. **Biological Interpretations of Results.** A hallmark of self-attention is its multiplicative capability for third order interactions in modeling local and non-local circuit interactions in an image-dependent flexible way—features that cannot be easily implemented in a feedforward or recurrent-augmented CNN, even though, in principle, they are universal function approximators. Using self-attention to approximate the flexible gating function of the V1 circuit is the motivation for our study. We do not believe that neural circuits directly use the exact algorithms and mechanisms of computing Q, K, and V, or the interaction matrix based on the outer product of Q and K. However, we believe the cortical circuit performs a very similar function at the "computational level," as defined in Marr's framework. The performance benefit of the self-attention mechanism over feedforward CNN and simple recurrent-augmented CNN models suggests that flexible routing and non-local interactions likely play a role in the contextual modulation of V1 neurons. **Attention Layer Activity** We thank the reviewer for suggesting a more detailed analysis of the attention layer, particularly regarding the strength of interaction as a function of distance. We have observed that, on average, the magnitude of attention weights falls off marginally (less than 4%) with increasing distance from the center, indicating that the attention mechanism is indeed capturing non-local global interactions. Additionally, we have also included more preliminary results on visualizing attention in section A.6. **Correlation Measure** Correlation is a standard method for evaluating neural fitting. We also tested several other measures, as shown in section A.3. Due to the sparsity of the responses, these measures primarily capture the general trend. This is why we proposed novel measures specifically designed to evaluate the peaks. **Interpreting Q, K, V.** The self-attention layer is designed to functionally approximate the recurrent computations (potentially involving horizontal and feedback connections) in the primary visual cortex. We do not believe that neural circuits directly use the exact algorithms present in the Q, K, V computations within self-attention. Instead, we believe that Q and K can be interpreted as the sparse manifold transform described by Chen and Olshausen (2018). **Temporal Domain of Neural Activity.** This dataset contains only the mean response of each neuron to different stimuli, averaged over the duration of a trial and across trials for each stimulus. To extend this to the time domain, we could incorporate a 3D spatiotemporal input layer, similar to the approach used in Ganguli’s work for modeling retinal ganglion cells with deep networks. However, in our case, we don't have a temporal response (PSTH) for each stimulus that the model would need to predict. **Homology Between Neural Networks and Biological Neurons** "Homology" might be too strong a term. We believe that if a particular network architecture leads to better neural response predictions, it could suggest—though not conclusively prove—that the computation or algorithm associated with that architecture may offer a better model of the underlying algorithms and mechanisms of cortical circuits.
Summary
The manuscript asks if self-attention can improve predictions of primary visual cortex (V1) response to images. This is motivated by two facts: one is surround modulation data in V1, usually attributed to lateral and top-down feedback connections; and the other is that artificial recurrent architectures predict some aspects of V1 responses better than purely feedforward. Self-attention could augment feedforward networks with spatial context information, beyond or differently from what can be achieved with receptive field expansion due to successive layers and final fully connected layer. The authors execute a series of well thought-out comparisons between alternative architectures and learning schemes. The main results indicate that self-attention indeed improves predictive power, particularly when models are trained incrementally (learning the receptive field first). And that this improvement is more visible in the ability to capture responses to the most driving images for the neurons, rather than the overall response correlation across all images. This is interpreted as evidence that receptive field computations are more important to capture overall tuning, whereas surround computations are modulatory.
Strengths
Excellently well written and executed. The problem is situated in the relevant literature, the motivation and significance of findings are conveyed strongly, the novelty is not overstated and the design focuses sharply on the question being asked. The results are very convincing. A pleasure to read!
Weaknesses
Minor only. 1) The paper certainly adds to understanding neural system identification. But at a high level, do the result say anything about V1 contextual modulation itself? Or do the authors envision a path to that goal by fitting self-attention augmented CNNs? In any event, I don’t think this detracts from the value of the manuscript. 2) Even though not systems identification, this paper https://pubmed.ncbi.nlm.nih.gov/37738258/ is very relevant and in my opinion should be discussed.
Questions
1) It seems important to add a performance comparison between the self-attention augmented CNNs and CNNs with recurrent layers. Or to explain why not relevant. 2) Does any of the model capture the basic modulatory natural of extra-classical receptive fields, i.e. surround suppression and no response to an annular stimulus? The analysis of Fig. 6 goes in that direction but it’s not quite the same, if I interpreted it correctly.
Rating
8
Confidence
5
Soundness
4
Presentation
4
Contribution
3
We thank the reviewer for the positive review and thoughtful feedback. We address the reviewers' questions as follows: **Results in the Context of Understanding V1 Contextual Modulation.** The self-attention mechanism is distinct from both feedforward CNNs and Markovian local recurrent networks for two key reasons: (1) it captures non-local interactions, and (2) it enables flexible, image-dependent interactions. The ability of self-attention to improve upon the contextual mechanisms present in feedforward deep neural networks suggests that it integrates the surround context in a potentially unique way. We believe this could be related to the concept of third order interactions (Fei & Pitkow, 2023), which involves image-dependent multiplicative gating of recurrent interactions to mediate contextual modulation. This is one of the reasons we are particularly interested in using self-attention-augmented CNNs. **Pan et al. (2023).** Odelia Schwartz’s works are highly relevant to our study. Specifically, her earlier paper (Coen-Cagli et al., 2015) demonstrated that contextual influence is governed by flexible gating, which inspired our research. Self-attention offers a powerful mechanism for flexible gating, something that might not be achievable with the fixed interaction kernels in a recurrent circuit-augmented CNN. We appreciate the reviewer for highlighting this recent paper by Pan et al., which uncovers the facilitatory and inhibitory aspects of the contextual surround. We will certainly discuss this paper in our next revision. In our work, we use self-attention to model a flexible, image-dependent facilitatory and inhibitory surround. **Comparison Between Self-Attention Augmented CNNs and CNs with Recurrent Layers** Li et al. (2023) demonstrated that a large-scale ViT model with multiple layers and attention heads achieves state-of-the-art performance in predicting mouse V1 neural responses. Based on this, we expect that their ViT model, V1T, would outperform existing architectures. A CNN with a recurrent layer (as in Zhang et al., 2018) could, in principle, be expanded into a deeper network, as shown by Nayebi et al. (2021). The goal of our study is to evaluate why self-attention—an essential component of ViT—can complement other forms of contextual modulation and improve performance. This is why we did not compare our self-attention-augmented CNN with a recurrent-augmented CNN. However, the reviewer’s suggestion led us to reconsider. It could indeed be valuable to compare with the recurrent model, as it might highlight the advantages of flexible, recurrent interactions in self-attention over fixed recurrent circuits. Since the start of this discussion period, we have attempted to implement a CNN with a recurrent-augmented layer and matched hyperparameters. Unfortunately, there is no readily available recurrent-CNN implementation we could directly use for comparison, nor is there a recurrent module that is easily compatible with our architecture. Despite our efforts, our recurrent-augmented CNN has not yet been properly tuned to produce preliminary results. We will aim to have a functional implementation by the next revision. **Capturing Basic Modulatory Nature of Extra-Classical Receptive Fields.** Feedforward models alone are sufficient to produce the basic surround suppression effect. However, the self-attention mechanism plays a crucial role in generating the peak response. Figure 6 depicts the contributions of the classical receptive field and surround context in generating the predicted response.
Reply to response by the authors
I applaud the authors' effort to include comparisons against recurrent layers, I believe showing it in the final paper, at least for simple vanilla RNN, would make this a stronger contribution (though I agree with the other reviewer that a more extensive comparison with GRU, LSTM would be desirable in the long run). Regarding "Capturing Basic Modulatory Nature of Extra-Classical Receptive Fields. " I referred specifically to the basic empirical observations that defines surround modulation: 1) the center stimulus drives the neuron's response; 2) the surround stimulus by itself does not drive the neuron's response; 3) the [center + surround] stimulus modulates the response to the center stimulus alone. As far as I can tell, the manuscript does not show the second of the three elements above (I expect that the model will be able to reproduce this, but it seems a necessary sanity check for any model of V1 surround modulation).
We thank the reviewer for the clarification, and prepare the following results in response. See https://anonymous.4open.science/r/sacnn/iclr_2025_disscussion_figs.pdf for the relevant figures. We mapped the neural networks’ receptive fields using optimally oriented short bars. The left and center panel of Figure 1 illustrates two example neurons’ receptive fields, indicating that a short bar outside the classical receptive fields does not elicit a response greater than the baseline. The half-height maximum receptive field diameter is approximately 3 pixels (0.4 degree visual angle), corresponding to a 2-STD contour diameter of about 0.8 degree visual angle, consistent with experimentally determined receptive field sizes. The right panel of Figure 1 displays the distribution of mapped receptive fields of neurons for one site. To verify that our neural networks exhibit the extra-classical receptive field contextual modulation phenomena noted by the reviewer, we replicated following experiment by Cavanaugh et al. (2002) on our digital neurons (see [Cavanaugh et al. (2002) Nature and Interaction of Signals From the Receptive Field Center and Surround in Macaque V1 Neurons]). The following describes the methodology of the experiment. Neurons were presented with center grating stimuli in optimal orientation and spatial frequency of varying diameters (center-only stimuli), centered on their receptive fields. The diameter of the smallest center-only stimulus eliciting at least 95% of the neuron’s maximum response defines the GSF (grating summation field). Each neuron was also presented with surround grating stimuli (with gray apertures centered on the receptive field) of varying sizes. The classical receptive field was estimated as the aperture diameter at which the annular stimulus response is significantly below the neuron's maximum response to a circular grating patch. This aperture, referred to as the AMRF, represents where surround-only stimuli do not elicit significant responses above the baseline response. When the center-only stimulus is significantly larger than the size of the classical receptive field, surround suppression—a characteristic example of extra-classical receptive field modulation—is observed. Figure 2A shows four example neurons from Cavanaugh et al.’s study demonstrating the classical surround suppression phenomena. Figure 2B and 2C shows our replication of their experiment, comparing results for the baseline model (ff-CNN) and the self attention model (ff+sa-CNN). The classical receptive field size of our neurons estimated using AMRF was approximately 0.8 degree visual angle. We found that when the center-only stimulus exceeded this classical receptive field size, surround suppression—a hallmark of extra-classical receptive field modulation—was consistently observed.
Meta-review
This paper describes deep learning model for predicting neural activity in primary visual cortex that incorporates convolutional layers as well as self-attention. It shows that self-attention can indeed improve performance relative to feed-forward neural networks, particularly when models are trained incrementally. This represents is a novel approach to modeling V1 data, and offers the potential to help understand nonlinear information processing in real neural circuits. The reviewers praised the paper for its novelty, clarity, motivation, and significance, and I'm pleased to report that it has been accepted to ICLR. Congratulations! Please revise the manuscript to address all reviewer comments and suggestions, especially the points about figure clarity and legibility. (I also agree with the reviewers that it would be nice to see a comparison to the Li et al. (2023), if possible).
Additional comments on reviewer discussion
One reviewer (eVhj) felt that the paper did not go far enough to compare self-attention with other methods of recurrent processing (eg GRU, LSTM), and is therefore misleading to claim that self-attention improves prediction of neural responses relative to other forms of recurrence. However, I did not feel that the paper was making this claim (the authors assert in the rebuttal "we do not assert that self-attention is inherently superior to LSTM or GRU"), and it is not the case (to my knowledge) that there is other existing work that relied on other such forms of recurrent processing that the authors could have compared to. (So I think it is too high a bar to ask that the authors implement models with *other* forms of recurrence in order to say that theirs is better... this is certainly a valid idea for future work, however!)