AdaNCA: Neural Cellular Automata As Adaptors For More Robust Vision Transformer

Summary

The work presents present a neural network architecture that combines vision transformers and neural cellular automata. The work shows that this hybrid architecture competitively at image classification task with a downsampled version of Imagenet (224x224). Furthermore, the results show a small improvement on performance against adversarial attacks and OOD test against the chosen vision transformers baselines.

Strengths

- The paper makes use of hybrid architecture (NCA and ViT), a combination that remains underexplored in the current machine learning literature. - The presented model performs competitively against the baselines. - The paper is clear, well written, and results are well presented with an thorough appendix detailing training setup.

Weaknesses

- Readability of Table 1 could be improved: it is not indicated the values for adversarial attacks and OOD inputs: accuracy? error? In some cases high number are bold, in other low number are bold. - Please report whether the results are for the single best trained model. - The argument put forward by the authors to motivate the AdaNCA architecture is its competitive performance, however the magnitude of the performance improvement is limited, notably keeping in mind that the work is demonstrated on a downsampled version of Imagenet. - Since the architecture is presented as plug-and-play, it's unclear why the AdaNCA is not simply added to a ViT pre-trained models. Having a simple way of improving robustness of ViT with a quick training of the NCA would make the model much more impactful and scalable. ## Typos: - Plug-in-play -> Plug-and-play

Questions

- How does AdaNCA differs from Vision Transformer Cellular Automata model from Attention-based Neural Cellular Automata, Tesfaldet et al. 2022? -What's the motivaiton of using channel-weighted convolution for the perception part part of the NCA (what you call "dynamic interaction")? Wouldn't this be equivalent to the transformation that the update MLP does on concatenated channels vectors from standard depth-wise conv layers? - Why training AdaNCA from scratch rather than adding it to a pretrained ViT model? Could you report performance on this? - The paper conjectures that it is the stochasticity of the NCA that may be positively contributing to the AdaNCA performance for adversarial attacks. Have you tried introducing noise during training to the ViT baselines to compare performance? - Line 163 : “NCA typically queries the cell states at a random time step T”. I don't understand what do you mean with "NCA queries a cell state".

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

- No code is provided.

Summary

The authors propose the introduction of NCAs into VisionTransformers to improve the robustness against adversarial inputs as well as out of distribution data. This improves basic ViT architecture by up to 10% against specific adversarial attacks. In addition this modification, they propose Dynamica Interaction, for faster communication between cells/tokens. As the choice of the right layer to insert the NCAs is not trivial, they introduce a method to find the most efficient position based on the redundancy. The whole method has been evaluated against a good number of methods and on OOD data as well as adversarial inputs.

Strengths

- I highly appreciate the effort of integrating NCAs into existing architectures, to combine the established method with the specific traits that NCA can bring, such as the authors have shown: robustness - The authors show generally strong robustness improvements when comparing the "classical" ViT variants, RVT-B, FAN-B, Swin-B and ConVitB with their AdaNCA variants - The proposed method for determining the best position for inersting the NCAs is sound - The manuscript is generally a well written paper and good to follow. The appendix adds valuable extra information

Weaknesses

- My major concern is the choice of baselines. While generally a good selection has been made, i strongly dislike the choice of removing the additional ADL loss (according to [22] the ADL loss is more important than TAP). The only logical explanation I could find for this choice, is that it performs better than the NCA based method. - This is especially misleading since you call your baseline "TAPADL-RVT". But the "adl" in this method stands for the loss. - Considering that you do not use the "adl" loss Table 2 is particularly unfair, as these are OOD examples, something the loss particularly optimizes on. Especially considering that the difference is not very big even without it. - Why does Table 2 once compare with the standard baseline and once with TADPADL? - The argumentation in appendix D, why the comparison with classical / concatenating NCAs is missing is unclear. Why would this require NCAs with 10 million parameters? In general it is not clear, why the NCAs are this big More minor remarks: - "enables the modeling of global cell representations" in abstract is confusing - They choice of hyperparameters is unclear. Why is e.g. the EMA different inbetween AdaNCA setups?

Questions

- I very strongly encourage including the results for the original "TAPADL" [22], with the correct loss for training. I do not see it as a problem if you perform worse, which is likely the case, but it is absolutely necessary for completness. - Instead of doing this misleading modification to the TAPADL, did you try to add the ADL loss to your modified model? - How is it possible, that the NCAs have such a huge number of parameters? One major advantage of them is, that they do not need many parameters, so im curious why you chose to make them 1-2.5m parameters big. - Why did you choose the number of steps for NCAs so low? It is very atypical for NCAs to run for 2-3 steps. - It is not one of my major points, but can you call something an NCA if runs for 2-3 steps? - Your statement about the release of the code is unclear to me. Will you release it in case of acceptance?

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

The authors have adequately addressed the limitations

Summary

This paper introduces Adaptor Neural Cellular Automata (AdaNCA), a plug-and-play module designed to enhance the robustness and performance of Vision Transformers (ViTs). The innovation lies in integrating Neural Cellular Automata (NCA) as intermediary adaptors between the layers of ViTs. The paper demonstrates that AdaNCA can significantly improve the robustness of ViTs against adversarial samples and out-of-distribution inputs. The authors also propose a Dynamic Interaction mechanism to reduce computational costs and provide an algorithm to optimize AdaNCA placement within ViTs.

Strengths

- The integration of NCA into ViTs is a novel approach that addresses the robustness issue prevalent in current ViT architectures. - AdaNCA shows improvement in robustness with only a small increase in parameters - The results show improvements across 8 robustness benchmarks and 4 different ViT architectures. - AdaNCA demonstrates improvement in accuracy under adversarial attacks on the ImageNet1K benchmark

Weaknesses

- The experiments are primarily conducted on image classification tasks. It would be ideal to see how AdaNCA performs in other domains or tasks, such as object detection or segmentation, to assess its generalizability. - The scalability of AdaNCA to very large-scale datasets and higher-resolution images is not yet studied.

Questions

- Would be good to elaborate a bit more on how the Dynamic Interaction works and why it is novel - How does the developing pattern of the NCA look over time? Are differences visible between a version with and without dynamic interaction?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes.

Summary

This paper proposes a strategy for improving the image classification robustness of Vision Transformers (ViT) through the use of specialized networks that are inserted at strategically placed layers within the ViT model. These networks are called Adapter Neural Cellular Automata (AdaNCA) and are intentionally chosen due to the proven robustness characteristics of NCA on various tasks such as image generation and classification. Although there has already been a connection made between NCA and ViT models (ViTCA from Tesfaldet et al.), AdaNCA differentiates itself by not trying to be a ViT in and of itself (as ViTCA does), but by acting as an adapter that can be placed at various layers within a ViT to improve its robustness. Through exhaustive experimentation, the authors prove the viability of using NCA in a much larger scale setting than ever before (as far as I know) through this adapter-type approach, thus providing a new pathway for the NCA community to consider when it comes to practical applications. Another difference between ViTCA and AdaNCA worth mentioning is the downstream task at hand: small-scale image denoising vs. large(r)-scale image classification. A quick summary of what NCA are: they're a fairly recent (circa 2019-20) computational paradigm that build upon the much older model of Cellular Automata (CA). Both NCA and CA consist of a connected lattice of stateful cells whose states are recurrently updated through the repeated application of an update rule. The update rule consists of two stages: an interaction stage, where for each cell, information is gathered from its neighbouring cells; and an update stage, where this information is processed to produce a cell update, which can be applied in a residual manner or be directly treated as the new cell state. CA use a handcrafted update rule, with a popular one being Conway's Game of Life, while NCA use a learned update rule in the form of a neural net with convolutions in the interaction stage and an MLP in the update stage. The NCA update rule is trained via some downstream task, where the cell grid / lattice is evaluated against some target state after a certain number of cell updates. Due to the way NCA are trained (stochastic application of cell updates, repeated evaluation against target state at various points of cell lifetime via pool-based training, hidden states to facilitate cell message passing, etc.), they end up being fairly robust models, able to correct themselves in the presence of adversarial attacks (e.g., various types of structured or unstructured noise) and adapt to OOD situations. The three main contributions of this paper are as follows: 1. Presenting AdaNCA, a NCA-based small model that is able to be inserted at various points within a ViT to improve its robustness on image classification. On average, the classification accuracy gains on clean, noisy, and OOD data outweigh the added parameter and computational cost in a non-negligible (statistically significant) manner. 2. Introducing a new type of cell interaction strategy called "Dynamic Interaction" (and a Multi-Scale variant) that uses far fewer parameters and FLOPS when compared to a vanilla concatenation approach (typically used by previous NCA), while being more parameter efficient than a simpler summation-based approach. This is the part where you combine the information from the various filter responses in the interaction stage before providing it to the MLP that performs the update stage. 3. Proposing an effective strategy for picking the best ViT layers for inserting AdaNCA. This strategy is backed by an exhaustive analysis that proves the strategy's effectiveness in maximizing the robustness gains of inserting AdaNCA within a ViT. In short, they show that AdaNCA is best inserted between sets of layers where each set consists of highly redundant layers. Basically, AdaNCA helps information flow between two sets of layers where the two sets are not redundant from one another. There are some smaller things I'm leaving out of this summary due to how extensive the paper is in some areas, but this is the overall approach.

Strengths

* Originality: * The authors embark on the challenging task of proving the feasibility of NCA to a crowd where many don't believe in their usefulness and where many more have not even heard of NCA. Instead of pushing tired areas of machine learning research, these authors have taken a risk in pursuing a niche and not-yet-proven area of machine learning research and delivering on showing promising results on practical applications that the community typically cares for. In relation to other NCA research, the authors take the first meaningful step in applying them to a much larger scale setting than before (image classification on ImageNet1K and various other versions of it). * Several advancements are proposed: a Dynamic Interaction strategy that's quite original and seemingly effective (although I do have some gripes on how effective it really is), a dynamic programming algorithm for choosing the best layers to insert AdaNCA, new metrics (Set Cohesion Index, among other related ones), and the idea of using an NCA as an adapter to improve ViT robustness. * Quality: * The submission is technically sound for the most part. Most claims are well supported by exhaustive experimentation and analyses, all appropriately chosen to further their narrative. * The authors are honest about evaluating both the strengths and weaknesses of their approach. Although there are areas I have gripes with (which I'll mention in the Questions and/or Weaknesses sections). * Clarity: * For the most part, this is well-written paper. Clear, concise, detailed, and thoughtful. The authors clearly put in a lot of time and effort in trying to leave no stone unturned and I commend them for that. The main manuscript and the appendix is not only exhaustive in trying to prove the benefits of their model, but the experiments chosen, the details of their experiments they listed, and the commentary provided, are all indicative of a thoughtful team that really cares to push this area of research forward. * The appendix was clearly not a throwaway. I'd like to commend the authors in putting the same effort they did in the main manuscript for the appendix. Like the main manuscript, it was clearly and concisely written. Great work. * Significance: * Are the results important? * Yes. These results have given me a new perspective on how to apply NCA and I think the community at large would also appreciate these results. I believe this will convince some researchers, especially those specializing on model robustness, to pay closer attention to NCA due to the favourable cost-benefit tradeoff of using an NCA for improving the robustness of a ViT on image classification. * However, I absolutely do want the authors to answer my questions provided in the Questions section below. Especially the part on the usefulness of Dynamic Interaction and its multi-scale counterpart and if it's possible to use AdaNCA with a mostly frozen and pretrained ViT. * Are others (researchers or practitioners) likely to use the ideas or build on them? * I am confident that this work will see adaptations of it in the near future, although I do have concerns regarding Dynamic Interaction, Multi-scale Dynamic Interaction, and if it's required to jointly train AdaNCA with a ViT from scratch. These concerns are detailed in the Weaknesses and Questions sections below. * Does the submission address a difficult task in a better way than previous work? * Yes. They propose an alternative and effective approach to utilizing NCA for improving robustness on downstream tasks (specifically on image classification, but I'm sure this can be applied on other tasks). * Does it advance the state of the art in a demonstrable way? * Although I'm not sure if it advances the state of the art, as the authors don't make any clear indication that it does, they do demonstrate that it improves many ViT-based models on image classification, particularly on noisy or OOD input. One of the models they apply AdaNCA to is amongst the state of the art (state of the art rapidly changes these days anyways), so I'm convinced this approach would also improve whatever the actual state of the art ViT-based model is on image classification. * Does it provide unique data, unique conclusions about existing data, or a unique theoretical or experimental approach? * As far as I can tell, yes. I just want to say that my favourite part of the experimentation in the main manuscript and the appendix was the layer similarity analyses. Everything from the motivation for focusing on layer similarity, to introducing the Set Cohesion Index, to the figures showing the similarity structure before and after AdaNCA is applied, to the plots and commentary proving a relationship between layer redundancy and robustness, to the dynamic programming algorithm for placing AdaNCA in the best spot based on this proven relationship, and to comparing it with a no-prior approach, were all well done.

Also, I'd like to commend the authors for providing the following: * Providing detailed training hyperparameter information in the appendix (Table 4, 5, 6, 8, 9, 10). * Providing detailed information of datasets in the appendix (C.14). * Providing model and code license information in the appendix (C.15).

Weaknesses

* Originality: * Citation and comparison with related works needs some improvement. My concern here is elaborated upon in the Questions section below. Fortunately for the authors, I think these are easily addressable. * Quality: * Some incorrect / incomplete assumptions were made in various parts of the paper, particularly on parts commenting on where NCA get their robustness from. This concern of mine is detailed in the Questions section below. * The dropout-like compensation technique shown in Eq. 7 was not ablated and thus was not sufficiently motivated. I go into more detail about my concerns about this in the Questions section below. * The Dynamic Interaction strategy was not sufficiently motivated. It should be compared against the typical NCA approach of concatenation. Comparing Dynamic Interaction against a simple summation does not motivate the use of Dynamic Interaction over concatenation. * The Multi-scale Dynamic Interaction strategy was not sufficiently motivated as it hurt performance at three scales, which indicates a potential fundamental flaw in the implementation. The authors provide a brief and insufficient explanation as to why it may be failing at three scales. Some more digging would be helpful here, as multi-scale approaches in neural nets typically go beyond two scales. * Clarity: * All of my concerns regarding clarity are listed below in the Questions section. In short, there are some issues regarding redundancy and lack of explanation in some of the figures, tables, and equations; there's a lack of explanation of how NCA are typically trained; and there's a lack of commentary on the ablation study, making it difficult to appreciate the ablation results shown in Table 3. * Significance: * The Dynamic Interaction strategy was not compared against the typical NCA approach of concatenation, and so its significance as a technique to increase NCA interaction stage efficiency is unclear. Furthermore, it could have been compared against other similar approaches instead of against a simple summation approach (which would obviously be worse). For example, comparing it against a 1x1 conv applied on depthwise conv outputs or comparing it against using a non-depthwise conv. * The Multi-Scale Dynamic Interaction strategy hurt performance at three scales compared to a single-scale Dynamic Interaction, which limits the significance of Multi-Scale Dynamic Interaction. Theoretically, W_Ms should be able to zero out / ignore filter responses from useless scales, so why is it hurting performance when three scales are used? * It seems limiting to have to train an entire ViT alongside AdaNCA from scratch. To really push the adapter narrative, I would suggest adapting AdaNCA to a frozen or partly-frozen ViT. I've suggested an experiment in the Questions section below.

Questions

* L40-41: This statement is missing one crucial piece. It's not only the stochasticity and modulation of local information that make NCA robust against noisy input, it's also the recurrent steps within a single training step that makes them robust (coupled with the pool-based training that extends a cell grid's lifetime to many training steps). The recurrent steps allow the cells to explore a wide variety of states, many of them perturbed by the model itself (especially early on during training when the model is fairly untrained). Tab 3 in your paper also proves this, with the ablation of "Recur" causing the biggest drop in attack failure rate compared to the other ablations. So my suggestion is to state the importance of recurrent updates.

As a bonus, here's a paper I suggest taking a look at [1]. You'll be surprised to see the similarities between training an NCA (Mordvintsev-style) and training an Energy-based Model (EBM) using the technique in [1]. Just like the pool-based training technique that Mordvintsev et al. used, they use a sample replay buffer here. It shows how crucial the exploration and recurrence are for having a robust model.

[1] Du, Y., Mordatch, I. Implicit Generation and Generalization in Energy-Based Models. In NeurIPS 2019. * For the introduction (L26) and related works (L101-103, L89), I'd like to see a mention of how ViTCA (from Tesfaldet et al.) improves the robustness of ViT architectures. I understand that a brief mention of ViTCA was given in L111-113 but nothing about how it achieved a more robust ViT model (albeit with its own unique limitations). On that note, I find it a bit surprising that nothing is said about ViTCA under the context of ViT and robustness and how that compares with AdaNCA and its approach to ViT robustness. Both contribute to a more robust ViT but in different ways. I'd like to see a deeper comparison made between ViTCA and AdaNCA as well as an acknowledgement of ViTCA already having improved ViT robustness using an NCA (albeit with a different downstream task in mind). * L107: You forgot to cite Gilpin [2]! If I recall correctly, he was the first to realize CA under conv nets. I may be wrong, but either way, I think it's necessary to cite and comment on his work if you're bringing up Mordvintsev et al ([43] in the paper).

[2] Gilpin, W. Cellular Automata as Convolutional Neural Networks. In Physical Review E 2019. * Eq 3: Please put a quick explanation of the depthwise convolution operator as you did with the channel-wise concat operator in L142. Something like "where (*) is the depthwise convolution operation." It's just to cover your bases in terms of equation clarity. * Eq 4 and L142-143: I would suggest clarifying how S_out is used to update cells. "where cells get the updated states" is not clear. Perhaps you may want to show that S_out can be used as a residual to update cell state S, or that S_out is itself the new cell state. Either way, it's important to give this extra bit of clarity to the reader. * The Neural Cellular Automata paragraph in Section 3: This section is great for the most part, but I would definitely like to see an explanation of how the NCA update rule is typically trained. So far, only the rough architectural design and computations have been explained, but not how to train them, which is incredibly important. For example, perhaps you can mention at the end that after a certain number of recurrent steps (within a train step), a subset of each cell's state may be evaluated against some downstream task's loss function, or something to that effect. You can also make it clear that one may not necessarily need to use all parts of a cell's state, briefly mentioning the hidden channels that Mordvintsev et al (and many other NCA) use, which has been shown to be extremely helpful in facilitating cell communication. * Related to the above point, I would recommend clarifying some crucial differences between AdaNCA and other "more traditional" NCA that more closely follow Mordvintsev et al's approach. In other words, I would suggest talking about why there are no hidden channels in AdaNCA, what the initial cell states are during training (i.e., Mordvintsev et al. and other NCA approaches start with either a constant or randomly initialized cell grid, whereas here with AdaNCA it seems like the initial cell grid state is just the activations from the layer it's inserted after—correct me if I'm wrong), and why there's seemingly no pool-based training. * L158-162: "In previous NCA works, stochasticity is maintained during testing." ViTCA showed that this was not necessary and that it was possible to train with stochastic updates and test with synchronous updates. I would recommend mentioning this and contrasting it with how you've approached test-time synchronous updates. In particular, I'd mention how ViTCA does not use the dropout-style approach and merely switches to synchronous with no compensation. * On this note, I would like to see an ablation between using the dropout-style compensation vs not using it. Can you quickly run an experiment to compare the two settings? Basically, remove the 1/p in "Train" in Eq 7 and see how it compares with having 1/p. I'm very curious to see how useful this is. * L159-160 states "Such a scheme is problematic in discriminative tasks since reliable numerical outputs are essential." Isn't the whole point of training the NCA update rule to be agnostic to asynchronous cell updates is so that they can be robust to such a scenario during testing? In other words, it's been shown that one can achieve more or less the same results in an asynchronous testing setting by merely updating the cells for more iterations to compensate for those that are behind. The texture synthesis work from Niklasson et al. (Self-Organizing Textures, citation [45] in the paper), showed this in their time synchronization experiment. ViTCA from Tesfaldet et al. showed this in their ablation study, where the results were more or less the same at 50% update rate compared to 100% if you doubled the number of cell updates. Self-Classifying MNIST Digits from Randazzo et al. also showed that stochastic updates are no problem when it comes to reliable discriminative outputs. So I'm wondering why you say stochasticity during testing is problematic for discriminative tasks when other works have shown that not to be the case. * L165-166: "Hence, the trained model can effectively handle the variability and unpredictability of the input, thus being robust against noisy input." I would suggest rewording this sentence. Specifically, I would highlight that it's not just the randomness of when cells get updated that contributes to their robustness, but also that the recurrent updates coupled with the stochastic masking allows the update rule to explore many cell states, teaching it how to recover from such states and move cells towards a good target state. It's also important to note the update rule's own influence on the perturbation of cells early in training, where the update rule has not been sufficiently trained and so acts as a source of noise to itself. So in essence, you're also training the update rule to be robust against perturbations caused by itself. * Fig 2 & 3: I would suggest either removing (b) from Fig 2 and have Fig 3 explain the architectural details, or remote Fig 3 and provide the architectural details in Fig 2. You'll save a lot of space as there's a lot of redundant information between the two figures. Also, I would put a more detailed explanation of the architecture rather than "Overview of AdaNCA architecture". That means explaining the computation pipeline and the different computational blocks so that the reader can easily refer to the caption to explain anything they don't understand in the figure. This will mean some repeat of what was said in the main manuscript, but it will immensely help with clarity when someone wants to quickly glean your paper and understand what's going on. A good rule of thumb is to have the gist of the paper be understood from reading the figures and captions alone. * Eq 8: Please put a quick explanation of the convolution operator. * For Multi-scale Dynamic Interaction, is W_I shared across scales? I'm assuming it is, but just wanted a quick confirmation if it is or not. * For Multi-scale Dynamic Interaction (and the single-scale version), I would like to see a comparison with a vanilla concatenation approach. I want to know if the cost of more parameters and FLOPS is worth it with the concatenation approach. Otherwise it's difficult to judge the usefulness of Dynamic Interaction. Comparing it with a simple filter response summation scheme and showing improvements doesn't really motivate Dynamic Interaction in the context of typical NCA approaches. If you can show that Dynamic Interaction (and Multi-Scale) is more computationally efficient than the concatenation scheme (the relative reduction of accuracy is made up for by a greater relative reduction of parameters and FLOPS), then it solidifies the benefits of Dynamic Interaction and possibly sets a new standard for the NCA interaction stage. * For Multi-scale Dynamic Interaction, have you tried comparing the weighted summation from W_I and W_Ms with a 1x1 conv approach to mix the depthwise info (basically this is how ConvNeXT does it, depthwise conv followed by pointwise conv)? Have you also tried comparing Dynamic Interaction with a non-depthwise spatial conv? If not, can you provide some quick commentary on what you believe the pros and cons would be for each approach and how it would compare with Dynamic Interaction? * For Multi-scale Dynamic Interaction, have you tried using a larger filter instead of a larger filter dilation? I have a feeling that the sparsity induced by the dilation coupled with the sparsity of cell state updates contributed to the model having difficulty with three scales. * For Multi-scale Dynamic Interaction, have you tried modifying the two-layer convnet that produces W_Ms such that its receptive field covers the same receptive field of the max dilation? In other words, if there are three scales, that means you would need the two-layer convnet to have a receptive field of at least 7x7, but the problem is that its receptive field is 5x5. Can you verify if it's required to match the two-layer convnet's receptive field with the max dilation in order to see an improvement with three or more scales? * This could explain why it worked with two scales, as the max receptive field would be 5x5, which is within the receptive field of the two-layer convnet. * L224: I suggest explaining the "K(r = 0.6938, p < 0.001)" part. * Jointly training the ViT model and AdaNCA seems costly. Have you tried a training approach where you use a pretrained ViT model and have every layer but the one before and after AdaNCA insertion points frozen? Considering your layer set redundancy experiments, it seems like these "boundary" layers would be the only ones required to adapt to an AdaNCA that's also being trained. If this works, then this would really push the adapter narrative of AdaNCA. * Another experiment on top of this would be to have all ViT layers frozen and train AdaNCA to act as a feature cleaner where OOD and noisy inputs have their features adapted to the part of the feature manifold that each layer set expects. * Table 1: I suggest providing citations beside each of the metrics under "Adversarial Inputs" and "OOD Inputs". You can simply reuse the ones provided in 4.1. * Table 1: I suggest bolding the lowest param count and lowest FLOPS within each row triplet. * Table 1: I suggest changing the bolding of your model to italicizing or some other visual indicator so as to not confuse the reader with the intention of the bolded numbers. * Table 1: I suggest explaining what the bold means in the caption. * Table 2: Same recommendations as the three above. * Table 3: I suggest explaining the bolding in the caption. * Table 3: I suggest removing the bolding from "Ours." It's already obvious that it's your model. * Figure 6: I suggest not using yellow as the outline colour as it's very difficult to tell amongst all the other yellow-ish colours. Use a strong dotted black or some other colour-blind-safe colour. * Figure 6: I suggest clarifying what "Frequency of Noise" and "Magnitude of Noise" means in the caption. * Figure 6: I suggest specifying what kind of noise is being shown here. * Section 6: Negative impacts were not considered. Please provide brief commentary on potential negative impacts, even if you believe there are none. It's important to show that you've at least considered it. * Figure 4: I suggest briefly commenting in the caption on how AdaNCA does not negatively affect the layer-wise similarity in Swin-B. In other words, providing a smaller version of what was said in 4.2 about how AdaNCA preserves the original layer sim structure. * Figure 5: I suggest providing some brief commentary on (a) and (b) in the caption. * L288: "while overly large scale can lose local information." How so? Theoretically, the model should learn to discount larger scales if they're not useful, so why does including more scales hurt performance and how would it cause a loss in local information? To me, this suggests that W_Ms isn't working the way it's intended to, so there must be some sort of bug or something else entirely. It's just really odd to me that merely adding a third scale is worse than keeping it at a single scale. I'm keen to hear your thoughts on this and if you've looked into this any deeper. * L300: I'd like to see commentary on each of the ablations. "Our design choices contribute to model performance and robustness" is not enough. There should be deeper commentary on each of the ablations, explaining any gains or loss in performance from each (e.g., why does ablating RandS and Recur lead to the highest accuracy?), and comparing relative gains between each (e.g., which component contributes the most to model performance?). * L298: "Ablate the Dynamic Interaction that the..." -> "Ablate the Dynamic Interaction so that the..." * L271: "...networks to conduct the analysis." -> "...networks to conduct this analysis." * "FLOPs" -> "FLOPS". I would like to see my questions and suggestions above addressed to the best of the authors' abilities. My largest concerns lie with the lack of a clear and detailed comparison with the only other NCA model that operates under a ViT setting (ViTCA from Tesfaldet et al.), the relatively insufficient evidence of the usefulness of Dynamic Interaction and its multi-scale counterpart, and the lack of commentary on the ablation study.

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have addressed most of the limitations of their approach, as shown in the Limitations section (Section 5) of the main manuscript and across various points of the main manuscript and appendix. I would like to point out that the potential negative impacts of their approach have not been considered, and so I recommend they comment on this in Section 6 (Broader Impact), even if there are none.