Adaptive recurrent vision performs zero-shot computation scaling to unseen difficulty levels

Summary

The paper introduces a combination of Convolutional Recurrent Neural Networks (ConvRNNs) with a learnable termination mechanism from Adaptive Recurrent Neural Networks (AdRNNs), with the purpose of solving complex, variable-difficulty vision tasks. The paper adapts to the purpose a ConvGRU architecture, or alternatively a “LocRNN” architecture based on a computational model of iterative contour processing in primate vision. Both architectures, together with a ResNet baseline, are tested on the “Pathfinder” and “Mazes” tasks, with a variety of task difficulties. Importantly, the authors show that the AdRNNs architecture are able to generalize to task difficulties greater than those seen during training, requiring a number of computation steps beyond those used in training.

Strengths

The paper consists of an interesting and timely investigation into the hot topic of adaptive computation for problem solving. While the authors are not the first to work on this, they appear to be the first to have demonstrated that RNN models can be effectively applied for vision tasks with adaptive difficulty, and generalize to unseen difficulty levels. The work is clearly motivated, and the overall story and contribution is immediately apparent.

Weaknesses

I believe that the following points need to be addressed for my assessment of the paper to be changed more positively: * **[a]** Overall, the main contribution of the paper is the introduction of LocRNN, as all results, including the one on generalization to unseen difficulty levels for tasks, are downstream of LocRNN being a better architecture than alternatives. However, Section 4.3 appears crowded and hard to parse. I believe that the inclusion of figures illustrating the architectures and describing the differences between ConvRNN and LocRNN would greatly improve the presentation. Moreover, not much effort is spent intuitively justifying the reasons behind the improved performance of LocRNN over other ConvRNNs. * **[b]** In Line 197, it is mentioned that the computational models of cortical recurrent processing are described in detail in Supp. Info, but there is no such section in the supplementary material. Please include this section, to again better explain the intuition behind the introduction of LocRNN. * **[c]** There are small formatting issues. In Line 40, citation names are used without brackets even when they don’t naturally flow within the sentence.

Questions

What is the computational model of cortical processing relevant to the introduction of LocRNN? What is the intuition behind LocRNN’s improved performance and stability over other ConvRNNs?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

1 poor

Contribution

3 good

Limitations

The authors seem to have fairly acknowledged the limitations of their work in their “Limitations” section.

Summary

Authors combine Adaptive computation time (ACT) with convolutional recurrent neural networks to solve two tasks with which their generalization properties can be studied. These adaptive-timestep RNNs were found to halt quicker for easier problems while taking more steps for harder ones. This also leads to generalization to an unseen difficulty level.

Strengths

The paper is well written and the results are convincing for the claim. * Pathfinder task is known to be challenging (Tay et al., 2020) so high accuracy and generalization on that task is very interesting. * Adaptive choosing of timesteps within a dataset is important & valuable for neuroscience and cognitive science research since it allows for identification of harder and easier stimuli. * Inference-time timesteps being more than training is a strong indicator that the RNNs are generalizing.

Weaknesses

While I found the work interesting, I found the paper to be severely lacking in terms of novelty. The paper also misses some related works ([1], [2]). * As mentioned in the paper, Bansal et al., (2022) have studied this with the maze task. Authors say "recurrent networks used in their study are not adaptive, human intervention is required to specify the number of recurrent computational steps by brute force during the testing phase" which I do not agree with - since their RNN stops changing its response after a while, that point can be used as to decide when to stop. * [2] has studied this with Pathfinder task - even finding generalization to difficult levels. Between Graves et al. (2016), Bansal et al. (2022) and [2] I do not see the novelty of this work. 1. Spoerer CJ, Kietzmann TC, Mehrer J, Charest I, Kriegeskorte N (2020) Recurrent neural networks can explain flexible trading of speed and accuracy in biological vision. PLOS Computational Biology 16(10): e1008215. https://doi.org/10.1371/journal.pcbi.1008215 2. Drew Linsley, Alekh Karkada Ashok, Lakshmi Narasimhan Govindarajan, Rex Liu, Thomas Serre (2020) Stable and expressive recurrent vision models. NeurIPS 2020

Questions

1. I am curious to hear why the authors say RNNs used in Bansal et al., (2022) "are not adaptive, human intervention is required to specify the number of recurrent computational steps by brute force during the testing phase" since the RNN can be run till response stops changing. 2. What properties of the images makes the RNNs stop early vs late? For example, in Fig 4 some 19x19 mazes seem to need the timesteps as 25x25 mazes. Have you seen what causes this? Same for the other task. Can this lead to some interpretation/understanding of the RNNs and the solution they are implementing?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

1 poor

Limitations

NA

Summary

The manuscript presents adaptive recurrent networks for processing of static images. The proposed approach augments convolutional recurrent neural networks with the adaptive computation time mechanism, in which the RNN at each step computes an additional halting unit, the value of which is used to determine when to stop the iterative computation. This is shown to yield recurrent vision models that adapt their computational budget to scale with the difficulty level of tasks that require serial grouping operations, such as determining whether two points are located on opposite ends of the same path or segmenting the solution route of a maze. The experimental section also shows that adaptive recurrent networks with gating mechanisms successfully scale to solve the tasks at higher difficulty levels than seen during training.

Strengths

The proposed approach is novel and well-motivated. The experimental results confirm that the proposed models are able to extrapolate the learned knowledge beyond the training distribution to solve more difficult versions of the two tasks. This is quite impressive and I'm not aware of similar studies.

Weaknesses

The write-up is a bit difficult to follow: 1. It seems to me that some important details of the original ACT [1] halting mechanism are omitted in this manuscript or the proposed mechanism is a modified version, which should be clearly stated. I might be mistaken and only the formulation or notation is different. If so, please let me know. 2. Both $p_t$ and $P_{t'}$ are not really probabilities, as I understand it. You could maybe simply call $P_{t'}$ a quantity that, if it reaches a predefined threshold, halts computation. 3. $\phi$ in Equation 4 is not defined 4. It should be stated right after the first occurrence that LN stands for layer normalization. I think some details for reproducibility are missing or unclear: 5. The sentence in line 188 states that *four* choices are explored, then in 191 it is stated that *three* different recurrent implementations are explored. As far as I understood, the first of the four choices is a non-recurrent ResNet-30. It is not clearly explained how this model is implemented as Equation 2 only covers recurrent architectures. 6. The exact architectures including hyperparameters (e.g. number of kernels $d$ and their sizes) of the recurrent models are not provided. 7. Why is R-ResNet-30 equipped with ACT in Table 1, but not in Table2? I think the experimental setup should take the following into consideration: 8. For completeness, I think is would make sense to add a few more non-recurrent baselines. It seems intuitive that the considered tasks require multi-step reasoning, but it'd be important to show this empirically, e.g. by using a small vision transformer. I would be surprised if a transformer would completely fail on the task, given the fairly large number of training samples. I can, however, imagine that the proposed architecture performs much better on unseen difficulty levels. Showing this empirically would help in establishing AdRNNs as an important alternatives to current vision models. 9. Accuracies are not reported as mean and standard deviation over multiple runs with different random seeds. This is very problematic, since with some tasks and architectures, single runs may underperform or not converge to any meaningful solution at all, especially with binary success measures (such as exactly correct maze segmentation). Aggregating metrics over several runs (e.g. 5) would help the reader interpret how significant a different in performance really is (e.g. is it more than 1 standard deviation?) Regarding references: 10. I think the correct citation for *convolutional* GRUs would be [2] ## References 1. Graves, Alex. "Adaptive computation time for recurrent neural networks." arXiv preprint arXiv:1603.08983 (2016). 2. Ballas, Nicolas, et al. "Delving deeper into convolutional networks for learning video representations." arXiv preprint arXiv:1511.06432 (2015).

Questions

Overall, I think the proposed method is novel and has potential for impact, but given the issues mentioned in the weaknesses section, some more work is necessary for publication. I will consider to increase the score significantly should the issues be addressed during the author response period. I have some minor comments regarding the write-up: Line 63: starts upper-case after "and" in the previous item Line 100 and 105: Shouldn't it be "4)" and "5)" instead of "2)" and "3)"? Either way, it is a bit confusing. Equation 2: I'd explicitly state that $\mathbf{r}$ is the recurrent block and how $h_{-1}$ is defined. Also, I think it'd be good to explicitly state what $t$ and $t_\text{train}$ correspond to. Line 197: "in Supp. Info" should maybe be "in the Appendix" or "in the supplementary material" Line 266: "While ... whereas" ("while" or "whereas" should be removed) Line 340-343: I'd avoid nesting parentheses. ## Acknowledgement of rebuttals I have read the rebuttal and provided follow-up questions. The rebuttal addressed my concerns, except the addition of a strong non-recurrent baseline. I have accordingly increased the score slightly.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

4 excellent

Limitations

The authors state some limitations of the approach and propose directions for future research.

Summary

The primary subject of this paper is fostering computational efficiency in RNNs solving vision tasks by flexibly adapting the number of computing steps depending on the difficulty of the input. This is achieved through the addition of ACT (Adaptive Computation Time introduced by Graves, 2016) to recurrent vision models. The paper evaluates 4 models (3 with recurrence) on 2 visual reasoning tasks popular in the literature for studying RNN behavior (PathFinder and Mazes). One model is newly introduced here and given the name LocRNN. The key result put forward in the paper is that ACT-trained RNNs will indeed adaptively carry out more time steps before halting computation for difficult inputs (longer paths or larger mazes) than for easy inputs. Finally, another finding is that ACT-trained RNNs generalize to more difficult task conditions than those encountered during training.

Strengths

The paper touches upon a significant and topical challenge in deep learning: how to avoid ‘wasting’ computations on easy inputs while still solving the hard ones to achieve computational efficiency. Efforts in that direction are valuable from an ecological perspective, among other perspectives. Although the work doesn’t address this directly, mapping out which inputs are hard versus easy for a model can help acquire insights into the strategies a model has learned to solve a (visual) task. It is also of relevance for cognitive computational neuroscience and human-modal comparisons. The paper reads very smoothly. The questions are formulated clearly, and so are the methods. It is overall well written.

Weaknesses

It is good to see the prediction confirmed. Still, the finding that ACT will flexibly halt computation early for easier inputs does not tremendously advance the field, considering it’s what Graves (2016) already reported. With an ACT-like method (“DACT”) Eyzaguirre and Soto (2020) have already found this in visual reasoning tasks too. It’s worth a citation. Similarly, there have been other thorough demonstrations that vision RNNs can generalize to more difficult task conditions, even on the very same Pathfinder task. I encourage the authors to check out and cite Linsley et al. (2020) "Stable and expressive recurrent vision models". In particular, Fig. 3 in this current paper is explored in detail and analyzed in the Linsley (2020) article. The introduction of LocRNN then becomes the most novel aspect of the work, but there is not a lot of in-depth discussion in the main text about how it differs from prior models informed by neuroscience. At the very least, one would require a thorough comparison between the LocRNN and the model presented in Linsley et al. (2018) regarding their performance metrics and architectural similarities. Finally, can the authors comment on whether this approach would work in other scenarios of visual difficulty apart from simply spatial extent? The notion of visual complexity involves a host of other factors which a non-hierarchical model would fail to account for, and which needs discussion. Typos and minor fixes: Some of the citations appear to be in the wrong format (e.g., L40) L. 322: Fig.4 → Fig. 3? Fig. 3: in a revised version, probably good to discuss these results in the main text more, specifically with regards to the non-ACT controls. L 328: Fig. 4 has no panel c.

Questions

Please refer to the weaknesses section above.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Please refer to the weaknesses section above.

Summary

Authors show that using recurrent networks for adaptively processing static inputs for a variable number of iterations allows for zero-shot generalization to more difficult problems by simply unrolling the model predictions for more time steps at inference. Authors propose LocRNN and show model performance on the pathfinder and maze datasets, highlighting that their model uses a greater number of iterations for harder problems and can generalize to harder problems than those seen during training.

Strengths

Clarity of Writing. This was by far my favorite paper to read of my reviewing batch because of the clarity of writing. I was able to clearly understand the premise and motivation and follow along with the experiment setup and evaluation. The insight that recurrent networks can be used to scale up or down the amount of compute depending on the difficulty of the problem is well articulated. Compelling Results. Authors show that LocRNN beats competitive baselines on two standardized benchmarks. Impressively, Table 2 shows that LocRNN beats other baselines on extrapolation datasets by a wide margin.

Weaknesses

Motivation. Although I agree with the premise of the paper, I want to push back on using the biological inspiration of variable compute for neural networks. Although humans naturally can scale to harder and easier problems, state-of-the-art vision networks do not do this. Although authors cite efficiency as a reason for favoring RNNs for variable compute problems, no experimental evidence is provided. Limited Baselines. Similarly, although the focus of this paper is on adaptive compute, two strong baselines would be LocRNN trained with only one computational step and LocRNN trained with N computational steps (where N is some fixed number of maximum steps allowed). Given the significant engineering that went into the network architecture of LocRNN, the ResNet-30 baseline does not seem convincing to show the necessity of adaptive compute. Datasets. Although authors evaluate on two standard benchmarks, I would argue that these toy datasets do not represent real tasks. It would be more convincing to compete LocRNN against a state-of-the-art model on a more traditional computer vision benchmark (e.g. ImageNet, COCO, etc.) One place I think this approach could shine is in object detection. Given an image with a single near-field object, we should not need to spend a lot of compute. Similarly, given an image with many far-field small objects, we should likely spend more compute.

Questions

Intermediate Predictions. How do we know that the iteration that the model stops at is optimal? Can we evaluate the model prediction at intermediate steps and evaluate the model prediction beyond when it naturally stops? Architecture Diagram (Sec. 4.3). Providing an architecture diagram for LocRNN would help improve the clarity of Section 4.3. Please add this to the appendix.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

Authors highlight that their work explore this problem of adaptive compute on static images and will work on videos in future work. However, I would encourage authors to explore more complex problems within static image domains (e.g. classification, detection, segmentation of natural images instead of toy-like environments).

More about LocRNN architecture

### LocRNN architecture and why it works well With respect to LocRNN’s functioning, and its differences to other similar architectures such as ConvGRU, and hConvGRU (Linsley et al., 2018; Linsley et al., 2020), we have (1) Run experiments comparing these different architectures, (2) Described the origin of the dynamics of LocRNN in further detail, and (3) Qualitatively visualized the internals of LocRNN. We would like to highlight that the dynamics of LocRNN originate from a previously published model of cortical horizontal connections by Zhaoping Li (Li, 1998). The PDEs proposed in (Li, 1998) model (in continuous-time) the connections between V1 neurons. We take these PDEs, apply Euler integration to them in order to convert them into discrete-time difference equations which we then implement using convolutional Recurrent Neural Networks. The final RNN obtained from Li 1998 after Euler integration is what we refer to as LocRNN in our submission, and we contrast the architectures between LocRNN and GRU in Fig 8.

Reply to the Authors

We thank the authors for having included an illustration of LocRNN in the rebuttal pdf, and having clarified the origins of the architecture. I consider these points necessary for my score to remain a tentative accept.

Some items addressed, but

I have read all reviews, author responses, and considered the rebuttal pdf. I acknowledge that the authors addressed some of the issues mentioned in the weaknesses section of my review. However, since the authors only reported results of single runs (see item 9 in my review), it is difficult to judge how significant the differences between models are. Also, as mentioned in item 8, I think the consideration of additional strong non-recurrent baselines would've been important (ViTs were just an example). I will increase the score slightly to a borderline accept.

Reply to author rebuttal

Thank you the responses and the additional experiments. >"I am curious to hear why the authors say RNNs used in Bansal et al., (2022) "are not adaptive, human intervention is required to specify the number of recurrent computational steps by brute force during the testing phase" since the RNN can be run till response stops changing" > **Response:** As this is an important issue relevant to several reviews, we have addressed this general issue in our common response. We also tried the Linsley et al 2020 algorithm with a threshold on state change and it performed very poorly on Mazes (see Table 3 in the Rebuttal). I understand that there are downsides to Bansal et al. (2022) and Linsley et al. (2020) but that still doesn't mean that they are not adaptive. I would encourage updating the manuscript with the downsides and remove the claim that they are not adaptive. > "What properties of the images makes the RNNs stop early vs late? For example, in Fig 4 some 19x19 mazes seem to need the timesteps as 25x25 mazes. Have you seen what causes this? Same for the other task. Can this lead to some interpretation/understanding of the RNNs and the solution they are implementing?" > **Response:** The plot in Fig 4 plots halting step as a function of maze solution path length. There is a clear linear relationship between solution path length (not maze size) for the shorter path lengths. For longer maze path lengths, the required computation saturates. As you note, the RNN seems to have captured a factor beyond path lengths. For me it would be interesting if you used your adaptive timestep method to provide interpretations of an RNN - which I think is novel and beyond what was already done by Bansal et al. (2022) and Linsley et al. (2020). For me, this would push the paper above the threshold for publication. As it stands, given that ACT is a known method and Bansal et al. (2022) & Linsley et al. (2020) have studied adaptive time for Mazes & Pathfinder respectively, I do not see the novelty in simply introducing ACT for Mazes and Pathfinder. From reading the responses and the discussion, it seems to me that the main contribution is in fact the new RNN architecture (LocRNN) and not the training algorithm. However, it is still unclear to me why LocRNN is better than convGRU, hconvGRU, CORNet-S etc. Like the LocRNN model, hconvGRU and CORNet-S are also neuroscience-inspired. So, what is the critical novel circuitry in LocRNN that enables such great performance? For example, LocRNN has LayerNorm while convGRU, hconvGRU and CORNet-S don't seem to - how critical is this for superior performance of LocRNN? LocRNN has separate inhibitory and excitatory populations (which could explain superiority to vanilla convGRU) but so does hconvGRU - so where exactly does the improvement come from? I think a systematic analyses where the LocRNN's components are lesioned and explicitly contrasted to other neuroscience-inspired architecture is necessary to gauge the contributions of LocRNN architecture. To summarize : 1. **Limited advancement in adaptive computation compared to previous methods**: Although a previous method was added during rebuttal, I am skeptical of the results since it was done in a very short time-frame and some experiments are missing. Bansal et al. (2022) showed strong generalization for the maze task and Linsley et al. (2020) for the pathfinder task - I am skeptical without more experiments that they won't work for each other. I think more extensive evaluation is needed before this method can be concluded to be better - `all three methods {ACT, Bansal et al. (2022), Linsley et al. (2020)} X all models {LocRNN, convGRU, hconvGRU, CORNet-S} X all tasks {maze, pathfinder}`. As it is currently, we do not know where the improvements are coming from. 2. **Interesting model but unclear where the contribution is**: LocRNN model is interesting and seems to be bringing in performance improvements. But since it is similar to previously known models, it is unclear as to what advancement in the model is causing this improvement.

LocRNN contrasted w/ (h)ConvGRU - more discussion and additional random seeds

Dear Reviewers, We thank you for your quick responses to our first rebuttal. Following up on Reviewer 1Kd6’s request, we report the performance of both LocRNN and ConvGRU (which are the two main models of comparison in our work) across 3 random initializations. We thank the reviewer for raising this issue, we find that (1) LocRNN is consistently better performing than ConvGRU and (2) LocRNN’s performance across random seeds is more reliable with lesser variance when compared to ConvGRU. ### Why LocRNN works better than baselines Our initial exploration of this direction so far has suggested the following for LocRNN’s improved performance and the importance of its various components: We have found that the most important component of LocRNN is the incorporation of “hidden” (S) interneurons (not directly connected to the output readout) absent in simpler RNNs like GRU. Theoretically, we are motivated in this interpretation by the importance of hidden neurons in deep neural networks and the difference in performance between Jordan (output feedback) and Elman nets (hidden state feedback). Empirically, removing this feature from LocRNN results in reduced performance, which we hypothesize is why LocRNN performs better than ConvGRU. On a more specific note, Fig 7 in the supplementary rebuttal pdf shows state activations of the L (projection) population for a PathFinder example showing how the L neurons perform contour integration; we have similarly inspected the S (interneuron) population which have clearly different activations but are not as easy to interpret. These interneuron-analogues appear to be encoding complementary information that we are continuing to investigate. \textit{To explain the difference between LocRNN and the more complex hConvGRU we believe the following are critical.} LocRNN explicitly uses two separate populations of **laterally-connected** neurons inspired by cortical evidence of different horizontal connection patterns in V1 between interneuron and projection neurons. hConvGRU’s hidden state is split into H1[t] and H2[t], however, they are **hierarchically organized** unlike LocRNN (which is derived from Zhaoping Li’s model of contour integration in V1 where the two populations are laterally-connected). Extra connectivity and gating connections beyond those required to have separate interneuron and projection neurons appear to result in poorer performance due to the increased (potentially unnecessary) complexity In summary, LocRNN uses a unique implementation of lateral connections that is more faithful to biological projection and inter- neurons. In summary, our working hypothesis is that our network forms something like a “smallest necessary core” for good performance in these extrapolation tasks. As this is a critical question for RNNs in general, we will be further exploring these ideas in future work. | **Model** | **PathFinder-21** | **PathFinder-24** | **Mazes-19** | **Mazes-25** | |-----------------|-------------------|-------------------|------------------|------------------| | LocRNN (ACT) | **92.89 +- 0.90** | **85.81 +- 5.57** |**86.83 +- 2.94** | **49.99 +- 4.48** | | ConvGRU (ACT) | 82.63 +- 4.84 | 74.14 +- 6.52 | 75.1 +- 11.96 | 46.93 +- 4.2 | We will add these new results to our updated manuscript. NOTE: - All numbers above represent mean +- 1 SEM - Chance performance for PathFinder is 50%, whereas it is close to 0% for Mazes (where our performance metric measures the % of test-set mazes perfectly segmented)

Author discussion period is closing soon

Dear Reviewer, We would like to check if you have any concerns following our rebuttal and follow-ups that we may address during the author discussion period which ends on Aug 21st at 1 pm EDT. Thank you.