Summary
This paper introduces a new measure of visual artificial network computation 'time', $\xi_{cRNN}$, as the time-averaged uncertainty of a convolutional RNN trained with an evidential deep learning loss (Sensoy et al.). The authors proceed to analyse the dynamics of this network as it solves a range of classification tasks. Importantly, beyond the network learning to solve the tasks, $\xi_{cRNN}$ is generally well correlated with human reaction times, with which it also shares several qualitative features across tasks.
Strengths
Deep network models of the visual system have often focused purely on classification accuracy and neural representations when comparing to biological data. However, this neglects an important aspect of human behaviour, namely the time it takes to arrive at a decision. The present paper tackles this important question using recent ideas from the machine learning literature and represents an interesting step in the direction of capturing the temporal variability in human behaviour.
The authors analyse an impressive breadth of tasks and behavioural data and include many interesting analyses, both qualitative and quantitative.
Weaknesses
The major weakness of the submission is that while there are strong _correlations_ between $\xi_{cRNN}$ and human reaction times, there is less evidence that $\xi_{cRNN}$ is mechanistically similar to human reaction times, since it is fundamentally a measure of uncertainty rather than computation time. In particular, human reaction times generally involve a tradeoff between computation/evidence accumulation and decision making (as in common drift diffusion models). On the contrary, the cRNN has a fixed computational budget and has no need or even capacity for evaluating this tradeoff. This is in contrast to a few previous deep learning models in the literature that are capable of explicitly trading off computation and actions (e.g. Graves et al., Pascanu et al., and refs by the authors in the ML literature, and Jensen et al. in the Neuro/Cogsci literature). These considerations are important because e.g. task difficulty is likely to correlate with both uncertainty and reaction time, and this raises the question of whether a model of one is automatically a model of the other.
It might the interesting to compare the current model to alternative models that more explicitly have adaptive/variable computation time, such as a cRNN that computes until $\epsilon$ reaches a certain threshold (akin to classical drift diffusion models).
As the authors mention in L326, another potential weakness is that the present approach does not easily generalize beyond classification tasks, which form a small subset of the types of problems humans and animals are faced with during natural behaviour.
_References:_ \
Graves et al.: "Adaptive computation time for recurrent neural networks", arXiv 2016.\
Pascanu et al.: "Learning model-based planning from scratch", arXiv 2017.\
Jensen et al.: "A recurrent network model of planning explains hippocampal replay and human behavior", bioRxiv 2023.\
Bansal et al.: "End-to-end Algorithm Synthesis with Recurrent Networks: Extrapolation without Overthinking", NeurIPS 2022.
Questions
L106: what does 'without loss of generality' mean here?
L123 "our control models lack inference-time flexibility." What does this mean?
L144: It might be worth comparing to or discussing recent work by Bansal et al. (NeurIPS 2022), which develops a way of stably training convolutional RNNs with long computation times that generalize from small to large problem settings.
L156 "for a complete mathematical specification of this loss, refer to SI 1". $\epsilon$ does not seem to be defined in either the main text or SI beyond a qualitative description as 'uncertainty'. It would be good to provide a precise mathematical definition somewhere.
L204: it would be good to briefly explain how this distance measure by Jeurissen et al. is defined (or at least refer to SI where it can be described in more detail).
Figure 3b and 5c: it's not entirely clear to me how these 'activity maps' are computed?
Figure 7b: it might be worth talking a bit more about the differences between the human and model data here instead of doing linear regression and calling it a day. The cRNN data is decidedly not linear and exhibits something resembling a plateau followed by a sharp drop at high discriminability. This seems rather different from the monotonic decrease in human RT, and the authors could perhaps speculate on or discuss this difference.
Supplementary material: it would be worth having a separate folder with just a few example videos of each condition in addition to the larger set of videos currently provided.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
The authors have adequately discussed the limitations of the work.