Inverse decision-making using neural amortized Bayesian actors

Thank you for taking the time to review our paper from the broader perspective of machine learning in general. We have the feeling that the key idea and concept of our method in the context of computational cognitive science, computational economics, and computational neuroscience has not come across, unfortunately. We will respond to each of your comments and questions below in more detail. **[Lack of empirical advancements]** We understand the concern that empirical advancements are an important objective, but we want to stress several points here: 1. First, we would like to direct the reviewer’s attention to the empirical results depicted in Figure 5. These results provide evidence that much previous research on cost functions in sensorimotor control has relied on oversimplifying assumptions, that task-dependent cost functions have much more structure than previously assumed, and that subjective internal effort costs may grow neither linearly nor quadratically. 2. While we do have results regarding human behavior, see the previous point 1), we are proposing a new method in this paper. We carefully validated the method and applied it to empirical data, see the above point. In our view, the emphasis of our contribution lies in the development of the method and completely novel empirical insights in a scientific domain such as sensorimotor control are beyond the scope of the current paper. However, our method should lead to new empirical insights from experiments in cognitive science and neuroscience. 3. Moreover, there is empirical advancement from a methodological perspective. By uncovering identifiability issues and how they can be overcome, we provide experimentalists with guidelines on how to design experiments such that they can extract reasonable results from their data (Section 4.3, Figure 4). This is especially important since, e.g., many experiments investigating sensorimotor or economic actions to date do not incorporate different levels of perceptual uncertainty, which we identified as a requirement for identifiability between costs and priors as confounding behavioral parameters. **[Behavioral dataset is too simple]** We understand the concern that the behavioral experiments to which we applied our method might appear simple, since each trial involves subjects perceiving a single variable and performing a single response. However, the cognitive processes involved are by no means simple. Historically, experimental designs are largely limited to discrete actions, even though natural behavior is continuous. Once we move beyond discrete actions, additional cognitive processes such as action costs and motor variability play a role. There are no established state-of-the-art tools for inferring what drives human behavior in these more natural tasks, and we are confident that our method can contribute to bridging this gap. The relevance of such tasks e.g. in neuroscience has been put forward in recent reviews that have called for new experiments, to which our method should be applicable (Yoo, Hayden, Pearson, 2021). **[Challenge of high-dimensional models]** With all due respect, we do not understand where the idea arises that we would want to perform inference over models in the order of 100M parameters. Could the reviewer please clarify? We clearly state in the paper that we are concerned with models from cognitive science, neuroscience, and behavioral economics. Models in this domain typically have few parameters, usually within the range of 2–12 parameters. These models gain their strength not by the number of parameters, but instead by the interpretability of the parameters as perceptual, cognitive, or motor factors. Importantly, it is desirable to be able to manipulate experimental variables to test and verify the influences of parameters on behavior. All related studies that we mention in Section 2 as well as all examples referenced in the discussion or used in our evaluations only have a few parameters. More concretely, Bayesian models of perception often have a few parameters exactly like the model we consider in our evaluations: perceptual uncertainty, prior beliefs, behavioral costs, and motor variability. To give a few examples, some models have two free parameters (Petzschner & Glasauer, 2011; Neupärtl et al., 2021). Even more complex Bayesian models of perception only have up to ten (Battaglia et al., 2011) or twelve (Acerbi et al., 2014) free parameters.

**[HMC might not have the property of rapid mixing]** This is a very general concern about HMC, which, in our opinion, is misdirected since such general criticism could be applied to any study that uses this method. The main focus of our paper is on using neural networks to amortize Bayesian decision-making problems. This enables the use of any gradient-based sampler for the inverse decision-making problem. In particular, our method does not require the use of HMC, but can make use of any state-of-the-art sampler that uses the log posterior probability (and its gradient). Nevertheless, we understand concerns about MCMC convergence. To check the convergence of our Markov chains, we verified that the $\hat{r}$ statistic was below 1.05 in all cases. Additional references: - Yoo, S. B. M., Hayden, B. Y., & Pearson, J. M. (2021). Continuous decisions. Philosophical Transactions of the Royal Society B, 376(1819), 20190664. - Petzschner, F. H., & Glasauer, S. (2011). Iterative Bayesian Estimation as an Explanation for Range and Regression Effects: A Study on Human Path Integration. The Journal of Neuroscience, 31(47), 17220–17229. - Battaglia, P. W., Kersten, D., & Schrater, P. R. (2011). How Haptic Size Sensations Improve Distance Perception. PLoS Computational Biology, 7(6), e1002080. - Neupärtl, N., Tatai, F., & Rothkopf, C. A. (2021). Naturalistic embodied interactions elicit intuitive physical behaviour in accordance with Newtonian physics. Cognitive Neuropsychology, 38(7–8), 440–454. - Acerbi, L., Vijayakumar, S., & Wolpert, D. M. (2014). On the origins of suboptimality in human probabilistic inference. PLoS computational biology, 10(6), e1003661.]

Thank you for taking the time to review our paper. We appreciate your feedback and will go through it one by one in the following. **[Extension to different tasks and overall applicability]** We understand your concerns about the applicability of our method to other domains. However, even if the method were only applicable to sensorimotor tasks, this would, in our view, be a major contribution. The reason is that ample research has investigated human sensorimotor control using a few canonical cost functions (e.g. Kording & Wolpert, 2004) and here we extend the cost functions that can be considered considerably allowing for parameterized effort costs, which is e.g. implicitly common in resource-rational approaches. A recent (Sohn & Jazayeri, 2021) publication also noted the difficulty of such inferences, for which the present manuscript proposes a resolution. But, beyond this broad applicability, we are confident in our model’s generalizability, because the formal setting’s only constraint is that the agent has uncertainty over a latent variable and performs an action associated with a cost. This is also the case, for example, in behavioral economics, where models with “cognitive uncertainty” have seen a recent surge in popularity (e.g. Khaw et al., 2021; Enke & Graeber, 2023; Barretto-Garcia et al., 2023). In these models, the log-normal assumption we use to model the likelihood of the sensory measurement $m$ has also become increasingly common (Khaw et al., 2021; Enke & Graeber, 2023). This assumption will also generalize to other domains and modalities where Fechner’s Law holds, e.g. numerosity (internal cognitive representations), distance and size perception (visual), pitch perception (auditory), etc. In cases where the log-normal assumption does not hold, e.g. circular variables, the log-normal assumption can be dropped in the perceptual component and our method could still be used with any other distribution, as long as there is a closed form of the subject’s posterior belief about the stimulus $p(s | m)$. For circular variables, this could be the Von Mises distribution, for which conjugate priors (Guttorp & Lockhart, 1988) exist. Our method could then be used to perform model comparison over different perceptual distributions and cost functions. Of course, we are also interested in cases where there is no analytical expression for the posterior distribution, and we have addressed this as future work in the discussion section. It would be helpful if you could clarify what you mean by “advanced cognitive reasoning tasks”. We would be happy to discuss relevant models and how they relate to the method we are proposing. Additional references: Guttorp, P., & Lockhart, R. A. (1988). Finding the location of a signal: A Bayesian analysis. Journal of the American Statistical Association, 83(402), 322-330. **[Scalability]** This is an important point, thank you for the question. The amount of training required will likely scale with the number of parameters. While the scalability of the network is an empirical question, we are confident that the training should not be too computationally expensive with more parameters. Currently, training of a network takes less than 30 minutes and converges quickly on the CPU of a standard laptop computer (e.g. with Intel Core i7-8565U CPU). The network architecture is rather small and could easily be extended to deeper and more expressive networks. Thus, in terms of architecture and training time, we get excellent results while still being multiple orders of magnitude below the computational cost of most modern machine learning applications. For the kinds of models we have in mind, the number of parameters should not increase too much. Additional parameters could be introduced in the cost function, but the number of parameters should remain rather low to guarantee explainability. For more complex stimuli (e.g. multidimensional stimuli like color spaces), perceptual priors and cost functions with as many parameters as there are dimensions could be possible. If there are concrete “more complex cognitive reasoning tasks” you have in mind, we would be happy to discuss those.

**[Clarification of network architecture and hyperparameters]** This is all provided in the paper. Section 3.1.1, 3.1.2 and 3.3 specify the training procedure, architecture and implementation details, respectively. Appendix A provides pseudocode for training the neural network, Appendix D.1 specifies the distributions for generating training data and Appendix D.3 explains an inductive bias used in the output layer. In summary, we use four hidden layers with [16, 64, 16, 8] nodes and swish activation functions. We train the network in an unsupervised fashion directly on the cost function by sampling random parameter configurations for 500,000 steps. We use the RMSProp optimizer with a learning rate of $10^{-4}$ and a batch size of 256. Our reasoning for these choices was that they were sufficient for good performance on the models we investigated in this paper. Of course, future work could investigate different architectures and training regimes in terms of data efficiency or computational cost. One could even think about network architecture search, but this is beyond the scope of the current paper.

Thank you very much for your detailed suggestions on how our work can be improved. We appreciate a lot and will respond to each point separately below. **[Optimality Assumption]** We respectfully would like to disagree with this point. Yes, we agree that the optimality assumption in forward modeling, i.e., in models from the perspective of the researcher, such as ideal observer, ideal actor, and ideal learner, is a very common one in perceptual decision-making, economic decision-making, motor control, and other cognitive tasks. We also fully agree that people are not always optimally tuned to the statistics of the task at hand, or act to only fulfill the instructed task. However, this is precisely the reason why we have developed methods for inverse decision-making to infer under which assumptions the behavior would have been optimal! Without revealing our identity, we have contributed to inverse modeling over the last twenty years precisely to address “real-world scenarios”. Using such methods, we could show that, indeed, human participants can have false beliefs and idiosyncratic subjective costs, which lead to systematic deviations from “optimal” behavior. Thus, the “suboptimality” can be explained in the inverse modeling framework. Therefore, optimality is not a weakness, but allows inferring the subjective parameters of behavior. We agree that the formulation of behavior as being rational needs careful modeling of the possible influences of individual subjects’ behavior, but inverse models in general, and the one we have developed here, make it possible to investigate the reasons for the deviations from “optimality”. We take this comment by the reviewer as motivation to extend the discussion of this point in the revised version of the manuscript and will mention relevant literature on the optimality versus suboptimality debate, e.g. in perceptual decision-making (Rahnev & Denison, 2018). **[Reparameterization Trick]** For all models with continuous responses we are aware of, and for all related work cited in the paper, exponential family or other distributions amenable to the reparameterization trick should work. In our view, this covers an enormous amount of behavioral experiments in psychology, cognitive science, and neuroscience, including motor control. Thus, the developed methods should be exceedingly useful to the scientific community. If there are relevant models we have missed, it would be great if you could please name examples from the literature. In such cases, recently developed methods for computing gradients of more general probabilistic programs without the reparameterization trick will be useful to compute the gradient of the expected loss (e.g. Lew et al. 2023; https://dl.acm.org/doi/abs/10.1145/3571198). **[Overall presentation & figures]** Thank you for raising the questions about our figures and labels below. We respond to them one by one and will revise our paper accordingly. We have thoroughly revised the figures also with the points raised below. *[[Figure 2A - Posterior Predictive]]* Sorry about the confusion in the legend of Fig. 2A. The shaded area is a posterior predictive distribution over the responses performed by the subject, given their actually observed responses. We will rename it as $p(r_{pred} \mid s, r)$ in the revised version. *[[Figure 2B - Missing label]]* Yes, sorry for omitting the label! This is the marginal distribution of $\sigma_0$. The y-axis will be labeled $p(\sigma_0)$. We will adjust all labels of marginal distributions accordingly. *[[Figure 2C - x-axis labels]]* Thank you, we provided separate labels on the x-axis for the analytical and the NN model, but recognize that this is hard to see. We will separate the two labels “nn” and “analytical” more clearly to avoid confusion. *[[Figure 3B - Asymmetry Parameter]]* Yes, thank you for catching this! And sorry, it should indeed have been $\alpha$ also in the caption. *[[Figures 3B, 3C, 4]]* Thank you. These are good suggestions, which we will use in the revised version.