Probabilistic inverse optimal control for non-linear partially observable systems disentangles perceptual uncertainty and behavioral costs

Summary

This paper proposes an algorithm for inverse optimal control for agents operating in a partially observable Markov decision process, using only state trajectories. Given state trajectory data, dynamics model and observation model, the algorithm estimates the parameters by three steps: first, a policy is estimated using iLQG; second, the belief dynamics is estimated with EKF; finally, filtering is done by linearizing belief propagation. The likelihood is then maximized by back-propagation. The authors tested the proposed algorithm on several synthetic problems and compare it with a baseline based on maximum causal entropy. The results show that the proposed approach can better estimate the unknown parameters from behaviors than the baseline.

Strengths

The problem studied here is relevant to community. I think especially that the authors consider the case without action information, which makes the setup more practical. The writing of the paper is easy to follow. The introduction is well motivated and the related work discussion is informative. The proposed idea is clean and is new to my knowledge. The authors discuss in details about the limitation of this work.

Weaknesses

While the writing is easy to follow, it is unclear what the assumption is being made to use the proposed method. From Algorithm 1, the authors assume knowledge of the dynamics, observation models, and state trajectory, but the steps of Algorithm 1 requires more information than that, e.g., the cost is needed in iLQG. In addition, it is also unclear how the proposed algorithm actually operates; the authors describes how filtering and approximate likelihood (given all the parameters) is computed in details, but do not mention how the unknown parameters are actually estimated (the authors mention "using gradient-based optimization" with automatic differentiation, but it is unclear what the computational graph entails). As a result, while I get the high level idea, I do not think I fully understand the working details of the proposed scheme, and therefore it is hard to give it an accurate evaluation. I think the authors should compare more with recent IOC or IRL works, in the experiments, and consider more realistic datasets (rather than synthetic ones). In addition, it would be more informative if the authors can better highlight their contribution in terms of the specific problem difficulties here (e.g., due to missing action). Lastly, the proposed method is limited to low dim problems as the authors also point out.

Questions

1. What is the assumption of known information made in the paper? It is not clearly specified in the current writing. From Algorithm 1, it seems that assuming only the dynamics and observation models, as well as state data trajectories. But in iLQG the cost is also needed, and in the EKF step the belief dynamics (i.e., beta_t) is needed. What are they in the experiments? The algorithm also uses the prior of belief. Is that assumed to be given too? 2. How does the learning actually work? The paper mentions the usage of automatic differentiation. But how is that actually done and what is the computation graph involved? Algorithm 1 requires multiple linearization steps in iLQG, EKF, and the filtering. In particular, iLQG is an iterative process, do you also backpropagate through them? 3. While the paper supposes only state trajectories are given, it seems to make assumption on the knowledge of the belief dynamics, or what the belief of the agent is. It is not clear whether this is a stronger or a weaker assumption. 4. The current experiments only consider rather small and synthetic problems. Can you include results on some more realistic datasets? In addition, what are the agents that generate the data in the experiment (I can only tell iLQG is used for the light-dark domain)? It would be also good to test the proposed algorithm the estimation of non-iLQG agents, since the proposed one is based on iLQG, so that we can know the generality of the proposal.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

Yes, the authors made clear what most limitations are. It would be also good to point out the limitation due to limited empirical evaluations done in the paper (e.g., due to the synthetic, low-dim nature, the agent policy's form, the parametric structure, etc.)

Summary

The paper introduces a new approach to inverse optimal control which is able to deal with partially observable systems in which action signals are not known. Most existing approaches only work in fully observable systems where actions are known. The paper introduces a probabilistic formulation for inverse optimal control and uses maximum likelihood to estimate the costs and parameters of the system. To make the likelihood tractable, they use local linearization similar to iLQG. The paper tests the approach on 4 tasks and shows improved performance over a maximum causal entropy-based baseline.

Strengths

I am not familiar with this area of research and most of the concepts in the paper are new to me but I was still able to follow most of concepts introduced in the paper hence I think the writing is mostly coherent and fairly easy to follow.

Weaknesses

I don't have any weaknesses to point, I mainly have questions which I will write below.

Questions

1) My first question is about baselines. The authors use a maximum causal entropy-based baseline for the experiments. From my understanding, maximum causal entropy is mainly used for exploration in RL but it is not clear to me what is the IOC-based method used in MCE to estimate the cost function. Does it use something like ILQG or EKF? 2) What functions are used to model f and \beta in equation 2? From my understanding, the functional form should be known beforehand which would be quite limiting for a setting where this information is not known.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

1: Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have discussed all relevant limitations.

Summary

The authors propose an inverse optimal control method to handle the challenging case of inferring an internal model in a non-linear partially observable system, when the action sequence is not observed. Quantitative evaluation of the proposed method was shown for some classic control problems. The authors also demonstrated through an example the potential of their method in disentangling perceptual factors and behavioral costs.

Strengths

The written language is easy to understand, and well organized. This work is a novel combination of ideas from some well-established techniques. The method should be of particular interest since intended actions of other agents are often only partially observable.

 The value of the method is highlighted by a comparison with an alternative method that mistakenly attributes perceptual uncertainty to subjective preferences. The authors’ IOC is able to correctly infer that much of the action in this task is driven by exploration rather than exploitation.

Weaknesses

It would be helpful to highlight more strongly that they can distinguish preferences from uncertainty. The authors rightly mention this virtue in the title, so I think it deserves more attention in the text. On the other hand, for that section, I would say that stronger evidence is needed, for example on a more complex task than the light and dark one. Uncertainty and cost are sometimes nonidentifiable: It could be too costly to correct a certain mistake, or it could be too uncertain to justify an optimal action, and these effects can be indistinguishable from the outside. However, when the uncertainty is dynamic while the cost function is static, these factors are dissociable. This should be addressed by the authors.

Questions

The authors describe their method as uniquely accounting for unobserved actions. But other Inverse Optimal Control methods allow for stochastic policies [e.g. Wu, Kwon et al 2020], in which actions can equivalently be interpreted as a latent intended action distorted by noise, or as a draw from a known policy. Aren’t these equivalent if the policy is optimized with action noise? The authors should show a sanity check that their linearization successfully allows us to estimate actions near enough the ground truth. It would be helpful to discuss more of the limitations from combining iLQG with EKF. Do the authors allow for controls that depend on a dynamic covariance, or just estimates of the mean? This was a bit unclear from their writing. It’s likely, given the techniques, that their method does not account for controllable uncertainty. Is this correct? I would like to see more evidence for the main claim, as suggested in the title, that the method disentangles perceptual uncertainty and behavioral costs. To show that taking into account the agent’s beliefs helps, it would be helpful to compute how uncertain the internal belief is while the agent moves towards the light before turning towards target, and plotting how it changes with subjective reward parameter c.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have a thoughtful discussion of limitations.

Summary

This paper targets for solving the inverse optimal control problem for partially-observable stochastic non-linear dynamics with no observation of the action. To estimate the parameters of the cost function in a stochastic non-linear system, the author first derives a likelihood function for the model parameters. They then approximate this likelihood by locally linearizing the system using a combination of iterative linear quadratic Gaussian (iLQG) and Extended Kalman filter (EKF) techniques. The proposed inverse method makes explicit assumptions about the dynamics of the control tasks, the agent's belief in a stochastic environment, and the structure of the cost function. The authors evaluate their approach on four simulation environments where structured noise is added to the dynamics of the environment to introduce stochasticity into the system dynamics. In comparison to the baseline, the proposed approach demonstrates better uncertainty estimation.

Strengths

In general, I find this paper easy to follow and variables in the equations are well defined. The derivations are nicely structured. The details are well documented in the Appendix and the assumptions and limitations are explicitly considered.

Weaknesses

I suggest the author clearly define their contribution in this paper, as the likelihood formulation of the inverse optimal control problem has been derived in another work, \textit{Inverse Optimal Control Adapted to the Noise Characteristics of the Human Sensorimotor System}. To enhance clarity, I recommend placing the likelihood formulation in the background section. Claiming that their approach can disentangle perceptual uncertainty and behavior costs might be too bold for two reasons: firstly, the author demonstrates this claim in a single simulation environment only, and secondly, the structure of perception uncertainty and behavior cost function is known. However, in reality, obtaining the structure of these elements is difficult. My major concern pertains to the number of assumptions made in different aspects of the system to obtain the proposed closed-form solution. For instance, all noises added in the experiments are assumed to be Gaussian distributed. These assumptions render the proposed approach challenging to apply in real-world scenarios, and the experiments were solely conducted in simulations. In addition, potential solutions to these assumptions i.e. addressing multi-modals belief utilizing SMC, are only briefly mentioned in the conclusion section.

Questions

In the reaching task and the navigation task, why is the goal position not inferred? In the conclusion, the authors mentioned that their method may not be effective for high-dimensional parameter spaces. It would be beneficial for the community to know the accuracy of the uncertainty estimation as the number and dimension of parameters scale. In the sensorimotor domain, do you always have a well-structured cost function that you can obtain beforehand? I.e. I think your approach would not work if the formulation of the behavior cost function like eq.8 is not given.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Limitations are well documented in the paper.

Summary

This paper introduces a probabilistic approach to inverse optimal control for partially-observable stochastic non-linear systems with unobserved action signals. It derives an approximate likelihood function for the model parameters by linearizing the system around the observed trajectories and tracking the agent's belief distribution. This method is demonstrated to accurately infer the parameters better than the baseline maximum causal entropy (MCE) approach on two classic control tasks and two human behavioral tasks. Additionally, it shows the ability to disentangle the influences of perceptual uncertainty and behavioral costs on information-seeking behavior sources of information-seeking behavior.

Strengths

- The introduction gives a clear background, which is helpful for someone like me who is less familiar with this field. It provides a clear understanding of the research question and the existing approaches. - This paper proposes a new probabilistic approach for inverse optimal control in stochastic non-linear systems with missing control signals and partial observability, which outperforms the baseline MCE model and also provides an interpretable representation of the parameter space. - The proposed method is evaluated on different tasks, including two human behavioral tasks, which might inspire further investigation on how the method can be applied to neuroscience and cognitive science studies.

Weaknesses

A more comprehensive investigation into the method's robustness under various conditions would strengthen the paper's contributions.

Questions

Why is it necessary to assess the proposed method's performance on pendulum and cart pole tasks, despite their deterministic nature and lack of partial observability?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The authors have adequately acknowledged and addressed the limitations of their methods, which include the following: (1) The focus on tasks that can be well-solved by control methods based on linearization and Gaussian approximation (iLQG and EKF), which may not fully capture the complexity of naturalistic behavior, and (2) the concern that the method might not scale effectively to high-dimensional parameter spaces as optimization in a high-dimensional non-linear space can potentially lead to getting stuck in local minima.

Summary

This paper is a strong contribution on the Inverse Optimal Control problem when actions cannot be observed. It is particularly interesting their modelling of the agent and the “researcher” observer. The mathematical depth is good and sound. The results may be enough for a theoretical paper. However, there are too many unknowns to be ready for acceptance, particularly some definitions like partial observability related to noise. Another issue is the clarity regarding the baseline used, is it self-programmed?, is it taken from previous literature?. It seems that it does not work for any of the problems tested.

Strengths

- The mathematical depth is good. - Inverse optimal control plus noise estimation is relevant for many applications; especially for motor control research. - Results may be enough for a theoretical paper.

Weaknesses

- Narrative: While the contribution looks very promising, the explanation of the contribution in the introduction is not enough clear to understand what is the main focus of the paper. Furthermore, there is a need to jump forward and backward in the text to fully understand the details of the approach. Particularly, it is sometimes complicated to see if the authors are talking about the agent or the observer (“researcher”). Another example: I had to read the whole article to understand the title. - The partial-observability may be controversial as it is defined. The idea of tracking the belief of the agent is interesting, but why not using directly the observed state? So then is it the noise that the algorithm is tackling as partial observability. Thus, then the complexity depends on the type of noise. - Following the previous comment, while the method may be used for any parameter estimation (as authors state) why only estimating the observation/motor noise? Btw, observation noise of the agent or the observer? From the text I assumed that the authors were doing system identification at the same time. But they are estimating the noise. - Baseline explanation needs further description for the sake of clarity.

Questions

- The description of the variables v_t and w_t could be improved - Why there is a final cost that is independent on the action taken? Conversely to the previous time steps. - What is πt(xt) ? (section 2.2) - Why stochastic policy has a capital Pi? - Not clear why the EKF is introduced. - “In the inverse optimal control problem, the goal is to estimate parameters θ ∈ Rp of the agent’s optimal control problem given the model and trajectory data. These parameters can include properties of the agent’s cost function, the sensory and control systems of the agent, or the system’s dynamics.” This is a definition proposed by the authors. Originally, IOC is to recover the cost function from expert demonstrations. Getting system parameters is called system identification. - “cost of final velocity cv” Is this related to the cost definition with the final cost at end time? - “observation noise” of what quantity? Experimenter, agent? - “Note that past methods based on MCE are usually limited to estimating cost functions, so that parameters such as the agent’s noise therefore cannot be inferred.” This sentence helps a lot to understand the authors approach. - It is not clear why the dynamics of the agent state and its beliefs only depends on the state and the belief: “closed-form expression for p(xt+1, bt+1 | xt, bt).” - “we applied a baseline method based on the maximum causal entropy (MCE) approach [3].” Which method? Self-programmed or taken from the literature? Baseline text could be improved. In the end I am not sure how many baselines are being used. - “expressed as a non-quadratic cost function of the joint angles” Is this really true. Besides, while I understand that there is not enough space for everything on the paper, the description of the input output system would be nice to place it in the main paper so there is no need to go to the appendix to read that the arm is controlled with torque. - It is not clear why the baseline cannot even recover the action cost properly. Would it work in the absence of noise? - The agent navigation dynamics can be simplified to use the turn rate as the control action: [dot{x}, dot{y}, dot{theta}]¨= [cos(theta)w, sin(theta)w, u]

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

2 fair

Contribution

3 good

Limitations

Limitations are sufficiently addressed. My only constructive suggestion is to mention that noise covariance estimation for non white-noise sources is a very important problem in control. So I would include it in the discussion.

Summary

This paper presents the new formalization of inverse optimal control on partially-overevable Markov decision processes. The authors argue that most existing works on inverse control or inverse reinforcement learning focus on fully-observable Markov decision processes. Their approach extends iterative linear quadratic Gaussian (iLQG) and Maximum causal entropy (MCE) reinforcement learning, introducing local mineralization to achieve tractable likelihood. The method is evaluated through simulations.

Strengths

+ The formulation and derivation of the method are solid and well-motivated. *Weaknesses *Limitations

Weaknesses

- The derived method seems natural, making it challenging to identify the novelty of the proposal. - The evaluation lacks sufficient quantitative and qualitative analysis, as it only covers simple settings and compares with the MCE approach ~~without using the proposed method~~.

Questions

1. Considering the numerous studies on inverse reinforcement learning and inverse optimal control without the assumption of linearization of the dynamical system and Gaussian approximations, a justification is needed in terms of applicability to robot learning. A comparison with methods without such assumptions should be included. Can the method handle more dynamic tasks such as walking? 2. Clarify the difficulty of the problem in a more intuitive way to explain why this problem remains unsolved. The formulation is somewhat straightforward. Therefore, potential readers may wonder why the problem is still unsolved.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

This is just a suggestion. Regarding POMDP in robot learning, exploring the relationship with world models (e.g., [1,2]) would be valuable. [1] Hafner, Danijar, et al. "Dream to control: Learning behaviors by latent imagination." arXiv preprint arXiv:1912.01603 (2019). [2] Tadahiro Taniguchi, Shingo Murata, Masahiro Suzuki, Dimitri Ognibene, Pablo Lanillos, Emre Ugur, Lorenzo Jamone, Tomoaki Nakamura, Alejandra Ciria, Bruno Lara & Giovanni Pezzulo (2023) "World models and predictive coding for cognitive and developmental robotics: frontiers and challenges," Advanced Robotics, 37:13, 780-806

Summary

In this paper, the authors propose a method to infer an agent’s internal model in a Partially Observable Markov Decision Process (POMDP) when the agent’s actions are non observable. Using local linearization, the authors show how a closed form approximation of the likelihood function for state trajectories can be constructed to subsequently yield maximum likelihood estimates. They also show that when there are confounding factors that can lead to the same behavior, the proposed method is able to disambiguate the factors better compared to the baseline. **After author's rebuttal:** I appreciate the authors offering clarifications. My main concern was around the significance of the contribution, which I based on the references discussed in the paper since I haven't been working on this specific area myself. However, looking at the other reviews and the author's comments, it appears the paper does more than just relaxing a simple assumption or two. I also mentioned that if the inference mechanism is aware of the generative model, one would expect better estimation in general. I make this comment from a Bayesian perspective so if the observations are from a Gaussian distribution and we model it as such, we expect to learn the sensible parameters with enough data as opposed to an unaware model that assumes, say, a Laplace distribution. I would need to think more as to why this might not hold here, as I am still inclined to believe it does. In any case, based on the discussion of the contributions, I am changing my rating from 4 to 5.

Strengths

The paper does a good job at laying down the groundwork for the problem, going over previous work and pointing at the potential shortcomings of existing methods. The motivating example in Fig. 1 works well and the paper’s structure follows naturally. The experiments are well-designed and the results are clearly discussed establishing the technique’s superior performance over the baseline.

Weaknesses

However, in my opinion, the contribution lacks significance for acceptance at the venue. The problem formulation is slightly more general than existing work but only marginally so. The linearization to get approximate likelihood is clean but not novel. The results are not surprising since a technique that is aware of the generative assumptions is expected to lead to better posterior estimates. The disambiguation behavior follows from better estimation.

Questions

NA

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

NA

Additional comments on weaknesses

[Explanation of contribution] To summarize our contribution in one sentence, we introduce an inverse optimal control method that can deal with partially observable stochastic systems, where the agent’s action signals are unknown to the researcher. This requires distinguishing between the control problem from the agent’s viewpoint (Fig. 1A) and the inference problem from the researcher’s viewpoint (Fig. 1B). We have tried to be precise in our wording and always clearly distinguish these two perspectives, but we are happy to elaborate if any concrete passages in the paper are unclear. As for the title, we think that the title both captures the problem setting (stochastic partially observable systems) and the unique selling point (disentangling perceptual uncertainty and behavioral costs). See also the answer to all reviewers at the beginning of this rebuttal. [Partial observability] In IOC we take the perspective of the researcher, who observes a trajectory of states $x_t$, but does not have access to the agent’s noisy observations $y_t$ (see Fig. 1 in the paper). The agent cannot directly observe the true states $x_t$, but instead receives noisy / partial observations $y_t$. Based on these observations, the agent forms a belief state $b_t$ (e.g. using the EKF). We as researchers cannot observe this belief state, because it is a quantity internal to the agent. The likelihood $p(x_1:T)$ depends on the true states $x_t$, which are observed by the researcher. To compute this likelihood, the unobserved internal variables of the agent need to be marginalized out. Although this way of formalizing the IOC problem might be novel, we do not think it is controversial. [Observation noise] We infer parameters of the cost function and of the noise model. For inferring the parameters correctly, the problem still needs to be uniquely identifiable. For the regarded problems, it is therefore not possible to do full system identification while estimating the cost function, as the problem quickly becomes highly unidentifiable. We therefore limit our evaluation to problems with few interpretable parameters that are expected to be uniquely identifiable. Experiments in cognitive and motor science are usually designed this way. Importantly, dynamics in these settings are usually well characterized, e.g. derived from first principles such as kinematics or measured empirically in separate experiments. [Baseline description] For the sake of readability, we preferred to have a rather short intuitive explanation of the baseline in the main text. In Appendix B, there is a detailed and more formal description, which should contain all necessary descriptions. If there are any further concrete questions about the baseline, we are happy to answer them and improve our description.

Thank you for the clarification

Thanks for the comments. I really think that the paper can be improve the clarity of the presentation. Particularly, being clear about partial observability vs noise so the experiments contribution is clear. I would reread the paper in detail with the new comments and make my final evaluation.