Dear reviewer,
Thanks for mentioning (1) why it should be useful to learn a reward function incrementally given that this increases the reward unidentifiability issue, and (2) the writing of the problem setting. We address these two comments below.
**For the comment (1)**, we answer from two aspects: (i) the connection between learning a useful reward and reward identifiability, and (ii) why the incrementally learned reward is useful.
(i) The fundamental usefulness of the learned reward is to explain the expert demonstrations. As mentioned in the first paragraph in our previous response, solving the reward identifiability issue is sufficient but not necessary to learn a reward function to explain the demonstrations. We agree that learning a reward incrementally can make it harder to solve reward unidentifiability issue. However, our algorithm can still learn a reward that explains the expert demonstrations despite the learned reward function may be different from the ground truth. In fact, shown in Table 1 in our paper, the optimal policy corresponding to the learned reward can generate trajectories whose cumulative ground truth reward is close to the expert cumulative ground truth reward, i.e., the learned policy generates trajectories close to the expert trajectories.
(ii) Moreover, compared to imitation learning that directly learns an imitating policy, our learned reward can be used for conterfactual analysis [C8], e.g., instantly estimating the policies under different environment dynamics. Take the stock market as an example, the reward function captures the investor's risk tolerance [C11]. In our experiment, we learn the investor's risk tolerance incrementally in the stock market of Dow Jones Industrial Average (DJIA). The incrementally learned reward can instantly update the risk tolerance of the target investor. The instantly updated reward has two benefits. First, we can use the latest reward to recommend stocks as soon as possible, which gives us advantages to win this customer before our competitors. Second, we can use the latest reward to estimate the investment behaviors preferred by the target investor in some other stock markets, e.g., CCXT (cryptocurrency) and QuantConnect (US securities) [C12], so that we can not only recommend stocks in the stock market where we collect data, i.e., DJIA, but also recommend stocks in other stock markets, e.g., CCXT and QuantConnect. If we use imitation learning, we can definitely learn an instant imitating policy and use this imitating policy to recommend stocks of DJIA as soon as possible. However, the imitating policy learned by imitation learning cannot be used to recommend stocks in CCXT and QuantConnect because they have different dynamics. In contrast, we can still apply the learned reward to these two new markets to learn the corresponding optimal policies and use the corresponding optimal policies to recommend stocks, because the investor's risk tolerance does not change even if the stock markets are different. We include an additional experiment result below where we use the data collected in DJIA to learn the reward function of the target investor and then apply the learned reward to CCXT and QuantConnect. We compare the cumulative reward between the optimal policy under the learned reward and the expert in these two different markets.
| | CCXT | QuantConnect |
|:---:|:---:|:--:|
| Learned policy under the learned reward | 562.18 ± 34.02 | 491.72 ± 18.25 |
| Expert | 571.83 ± 27.16 | 502.82 ± 14.26 |
From the results above, we can see that even if in new stock markets with different dynamics, the learned reward from DJIA can still be used to generate policies in the new stock markets and the generated policies can still achieve similar cumulative ground truth reward with the expert in the new stock markets. This shows that the learned reward successfully captures the investor's risk tolerance.
**For the comment (2)**, we will add more explanations about the problem formulation (2)-(3) as follows: (i) we will highlight that the method we use to learn a reward function is maximum likelihood IRL (ML-IRL) and provide the intuition of ML-IRL where we aim to learn a reward function that makes the demonstration most likely; (ii) we will show our connection to the standard IRL where our problem formulation (2)-(3) will reduce to the formulation of ML-IRL when the complete trajectory can be observed; (iii) we will discuss the motivation to learn a reward function incrementally.
[C11] Qizhou Sun, X. Gong, and Y.-W. Si, "Transaction-aware inverse reinforcement learning for trading in stock markets," Applied Intelligence, 2023.
[C12] Xiaoyang Liu, et al. "FinRL-Meta: Market environments and benchmarks for data-driven financial reinforcement learning." NeurIPS, 2022.