Summary
This paper concerns the topic of Inverse Optimization (IO) which deals with learning the objective function of the decision maker given previous data from an expert decision maker. By kernelizing the objective function, they extend the framework to features from a potentially infinite-dimensional space. The main contributions of this paper is a scalable SDP formulation of the kernel inverse optimization problem and a Sequential Selection Optimization algorithm that works by selecting a batch of variables at random or according to a KKT-based heuristic to optimizer over. The effectiveness of the SSO algorithm is shown using experimentally using imitation learning tasks from the MuJoCo benchmark.
Strengths
The mains strength of this paper is the presentation of a scalable SDP formulation of the kernel optimization problem and a thorough experimental evaluation across a variety of MuJoCo benchmark tasks which includes an ablation study to evaluate different combinations of variable selection and warm-up strategies.
Weaknesses
The main weakness is the lack of convergence guarantees for Sequential Selection Optimization (SSO) algorithm for the given different variable selection strategies. However, this is not a major drawback as the main contribution is a scalable SDP formulation for the kernel inverse optimization problem. The authors used Gaussian kernel for all experiments but it would be good to have more discussion about how type of kernel influences the performance of the SSO algorithm or quality of the solutions. There is also a lack of detail about how the baseline methods BC(TD3+BC) and BC(CQL) and Teacher were evaluated (e.g. hyperparameters, amount of data used and number of iterations to converge). There is a lack of convergence plots for SSO and other baselines in Table 1 and 2.
Questions
1. In Section 4.2, how does one concatenate the optimal solutions of the n solved small problems to form an initial guess? Have you evaluated different possible ways to form an initial guess?
2. Have you considered evaluating your SSO algorithm on different types of kernel functions?
3. For Table 1, you report only the amount of data the KIO+SSO method used but you didn't report how much data the BC(TD3+BC) and BC(CQL) offline reinforcement learning algorithms used. Is there a reason for this?
4. For Table 1, why are the scores for IO so uniformly low compared to KIO? Is it possible that IO and KIO may perform similarly in simpler imitation learning tasks?
5. For Table 2, it is quite interesting to see that SCS and SSO converged to more or less the same objective value and it would be interesting to see how many iterations (and time) it took for SCS to converge vs. how many iterations for SSO to converge. This could be better represented in a plot. The convergence could be bad in the beginning due to low quality of the initial guess and then improve as the algorithm converges. Its fairly obvious that the scores will be the same range for all trials if the objective value is the same, so it doesn't help to report objective value and score side-by-side.
6. Have you considered reporting the speed of convergence for KIO + SSO method vs. BC(TD3+BC) and BC(CQL) offline reinforcement learning algorithms in terms of a plot of objective value vs. # of iterations or objective value vs. time?
Limitations
The authors acknowledge the limitations of their work, which are (1) lack of convergence guarantees for Sequential Selection Optimization (SSO) algorithm for the given different variable selection strategies and (2) the large memory requirements.