Summary
This paper considers how to evaluate the quality of a given dataset for an imitation learning task. It is well-known that IL suffers from distribution shift. While much work on IL focuses on improving algorithms, the authors propose that another important approach to making IL more effective is to curate better datasets and demonstrations from experts. Existing approaches to evaluating the quality of datasets focus on state coverage / diversity, trying to ensure that the expert demonstration contains instances of a variety of different states.
The authors discuss in some depth how state diversity is an incomplete measure of dataset quality, specifically because it ignores the actions taken in the dataset; the goal of the paper is to find an improved metric for evaluating dataset quality. he paper proposes action diversity and transition diversity. The authors then consider the distribution shift from the expert policy to the learned policy under a given f-divergence (KL divergence by default), as having a large distribution shift can cause problems in IL.
They then show a number of properties of their measure and relate it to action diversity and transition diversity , and discuss how these metrics can be applied to improve IL in general (4.3, Implications for Data Curation). They claim that their work suggests that action consistency in the expert demonstrations leads to better outcomes, while state diversty and system noise in the dataset are good or bad depending on context.
Finally, they perform experiments on two simple simulationtasks, comparing the effects of adding policy noise to adding system noise in the training set and comparing how demonstration datasets of varying (known) quality correlate with various metrics.
Strengths
The main idea of finding improved metrics for dataset quality, above and beyond state space coverage, is interesting. Furthermore, the dichotomy that their analysis exposes between expanding state space coverage via higher action divergence versus higher transition divergence is an interesting one, showing that not all state space coverage is created equal. The theoretical results linking Q with action divergence are also interesting and well-presented.
While there are questions left by the experiments (see Weaknesses and Questions) one extremely interesting finding is that the additon of a small amount of system noise vastly improves the effectiveness of training with policy noise (compare Figure 2 (c)(d) with Figure 2 (i)(j)), especially when the number of demos is low. The effect of this is so strong that it would have been better to focus more on it when discussing the results.
With a few minor exceptions, the paper is also very well written. The essential ideas are clearly stated and easy for the reader to absorb.
Weaknesses
This paper would be greatly improved by reducing repetition and adding more experiments. The points made in the Intro are repeated at least 5 times. The paper would be much better if the authors agree to remove this redundancy and add more thorough experiments.
he biggest weakness is a key property of the proposed measure (equation (6)), which is tied to a strength of the work ("data quality is heavily tied to the algorithm A that leads to the learned policy \pi_A as well as the expert policy \pi_E"): since it takes the learned policy \pi_A as well as the expert policy \pi_E, actually measuring the quality Q may require running A, which somewhat diminishes the value of the data quality metric since the it may not be usable when collecting the data (note that most common metric, state diversity in the training dataset, can be measured without running A). Even one of the proposed proxies for Q, action divergence, requires having the learned policy to measure against the expert policy.
There is an inconsistency in lines 196 and 199-200 with Theorem 4.1: it says the data quality is "lower bounded by the action divergence" but Theorem 4.1 shows that action divergence provides an upper bound to the data quality. Since upper and lower bounds for dataset quality have different implications this is very important to fix.
While Figure 2 (i)(j) represents the most interesting finding (in my opinion) of this work, it doesn't escape notice that there is no corresponding Figure 2 (k)(l) showing the same plots for the Square nut task. These should absolutely be included to show whether or not the finding in (i)(j) generalizes to at least one other task (otherwise there is the possibility that (i)(j) is simply a property of one particular setting).
The authors do not include a limitations section.
Minor weaknesses and typos:
In (3), the sum should go for t=1 to T, not H
Figure 2 caption: In "X-axis corresponds…" should probably write "injected system noise" twice (rather than "injected Gaussian noise" the first time and "injected system noise" the second time). They're both system noise and both Gaussian, right? Writing it differently may convey the mistaken impression that they might not be both Gaussian, whereas the difference is just when the noise is added. Notation is also confusing, it seems \sigma_s is used both to refer to system noise at evaluation (in the figure itself) and to system noise added in the training set (line 342).
Table 1: instead of merely stating that state similarty is the "opposite of state diversity" (line 360), it would be much better to have a more precise definition. Additionally, it is claimed that "state diversity once again is not correlated with performance on the task" (line 370, just above Table 1) but Better/Okay/Worse all seem to have similar state similarity values while Proficient Human (PH) has lower state similarity (i.e. higher state diversity) and has significantly better success rates – doesn't this mean that it is correlated? Claims of no correlation should probably come with the actual correlation number, since it isn't obvious from the table that there really is no correlation.
Equation (6) should probably be in a Definition since it's one of the key contributions of the work.
4.3 Implications for Data Curation might be better placed after the experiments section.
Questions
This paper would be greatly improved by reducing repetition and adding more experiments. The points made in the Intro are repeated at least 5 times. The paper would be much better if the authors agree to remove this redundancy and add more thorough experiments.
A number of the authors' conclusions (regarding what their results imply about creating and evaluating IL datasets) do not seem to follow obviously from their results and experiments. For instance, they claim in 4.3 (line 294-295): "we posit that expert demonstrators should encourage paths with high system entropy [up to a point]" – this doesn't really seem to follow from their experiments, in which system noise is added to a fixed policy, rather than changing the policy itself.
The authors claim to show that action divergence is undesirable and leads to worse outcomes, while transition diversity (in moderation) can improve results; this is demonstrated by adding policy noise during training and comparing the resulting policy to that learned from a dataset with system noise. However, this only shows that action divergence is harmful when it results from noise being artificially added (in which case the expert can be viewed as only partly trying to achieve the task, while also being partly random).
What about the effect of action divergence resulting from multi-modality (i.e. when many very different courses of action can achieve the goal in a reasonable way)? It is not at all clear that the proposed measures of data quality will work for multi-modal tasks – which have different solutions, e.g. even simple ones like moving around an obstacle (where you can go to the left or right).
It would be interesting to consider how this can be used to guide human expert demonstrations; while I'm not convinced by the discussion in section 4.3, at least without more details (see the first item under Questions), there are possibly other interesting avenues to consider, such as adding noise to the human expert's controls when conducting demonstrations.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Limitations
There is no Limitations section in the paper.
The limitations of the work are only very briefly addressed (lines 390-392) and need more expanding; in particular, the weakness mentioned (that the data quality metric proposed is very hard to measure in real datasets) is a serious one and deserves a much broader discussion.