Response to Review
We thank the reviewer for their feedback. We address their concerns below:
We would like to clarify our exposition of GVA. First, note that our mechanistic understanding of GVA arises from *both* careful empirical findings and theoretical explanations. To reiterate, the empirical findings for GVA are:
(a) More data and/or regularization **do not mitigate GVA**.
(b) Increasing batch size improves, but does not entirely ameliorate, GVA.
(c) EMA, which reduces iterate variance, nearly entirely fixes GVA
Through our analysis of local perturbations around imitator policies at different stages of training (Fig 1 right and Fig 13), we observe that while the *validation loss* landscape is relatively insensitive to parameter perturbations, the *policy evaluation loss* landscape is very sensitive to these.
In summary, we conclude that (i) evaluation loss is highly sensitive to changes in parameters and (ii) that methods which reduce variance in iterates/gradients seem to ameliorate this sensitivity.
The role of our theory is to show that the above hypothesis is mathematically plausible, and revealed even without considerations of non-convex representation learning.
**Proposition 3.1**: We wish to clarify the precise statement of Proposition 3.1, which is intended as an example of exponential error amplification but not intended to be perfectly representative of what we observe in experiments. Proposition 3.1 **does not** state that for every delta, the BC error is exponential-in-H larger than validation. Instead, it shows that
For all delta > O(1/H), there **exists** a perturbed policy of error delta which witnesses exponential blow up
For **all** delta < O(1/H), there is no exponential blow up.
The point of the proposition is that, to ensure no exponential blowup in the worst case, for long rollout horizons, the BC loss must be made sufficiently small. It may be possible, with long training times and sufficient learning rate decay, that one could make the BC loss sufficiently small as to ensure the rollout reward does not suffer from exponential blowup. Indeed, our experiments corroborate this qualitative finding (point to Fig): that if we train for a longer time with sufficient learning rate decay, we do get consistently large rollout reward. This, however, is wasteful in terms of training time, as EMA does just as well without excessive numbers of training epochs.
**Cliff Loss:**
Surprising, we find that GVA is not a major obstacle when learning to control linear systems (which includes the example in Prop 3.1). If we examine how the cloned policy behave during rollout time, we found (and will include figures in the final revision) that the deviation between cloned policy and expert, under the rollout of the cloned policy, starts to deviate wildly after a specific time step in the RL environment. We interpret this as the presence of dynamical stiffness in the task, e.g. the contact that the locomoting agent makes with the floor surface. Importantly, MuJoCo environments are well approximated by Piecewise Affine Systems, a classical model for contact dynamics.
In the revision, we will clarify that the cliff loss is a simple and tractable mathematical abstraction of the above phenomenon. The cliff models a “two-region” contact system, where ‘falling off the cliff’ corresponds to a change in region.
**Questions**
This is a good question that we will address in the revision. To be precise, in our particular example in the proof of Proposition 3.1, there are good subsets only around parameters that are `marginally stable' and thus many policies do not have these good neighborhoods. We believe this phenomenon to persist in the general case, with Figure 1 (right) and Figure 13 providing empirical evidence in favor of this claim in the MuJoCo Walker environment.