Summary
This work introduces a method of learning a visual behavior encoding for reinforcement learning tasks, D3IL.
From a TPIL behavior feature encoder baseline, D3IL introduces dual feature extraction (including both behavior and domain feature encoders) and dual cycle-consistency (cycle consistency on image and feature levels). From inputs of unpaired target and source images, D3IL translates across domains and then cycles back to the original domain, via a pair of generator networks which take in source images and domain targets. Further, image reconstruction consistency, feature vector similarity, and feature vector consistency is added for further loss and stability.
The domain and behavior feature encoders and target generator from above are taken and used to train a reward generating discriminator, $D_rew$, which distinguishes between expert and learner behavior (rather than expert and non-expert data used for training the previous models). This is used alongside the target behavior encoder to train SAC RL policies for the given tasks.
D3IL is tested on many domain transfer tasks with variations in visual effects, degree-of-freedom, internal dynamics, and robot embodiment - D3IL demonstrates superior performance on the target domain compared to baselines of TPIL, …. It is further demonstrated on an AntUMaze task where direct RL learning in the target domain is difficult, but possible with D3IL. Ablation studies show that the contributions of each of the consistency losses across feature and images, and cycle consistency, are all important and contribute to the final performance.
Strengths
#### Originality
The work builds on top of TPIL to build a larger framework for cross-domain imitation / feature encoding, by incorporating dual feature extraction and cycle consistency, and training a further reward-generating discriminator.
#### Quality
The experiments are detailed, covering four categories of domain change and with ablation studies on components of the loss. The experiments are run over multiple seeds and the method is compared against several baselines (TPIL, DisetanGAIL, GWIL) to present a good picture of the overall learning and domain transfer capability. There are further many more detailed experiments in the appendix to which present good coverage.
#### Clarity
The work is generally well-written and the description of D3IL which draws from premises of prior TPIL work is understandable. The explanations of losses and model composition are detailed and the diagrams help with understanding the proposed methodology. The experiments are easy to follow. Further task explanation and experimental methodology is well explained in the appendix.
#### Significance
The method seems broadly applicable across tasks and domain transfer goals in simulation.
The experiment on AntUMaze is particularly motivating as towards the practical applications of the method, demonstrating that a task which could not be solved directly could be improved by training from an imitation agent learned on a simpler task - in a sort of curriculum learning.
Weaknesses
#### Quality
The dual feature extraction, in particular the domain encoder added from TPIL, is motivated on lines 135-136 as to “check that the obtained feature well contains the required information without information loss, based on image reconstruction and consistency check.” However, it’s still unclear why specifically it is necessary to pass observations as input to domain encoder, as the work assumes that the policy knows what environment it is operating in. Would an ideal domain encoder be able to output a one-hot vector indicating its domain to minimize all the domain-encoder relevant losses? As there are separate generators for each domain (source/target), should it be clear without any domain conditioning what the “style” of each generator output should be? For example, in CycleGAN and its successors, the domain encoding is not required.
An ablation study without using a domain encoder, but a static method of encoding the domain or skipping the domain encoding entirely may be helpful.
#### Clarity
Unfortunately many of the experimental results are left to to the appendix including all of Sections 6.2 and 6.5, and reading the appendix seems necessary to have a full understanding of the model performance.
The method is somewhat hard to initially grasp from reading the text of Section 5 as there are many similar variables in the notation which differ by one letter or modifier, with many such modifiers. However, explanations are generally helped by Figures 1 and 3.
The limitations are also only mentioned in the appendix.
Misc:
- Line 191: is another form of consistency [that] guide[s] the model so that …
- Line 195: scheme significantly improve[s]
- Line 211: we need [the] expert behavior feature
#### Significance
The practical applications of the method seem best suited for cases where it is hard to train an expert in the target environment directly via RL, but having access to expert data on other tasks as observation-only would help. Many of these cases would be covered by real world scenarios, which is out-of-scope of the current work. Even on the AntUMaze problem, the learning process did not necessarily have to be done via adversarial learning and visual-only observations, as the full expert state would be available.
From the ablation in Table 1, it appears that the large majority of the performance improvement comes from image-level consistency and the feature-level consistency was only of minor gain, especially compared to the uncertainty/error.
As mentioned in the limitations in the appendix, the paper also introduces quite the complex process with significantly many more loss terms and two pipeline stages compared to prior work, which may make it harder to apply. Adversarial learning techniques generally require more careful tuning than supervised methods, and this work introduces two adversarially-learned stages.
Questions
Some discussion on the weakness-quality section would be most helpful
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
Some consideration of the limitations is explained in the weaknesses-significance section.