Generalized Decision Transformer for Offline Hindsight Information Matching

How to extract as much learning signal from each trajectory data has been a\nkey problem in reinforcement learning (RL), where sample inefficiency has posed\nserious challenges for practical applications. Recent works have shown that\nusing expressive policy function approximators and conditioning on future\ntrajectory information -- such as future states in hindsight experience replay\nor returns-to-go in Decision Transformer (DT) -- enables efficient learning of\nmulti-task policies, where at times online RL is fully replaced by offline\nbehavioral cloning, e.g. sequence modeling. We demonstrate that all these\napproaches are doing hindsight information matching (HIM) -- training policies\nthat can output the rest of trajectory that matches some statistics of future\nstate information. We present Generalized Decision Transformer (GDT) for\nsolving any HIM problem, and show how different choices for the feature\nfunction and the anti-causal aggregator not only recover DT as a special case,\nbut also lead to novel Categorical DT (CDT) and Bi-directional DT (BDT) for\nmatching different statistics of the future. For evaluating CDT and BDT, we\ndefine offline multi-task state-marginal matching (SMM) and imitation learning\n(IL) as two generic HIM problems, propose a Wasserstein distance loss as a\nmetric for both, and empirically study them on MuJoCo continuous control\nbenchmarks. CDT, which simply replaces anti-causal summation with anti-causal\nbinning in DT, enables the first effective offline multi-task SMM algorithm\nthat generalizes well to unseen and even synthetic multi-modal state-feature\ndistributions. BDT, which uses an anti-causal second transformer as the\naggregator, can learn to model any statistics of the future and outperforms DT\nvariants in offline multi-task IL. Our generalized formulations from HIM and\nGDT greatly expand the role of powerful sequence modeling architectures in\nmodern RL.\n