Learning Cross-Domain Correspondence for Control with Dynamics Cycle-Consistency

At the heart of many robotics problems is the challenge of learning\ncorrespondences across domains. For instance, imitation learning requires\nobtaining correspondence between humans and robots; sim-to-real requires\ncorrespondence between physics simulators and the real world; transfer learning\nrequires correspondences between different robotics environments. This paper\naims to learn correspondence across domains differing in representation (vision\nvs. internal state), physics parameters (mass and friction), and morphology\n(number of limbs). Importantly, correspondences are learned using unpaired and\nrandomly collected data from the two domains. We propose \\textit{dynamics\ncycles} that align dynamic robot behavior across two domains using a\ncycle-consistency constraint. Once this correspondence is found, we can\ndirectly transfer the policy trained on one domain to the other, without\nneeding any additional fine-tuning on the second domain. We perform experiments\nacross a variety of problem domains, both in simulation and on real robot. Our\nframework is able to align uncalibrated monocular video of a real robot arm to\ndynamic state-action trajectories of a simulated arm without paired data. Video\ndemonstrations of our results are available at:\nhttps://sjtuzq.github.io/cycle_dynamics.html .\n

Paper

Similar papers

© 2026 NYSGPT2525 LLC