Standard dynamics models for continuous control make use of feedforward\ncomputation to predict the conditional distribution of next state and reward\ngiven current state and action using a multivariate Gaussian with a diagonal\ncovariance structure. This modeling choice assumes that different dimensions of\nthe next state and reward are conditionally independent given the current state\nand action and may be driven by the fact that fully observable physics-based\nsimulation environments entail deterministic transition dynamics. In this\npaper, we challenge this conditional independence assumption and propose a\nfamily of expressive autoregressive dynamics models that generate different\ndimensions of the next state and reward sequentially conditioned on previous\ndimensions. We demonstrate that autoregressive dynamics models indeed\noutperform standard feedforward models in log-likelihood on heldout\ntransitions. Furthermore, we compare different model-based and model-free\noff-policy evaluation (OPE) methods on RL Unplugged, a suite of offline MuJoCo\ndatasets, and find that autoregressive dynamics models consistently outperform\nall baselines, achieving a new state-of-the-art. Finally, we show that\nautoregressive dynamics models are useful for offline policy optimization by\nserving as a way to enrich the replay buffer through data augmentation and\nimproving performance using model-based planning.\n