In this work, we present an approach to deep visuomotor control using\nstructured deep dynamics models. Our deep dynamics model, a variant of\nSE3-Nets, learns a low-dimensional pose embedding for visuomotor control via an\nencoder-decoder structure. Unlike prior work, our dynamics model is structured:\ngiven an input scene, our network explicitly learns to segment salient parts\nand predict their pose-embedding along with their motion modeled as a change in\nthe pose space due to the applied actions. We train our model using a pair of\npoint clouds separated by an action and show that given supervision only in the\nform of point-wise data associations between the frames our network is able to\nlearn a meaningful segmentation of the scene along with consistent poses. We\nfurther show that our model can be used for closed-loop control directly in the\nlearned low-dimensional pose space, where the actions are computed by\nminimizing error in the pose space using gradient-based methods, similar to\ntraditional model-based control. We present results on controlling a Baxter\nrobot from raw depth data in simulation and in the real world and compare\nagainst two baseline deep networks. Our method runs in real-time, achieves good\nprediction of scene dynamics and outperforms the baseline methods on multiple\ncontrol runs. Video results can be found at:\nhttps://rse-lab.cs.washington.edu/se3-structured-deep-ctrl/\n