Moving object Detection (MOD) is a critical task in autonomous driving as\nmoving agents around the ego-vehicle need to be accurately detected for safe\ntrajectory planning. It also enables appearance agnostic detection of objects\nbased on motion cues. There are geometric challenges like motion-parallax\nambiguity which makes it a difficult problem. In this work, we aim to leverage\nthe vehicle motion information and feed it into the model to have an adaptation\nmechanism based on ego-motion. The motivation is to enable the model to\nimplicitly perform ego-motion compensation to improve performance. We convert\nthe six degrees of freedom vehicle motion into a pixel-wise tensor which can be\nfed as input to the CNN model. The proposed model using Vehicle Motion Tensor\n(VMT) achieves an absolute improvement of 5.6% in mIoU over the baseline\narchitecture. We also achieve state-of-the-art results on the public\nKITTI_MoSeg_Extended dataset even compared to methods which make use of LiDAR\nand additional input frames. Our model is also lightweight and runs at 85 fps\non a TitanX GPU. Qualitative results are provided in\nhttps://youtu.be/ezbfjti-kTk.\n