Vision-and-language (V\\&L) reasoning necessitates perception of visual\nconcepts such as objects and actions, understanding semantics and language\ngrounding, and reasoning about the interplay between the two modalities. One\ncrucial aspect of visual reasoning is spatial understanding, which involves\nunderstanding relative locations of objects, i.e.\\ implicitly learning the\ngeometry of the scene. In this work, we evaluate the faithfulness of V\\&L\nmodels to such geometric understanding, by formulating the prediction of\npair-wise relative locations of objects as a classification as well as a\nregression task. Our findings suggest that state-of-the-art transformer-based\nV\\&L models lack sufficient abilities to excel at this task. Motivated by this,\nwe design two objectives as proxies for 3D spatial reasoning (SR) -- object\ncentroid estimation, and relative position estimation, and train V\\&L with weak\nsupervision from off-the-shelf depth estimators. This leads to considerable\nimprovements in accuracy for the "GQA" visual question answering challenge (in\nfully supervised, few-shot, and O.O.D settings) as well as improvements in\nrelative spatial reasoning. Code and data will be released\n\\href{https://github.com/pratyay-banerjee/weak_sup_vqa}{here}.\n