Learning to Relate Depth and Semantics for Unsupervised Domain Adaptation

We present an approach for encoding visual task relationships to improve\nmodel performance in an Unsupervised Domain Adaptation (UDA) setting. Semantic\nsegmentation and monocular depth estimation are shown to be complementary\ntasks; in a multi-task learning setting, a proper encoding of their\nrelationships can further improve performance on both tasks. Motivated by this\nobservation, we propose a novel Cross-Task Relation Layer (CTRL), which encodes\ntask dependencies between the semantic and depth predictions. To capture the\ncross-task relationships, we propose a neural network architecture that\ncontains task-specific and cross-task refinement heads. Furthermore, we propose\nan Iterative Self-Learning (ISL) training scheme, which exploits semantic\npseudo-labels to provide extra supervision on the target domain. We\nexperimentally observe improvements in both tasks' performance because the\ncomplementary information present in these tasks is better captured.\nSpecifically, we show that: (1) our approach improves performance on all tasks\nwhen they are complementary and mutually dependent; (2) the CTRL helps to\nimprove both semantic segmentation and depth estimation tasks performance in\nthe challenging UDA setting; (3) the proposed ISL training scheme further\nimproves the semantic segmentation performance. The implementation is available\nat https://github.com/susaha/ctrl-uda.\n