Video QA challenges modelers in multiple fronts. Modeling video necessitates\nbuilding not only spatio-temporal models for the dynamic visual channel but\nalso multimodal structures for associated information channels such as\nsubtitles or audio. Video QA adds at least two more layers of complexity -\nselecting relevant content for each channel in the context of the linguistic\nquery, and composing spatio-temporal concepts and relations in response to the\nquery. To address these requirements, we start with two insights: (a) content\nselection and relation construction can be jointly encapsulated into a\nconditional computational structure, and (b) video-length structures can be\ncomposed hierarchically. For (a) this paper introduces a general-reusable\nneural unit dubbed Conditional Relation Network (CRN) taking as input a set of\ntensorial objects and translating into a new set of objects that encode\nrelations of the inputs. The generic design of CRN helps ease the common\ncomplex model building process of Video QA by simple block stacking with\nflexibility in accommodating input modalities and conditioning features across\nboth different domains. As a result, we realize insight (b) by introducing\nHierarchical Conditional Relation Networks (HCRN) for Video QA. The HCRN\nprimarily aims at exploiting intrinsic properties of the visual content of a\nvideo and its accompanying channels in terms of compositionality, hierarchy,\nand near and far-term relation. HCRN is then applied for Video QA in two forms,\nshort-form where answers are reasoned solely from the visual content, and\nlong-form where associated information, such as subtitles, presented. Our\nrigorous evaluations show consistent improvements over SOTAs on well-studied\nbenchmarks including large-scale real-world datasets such as TGIF-QA and TVQA,\ndemonstrating the strong capabilities of our CRN unit and the HCRN for complex\ndomains such as Video QA.\n