A deeper understanding of video activities extends beyond recognition of\nunderlying concepts such as actions and objects: constructing deep semantic\nrepresentations requires reasoning about the semantic relationships among these\nconcepts, often beyond what is directly observed in the data. To this end, we\npropose an energy minimization framework that leverages large-scale commonsense\nknowledge bases, such as ConceptNet, to provide contextual cues to establish\nsemantic relationships among entities directly hypothesized from video signal.\nWe mathematically express this using the language of Grenander's canonical\npattern generator theory. We show that the use of prior encoded commonsense\nknowledge alleviate the need for large annotated training datasets and help\ntackle imbalance in training through prior knowledge. Using three different\npublicly available datasets - Charades, Microsoft Visual Description Corpus and\nBreakfast Actions datasets, we show that the proposed model can generate video\ninterpretations whose quality is better than those reported by state-of-the-art\napproaches, which have substantial training needs. Through extensive\nexperiments, we show that the use of commonsense knowledge from ConceptNet\nallows the proposed approach to handle various challenges such as training data\nimbalance, weak features, and complex semantic relationships and visual scenes.\n