Hierarchical video search and recognition system

Described is a system for content recognition, search, and retrieval in visual data. The system is configured to perform operations of receiving visual data as an input, processing the visual data, and extracting distinct activity-agnostic content descriptors from the visual data at each level of a hierarchical content descriptor module. The resulting content descriptors are then indexed with a hierarchical content indexing module, wherein each level of the content indexing module comprises a distinct set of indexed content descriptors. The visual data, generated content descriptors, and indexed content descriptors are then stored in a storage module. Finally, based on a content-based query by a user, the storage module is searched, and visual data containing the content of interest is retrieved and presented to the user. A method and computer program product for content recognition, search, and retrieval in visual data are also described.