Thank you for your constructive feedback and positive comments about our benchmark.
**Answer to question 1**
We agree that our scope of VQA is smaller than the scope of VQAv2. However, this is a deliberate choice: Evaluating with a large scope of mixed questions makes it difficult to find the exact strengths of models quantitatively. Our benchmark aims to provide a clear and constrained environment to evaluate specific capabilities of VQA systems: object, activity, and attribute recognition.
Figure 1 (a) shows our decision to crop the relevant object in ImageNet. We ask “What’s this?”, and the model responds with “A tree with no leaves” which is a reasonably good answer given the image-question pair. Even though the desired label is “porcupine”, the model gets distracted by other objects in the image. In a standard classification setup this is less of a problem, since the label space reduces the ambiguity of the answer. To stay with the example of Figure 1 (a), tree is not a category of ImageNet; therefore, a classification or retrieval model cannot answer with a tree class. On the contrary, in our open-ended setup, the model must answer the question in a free manner without knowing the label space, increasing the ambiguity when the image contains more than a single object. By focusing on a cropped object of interest, we effectively reduce the ambiguity of the answer, allowing for a more precise assessment of the model’s ability to recognize and classify the specific object. To allow comparison with other works that test on ImageNet, we also provide results on ImageNet with full images in Supp. Tab. 4 and 5.
In general, our scope of VQA includes asking about objects, actions and attributes. We acknowledge that there are many other facets of VQA systems that merit exploration, such as model performance on natural VQA datasets like VQAv2 and VizWiz. The scope of our benchmark could certainly be increased in future work e.g. by asking about object counts, spatial relations, compositional reasoning (e.g. having multiple object-attribute pairs in the same image and asking about their relations), testing generalization to rare scenes, or other scenarios. We believe our benchmark design and analysis of metrics will pave the way to test Text-VLMs on such abilities.
**Answer to question 2**
When developing ActivityNet-oVQA, we decided using only a single image, since the purpose of the whole benchmark is to evaluate models which answer questions on images. While extending this to VideoQA models is certainly a possible future direction, in this work our focus remains on images. Consistent with standard practice in the literature, we chose the middle frame of each segment as our input. Supp. Fig. 3 shows 4 example scenes which can all be answered given the middle frame shown in the figure.
To determine the effectiveness of selecting the middle frame among all others, we conduct an ablation study, testing all models over nine frame selections by dividing the segment into nine parts and selecting the closest frame. The results are shown in Supp. Fig. 10. We measure accuracy as the average response to the same three initial questions across all models. The results suggest that frame selection does not have a significant impact on performance, especially since frames are selected from short segments in which the relevant action occurs.
Additionally, our choice of using ActivityNet as a VQA dataset is motivated by its well-defined set of activities within a semantic hierarchy which allow us to ask follow-up questions about fine-grained activities.