Summary
The paper introduces VizWiz-LF, a new long-form visual question answering (VQA) dataset that focuses on visual questions posed by blind and low-vision (BLV) people. The datasets consists of 600 image-question pairs sampled from the VizWiz dataset. To ensure diversity of questions, the data is sampled from 4 question types, i.e., Identification, Description, Reading, and Others, which is based on the question taxonomy of VizWiz. Among the 600 image-question pairs, 230 are marked as unanswerable. For each image-question pair, 3 long-form answers are constructed -- one by human expert, one by GPT-4V, and the other by Gemini. The resulting answers from all annotations are more than 10x longer than the original VizWiz dataset. Each sentence in the long-form answers is annotated with functional roles (confirmation, answer, answer failure, auxiliary information, auxiliary information failure, explanation, suggestion, and miscelleneous) and information type (image content, image quality, external information). To construct the annotation, 180 long-form answers are annotated by human through a three-way annotations achieving a high fleiss kappa score (0.76 for functional roles and 0.87 for information types). The rest of the data is annotated by GPT-4 through few-shot in-context prompt, which achieves F1-score of 0.74 compared to human F1-score of 0.79).
Using the generated data, the paper demonstrates that both human expert and GPT-4V produce diverse functional roles, while Gemini often onlty cover answer roles since the long-answer often only consists of a single sentence. Additionally, Gemini also rarely abstain to answer even when the question is unanswerable. Furthermore, the paper also shows that model-based approaches, i.e., Gemini and GPT-4V, contains a higher percentage of external information which does not appear on the image. Human evaluation is also conducted to measure the human preferences of the VizWizL-LF considering 4 factors including relevance. helpfulness, plausibility, and fluency. The human evaluation is conducted by 20 BLV and 20 sighted people. All evaluator groups favoured GPT-4V and human expert annotations over Gemini. While sighted people give low helpfulness score for Gemini due to incorrect information provided as an answer, BLV group rated high plausability for all annotations. An additional automatic evaluation using ROUGE, METEOR, and BERTScore is also conducted further showing the effectiveness of GPT-4V for generating long-form VQA. The paper further explore on abstention strategy which helps to the model to abstain from answering when the question is unanswerable.
Reasons to accept
- The paper introduces a simple method to extend the existing VQA dataset into a long-form VQA which is beneficial for blind and low-vision people
- The paper introduces a new long-form VQA dataset which is annotated with per sentence functional roles and information type, and showcases its applicability for analyzing the behaviour of synthetic data generated by LLMs.
- The paper provides an extensive analysis of the quality of the generated data through both human and automatic evaluations, and provides a clear in-depth analysis of the results
- The paper showcases an experiment on abstention to improve the ability of LLMs to abstain from answering unanswerable questions.
- The paper provides clear, easy-to-follow, and detailed enough writing (except for the result tables) on each section with complete detailed information in the appendix.
Reasons to reject
- The results tables are difficult to follow with too many numbers and have no clear highlight of which part readers should focus on
- Section 6 feels rather detached from the other contents of the paper
- It is not unclear how long-form VQA and VizWiz-LF can be beneficial for blind and low-vision people since the human evaluation is only conducted on long-form answers with no comparison on short-answer VQA.