Recent advancements in large multimodal language models have demonstrated remarkable proficiency across a wide range of tasks. Yet, these models still struggle with understanding the nuances of human humor through juxtaposition, particularly when it involves nonlinear narratives that underpin many jokes and humor cues. This paper investigates this challenge by focusing on comics with contradictory narratives, where each comic consists of two panels that create a humorous contradiction. We introduce the YesBut benchmark, which comprises tasks of varying difficulty aimed at assessing AI's capabilities in recognizing and interpreting these comics, ranging from literal content comprehension to deep narrative reasoning. Through extensive experimentation and analysis of recent commercial or open-sourced large (vision) language models, we assess their capability to comprehend the complex interplay of the narrative humor inherent in these comics. Our results show that even state-of-the-art models still lag behind human performance on this task. Our findings offer insights into the current limitations and potential improvements for AI in understanding human creative expressions.
Paper
Similar papers
Peer review
Summary
This paper introduces YESBUT, a new benchmark for evaluating large vision-language models' ability to understand humor and contradictions in comics with juxtaposed panels. The benchmark consists of two-panel comics with contradictory narratives, along with annotations for literal descriptions, contradiction explanations, underlying philosophies, and titles. The authors design four tasks of increasing difficulty: literal description writing, contradiction generation, underlying philosophy selection, and title matching. They evaluate several commercial and open-source vision-language models on these tasks using both automatic and human evaluation. The results show that even state-of-the-art models struggle with these tasks, especially the deeper reasoning required for understanding contradictions and abstractions. The paper provides insights into current limitations of AI in comprehending complex human expressions and offers directions for improvement.
Strengths
Originality: The paper addresses a unique and underexplored area in AI research—understanding humor in comics through contradictory narratives. This is a novel problem formulation that pushes the boundaries of current Vision Language Model (VLM) capabilities. Quality: The data collection and annotation process are rigorous, involving multiple stages of human-AI collaboration and quality checks. The experimental design is comprehensive, evaluating multiple types of models on various aspects of comic understanding. Clarity: The paper is well-structured and clearly explains the motivation, dataset creation, task designs, and experimental results. Figures and examples effectively illustrate the concepts. Significance: Understanding humor and contradictory narratives in comics is significant for advancing AI's social and semantic comprehension. This research provides valuable insights into current AI limitations in this area and offers a pathway for future improvements, which is crucial for developing socially intelligent systems.
Weaknesses
1. The dataset size is relatively small (348 comics), which may limit the generalizability of the findings. The authors acknowledge this limitation. 2. The annotation process, while rigorous, relies heavily on human judges and GPT-4, which may introduce biases. The paper does not explore potential biases in the dataset, such as cultural specificity of the humor or potential annotator biases. 3. While the paper demonstrates that augmenting models with oracle descriptions improves performance, it does not thoroughly investigate why decomposing the task for vision-language models (by first generating descriptions) does not lead to consistent improvements. 4. The paper highlights the limitations of current VLMs but does not provide concrete suggestions or experiments on how to overcome these limitations. Including some preliminary experiments with potential improvements could strengthen the paper.
Questions
1. How might the performance of these models change if tested on comics from different cultural contexts or with different styles of humor? 2. Have the authors considered expanding the benchmark to include comics with more than two panels, to evaluate models' ability to understand more complex narrative structures? 3. Could the authors provide more insight into why decomposing the task for VLMs (by first generating descriptions) doesn't consistently improve performance, especially for the title matching task? 4. Can you provide more details on the types of biases that might have been introduced during the annotation process and how they were mitigated? 5. How do the models' performances correlate with their training data or pre-training approaches? Could this provide insights into what types of pre-training might be most beneficial for these tasks? 6. What are some specific strategies or architectural changes you propose for improving VLMs' understanding of contradictory narratives in comics?
Rating
7
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
The authors adequately address the limitations of their work, particularly acknowledging the relatively small dataset size and potential ambiguity in comic interpretation due to subjectivity. They also recognize that their benchmark may not cover all aspects of visual understanding required for more generalized AI applications. The paper does not explicitly discuss potential negative societal impacts. While the focus on humor understanding is generally positive, it may be worth considering potential misuse cases (e.g., automated generation of misleading or offensive comics) or biases that could be amplified if such systems were deployed at scale. Suggestions for improvement include: 1. Expanding the dataset to include a more diverse set of comics and narrative styles. 2. Implementing bias detection and mitigation techniques in the annotation and model training processes. 3. Considering the ethical implications of AI-generated content in real-world applications, ensuring that it respects cultural and social nuances.
Summary
The paper proposes a new evaluation benchmark to evaluate how much current VLMs understand the humor and uses the new benchmark to compare various VLMs and LLMs.
Strengths
Although some previous studies such as [7] and [10] have proposed the humor benchmark for VLM, like author mentioned in 112-114, the proposed benchmark is the first one which has two input images in each sample and the focus is the relation between two images. As far as I know, this is indeed novel. Based on the dataset, the authors proposed several tasks, which allow researchers to conduct more in-depth analysis on LLM's performance. The paper uses the benchmark to compare many VLMs and conduct some good analyses.
Weaknesses
The dataset size is relatively small. Both [7] and [10] have > 1k samples. The small dataset size might create some instability of the scores. I guess that is why the paper does not provide statistical significance analyses. The subjectiveness of the task and the noise of automatic metrics could intensify this problem. We can see that although GPT-4 achieves the best performance in most metrics, the correlations between BERT, R-2, and GPT are not super high for some models. Although automatic evaluation using GPT has shown to have high correlation with human evaluation, prior studies (e.g., https://arxiv.org/html/2404.13076v1) show that it is biased toward their own generation and my experience is that BERT score or R-2 are both not reliable metrics for creative writing and GPT evaluation is only more reliable when the dataset size is large. The paper does not provide human performance, which make interpreting the state-of-the-art performance more difficult. For example, are the scores of GPT-4 in Table 2 really good?
Questions
Could you provide some statistical significance analyses? Which GPT you are using for evaluation? GPT-4? GPT 3.5 turbo?
Rating
7
Confidence
3
Soundness
3
Presentation
3
Contribution
3
Limitations
The discussion of the weaknesses I mentioned could be added.
Thanks for the rebuttal and extra experiments
Based on the answer, I have increased the contribution score from 2 to 3. Although GPT4 and gpt-3.5-turbo-0125 are different LLMs, they might be trained on the similar data. If authors have time and budgets, I recommend to try other LLMs (e.g., Claude 3) for evaluating GPT4 and Claude 3 again and see if the results are different from what you got from gpt-3.5-turbo-0125.
Dear Reviewer wVUt, Thank you once again for your review and feedback. We appreciate the increased contribution score. We have included additional results using Claude-3 (claude-3-opus-20240229) as the base model for automatic evaluations, with the same evaluation prompts. The results are presented below: - Literal Description | Eval LLM | GPT-4 |Claude-3 | LLaVA-1.6-34B | LLaVA-1.6-13B | LLaVA-1.5-13B| |-----------------|-----------|--------------|----------------------|----------------------|----------------------| | GPT-3.5 | **3.76** | *3.28* | 2.86 | 2.96 |2.51 | |Claude-3 |**3.44** |*2.64* | 2.48 |*2.64* |2.02 | - Contradiction | Eval LLM | GPT-4 |Claude-3 | LLaVA-1.6-34B | LLaVA-1.6-13B | LLaVA-1.5-13B| |-----------------|-----------|--------------|----------------------|----------------------|----------------------| | GPT-3.5 | **4.03** | *3.79* | 3.51 | 3.36 |3.36 | |Claude-3 | **3.69** | *3.26* | 2.78 | 2.84 |2.65 | As shown, the results of Claude-3 display a trend similar to those of GPT-3.5. We will incorporate these findings into our revised paper. We will also modify our paper to clarify the previously raised points and incorporate your suggestions. If you have any further questions or concerns, please feel free to reach out, and we will be happy to address them. Best Regards, Authors
Thank you. I have increased the overall rating from 6 to 7 and have no further question.
Dear Reviewer wVUt, We sincerely appreciate your recognition of our efforts and the increased score. We are also grateful for your valuable suggestions and will ensure they are incorporated into our revisions. Best regards, Authors
Summary
This paper presents a benchmark YESBUT, which contains pairs of images exhibiting a sense of humor via juxtaposition. For each pair, the authors employed human-AI collaboration methods to annotate detailed tasks to assess why the humor can be understood, including literal description writing, contradiction generation, underlying philosophy selection, and title matching. Based on the YESBUT benchmark, there are rich benchmark results presented, including the comprehensive comparison with varying LLMs and VLMs and in-depth analyses to provide insights.
Strengths
1. The paper presents a new task of juxtaposition-based humor in multimodal settings. It can reflect the system's ability to interpret and generate nuanced, contextually appropriate responses, thereby improving its interaction with humans in more natural and engaging ways. The task is challenging because it requires models to make sense of the non-linear connections between two semantically related images and social reasoning skills to capture a sense of humor. The results also show the limitations of existing models in handling the task. 2. The YESBUT benchmark crafted to research the task is helpful. It contains rich human annotations reflecting varying perspectives for task evaluation. The benchmark can be beneficial for future research. 3. The paper presents rich experimental studies. The results and analyses offer a comprehensive view of the pros and cons of the cutting-edge VLM techniques and provide in-depth analyses to interpret where the challenges are and how to address them. 4. The paper is written very clearly with well-designed organizations, clear writing, and good presentations with cases, figures, tables, etc.
Weaknesses
1. The dataset is relatively small, with only 348 image pairs. Although I understand that the data is very hard to gather and the annotation is very labor intensive, a larger dataset will enable more sound evaluation results and allow the potential for model training (now it can only support model evaluation). 2. It would be good if the authors could also examine how the models can be further improved to tackle the task well. In the current version, most of the findings center on the limitations, but it would be good for the authors to also point out the directions of further studies in advancing the technology.
Questions
1. For philosophy selection, why it is formulated as the MCQs instead of text generation like other tasks? 2. How to ensure the diversity of the benchmark data (so the image covers varying testing scenarios)? 3. Would the prompts play crucial roles in models’ decision making? What will happen if different sets of prompts are used? 4. Why GPT-4 can engage in the annotation as well as the model comparison? Will that involve any bias?
Rating
8
Confidence
4
Soundness
3
Presentation
4
Contribution
3
Limitations
The dataset size is relatively small, which hence cannot be used for model training. Also, it would be good if the authors can provide insight into how to further advance the techniques to better tackle the task.
Summary
This paper investigates the capability of large vision language models (VLMs) to understand humor in comics through narrative contradictions. The authors introduce the YESBUT benchmark, comprising tasks designed to evaluate AI’s ability to recognize and interpret contradictory narratives in comics. Experiments are conducted with both commercial and open-source VLMs to assess their performance on tasks ranging from literal content comprehension to deep narrative reasoning.
Strengths
1. Innovative Benchmark: The introduction of the YESBUT benchmark provides a structured approach to evaluating AI’s understanding of humorous contradictions in comics. 2. Human-AI Collaborative Annotation: The use of a human-AI collaborative pipeline for data annotation is innovative and helps in obtaining high-quality annotations efficiently. 3. Insightful Analysis: The analysis of results, including error types and human evaluation, provides valuable insights into the challenges faced by current models in understanding humorous contradictions.
Weaknesses
Novelty: The concept of using VLMs to understand humor in comics has been explored in various forms in previous research. The YESBUT benchmark, though focused on narrative contradictions, might be seen as a slight variation on existing benchmarks that evaluate multimodal understanding and humor recognition. Limitation: The benchmark focuses on a very specific type of reasoning—juxtaposition—which may not be broadly applicable to other forms of humor or narrative understanding. This narrow focus limits the benchmark’s utility in evaluating general AI capabilities in humor comprehension. The task of matching comics with titles involves a high degree of subjectivity. Different annotators might have varying opinions on what constitutes a suitable title, leading to inconsistent and potentially biased evaluations. The dataset consists of only three hundreds of comics, which may limit the generalizability of the findings. Expanding the dataset could provide more robust insights. Annotation: The annotation process, which relies heavily on human-AI collaboration, might introduce bias. Human annotators may inadvertently guide the AI’s outputs, leading to annotations that reflect human reasoning more than autonomous AI understanding. Metrics: The primary evaluation metrics include BERT score, ROUGE-2 (recall), and GPT-based evaluation scores for literal description and contradiction generation tasks. While these metrics are useful for assessing surface-level content generation, they may not fully capture the depth of understanding required for humor comprehension. Suitability: Given the nature of the contribution, the work may be better suited for venues that specifically focus on datasets and benchmarks, such as the NeurIPS dataset/benchmark track and workshops or tracks dedicated to introducing new datasets. This setting would allow the authors to highlight the value of the YESBUT benchmark without the expectation of a significant theoretical or methodological breakthrough.
Questions
How did you address the potential subjectivity and variability in human annotations, especially for tasks like underlying philosophy selection and title matching? Did you measure inter-annotator agreement, and if so, what were the results? --- How did you ensure that the negative titles and philosophies were sufficiently challenging yet distinct from the correct ones? Can you provide examples where models struggled with these distinctions? For example, I feel that the example in Figure 2, the negative title "Graphs Don't Lie, People Do" is also quite suitable to the comic. --- Did you assess the models for any cultural or contextual biases in their understanding of humor? If so, what were your findings, and how do you plan to address these biases in future work? --- Could you provide more details on why you chose BERT score, ROUGE-2, and GPT-based evaluations as your primary metrics? Have you considered using other evaluation metrics that might better capture the nuances of humor understanding?
Rating
4
Confidence
4
Soundness
3
Presentation
3
Contribution
2
Limitations
It's discussed in Sec 8
Dear Reviewer, Thank you once again for your valuable review. With the discussion deadline approaching, we would like to ask whether our response has adequately addressed your questions. If there are any outstanding issues, we would like the chance to respond before the discussion period is over. Thanks again for your thoughtful review! Best regards, Authors
Thanks for your detailed response. Most of my concerns have been well addressed, and I’ve increased my score. Although the dataset scale is still a concern, I acknowledge the challenge to gather large-scale data and thoughtful consideration to manage the diversity of the data samples. Maybe it would be interesting to see how to improve model performance given small-scale data, yet it should be beyond the scope of this pilot benchmark study.
Dear Reviewer, Thank you once again for your review and feedback! We appreciate the increased score and will revise our paper to clarify the previously raised points and incorporate your suggestions. If you have any follow-up questions or concerns, please let us know and we would be happy to answer them. Best Regards, Authors
Responses to new updates
I engaged in the discussion relatively earlier, and I saw a few new messages coming up. So, I read the updates, and I still think it is a nice contribution to a novel task (to examine how LLMs capture a sense of humor given images in juxtaposition). No one has explored this before. Also, the paper shows rich experimental results showing existing limitations and the directions of future work. The work is very sound and the rebuttal has well addressed the minor concerns the original version may have. I see no reason for rejecting the paper despite the unavoitable limitation as a pilot study (if authors can face these limitations and well discuss them).
We sincerely thank the reviewer for their thoughtful and supportive comments. Your recognition of the novelty and value of our work is deeply appreciated and offers strong encouragement for us. We are committed to discussing the limitations and incorporating the suggestions to enhance our work. Once again, we are grateful for your valuable suggestions. Best regards, Authors
Concerns Addressed, Maintaining Positive Assessment
Thank you for your rebuttal. I have carefully read and considered your response to my review. I appreciate the detailed explanations and clarifications you have provided on several key points. Regarding the dataset size, I understand your rationale for the current scale and your plans for future expansion. This addresses my initial concern adequately. I am glad to see you have acknowledged the importance of bias mitigation in the annotation process. Your planned additions to discuss this in more detail will strengthen the paper. Overall, your responses have addressed my main concerns and questions effectively. The proposed additions and clarifications will certainly strengthen the paper. I believe these changes will result in a more comprehensive and impactful contribution to the field. Given your thorough response and planned revisions, I maintain my original assessment of the paper. The proposed work remains technically solid with high potential impact in its sub-area.
Dear Reviewer yTj4, Thank you again for your thoughtful review and suggestions. We appreciate your feedback and will make revisions to clarify the points you raised and incorporate your valuable suggestions. Should you have any further questions or concerns, please don't hesitate to reach out; we would be more than happy to address them. Best regards, The Authors
We thank the reviewer for the thoughtful feedback. We appreciate your acknowledgment of the complexities surrounding the use of social media scraping for research, recognizing it as an ongoing ethical concern that requires careful reflection and clarification. We would like to clarify that we have included a detailed ethics statement in **Section A of our Appendix**. However, recognizing the importance of this issue, we will move the ethics statement to the main paper and revise it to ensure it receives the appropriate emphasis. We would like to highlight that all the images in our benchmark are publicly available, and we have conducted a thorough manual filtering to exclude any data that may contain harmful content. Additionally, all images are used **only for evaluation purposes and not for model training**. When releasing the annotations, we will also include guidelines for benchmark usage to ensure it is **used strictly for research purposes only**. In following, we would like to respond to each point raised by the reviewer: ## Regarding the use of social media scraping for research 1. All the datasets used in our work are publicly available. We use Twitter's API solely to access images for annotation purposes. These images are not stored, posted, modified, or released as part of our dataset; only the annotations are planned to be shared. Our usage aligns with the Twitter Developer Agreement and Policy, which explicitly allows data collection, and is also mentioned in Fiesler et al. [1]. 2. Twitter's Terms of Service (https://x.com/en/tos) grant Twitter the authority to "use, copy, reproduce, process, adapt, modify, publish, transmit, display, and distribute" posted content. This legal framework supports our data handling practices using Twitter's API. 3. Our methodology aligns with established procedures in previous research on comic and meme datasets, as referenced in studies [2, 3]. We strive to ensure both ethical and legal compliance in our data handling. 4. We strictly adhere to copyright laws **by providing only the original links to the comics rather than distributing the images themselves, avoiding any infringement**. Finally, we acknowledge that the ethical implications of using social media data for research remain a topic of ongoing debate. We are committed to addressing this issue responsibly. We will include an acknowledgement and revise our discussion of the ethical considerations related to data scraping as a key component of our benchmark. [1] No robots, spiders, or scrapers: Legal and ethical regulation of data collection methods in social media terms of service. \ [2] Can Large Multimodal Models Uncover Deep Semantics Behind Images? ACL’2024 \ [3] MEMECAP: A Dataset for Captioning and Interpreting Memes. EMNLP 2023 ## Regarding the use of a particular artist's work Thank you for bringing this to our attention and we acknowledge the importance of discussion with the artist. We are currently reaching out to the artist involved in our research project to inform them that we will only be using their images for evaluation purposes, without any modifications to the original work. If the artist has any concerns, we will make it a priority to discuss them and address any issues that arise. ## Regarding potential bias introduced by LLMs or VLMs To address the social biases inherent in LLMs and VLMs, we employ a **rigorous**, dual human-AI collaborative annotation process, which has been highlighted and endorsed by technical reviewers 2fnt and yTj4. We want to clarify that the primary role of these models in our workflow is to assist human annotators, not to replace them. Every output produced by these models **undergoes a thorough verification process involving multiple human annotators**. To further minimize potential biases, we have taken several steps in our annotation process: 1. Diverse Annotator Backgrounds: Our annotators come from different genders and diverse cultural backgrounds, including North America and East Asia, providing a range of perspectives. This diversity helps to mitigate cultural and gender biases in humor interpretation. 2. Consensus Among Annotators: The annotation process incorporates multiple quality checks and verifications to ensure consensus among different annotators. Any contents with controversy and potential bias are filtered out. During the cross verification stages, annotations identified as biased by any annotator are properly modified. This process helps reduce biases stemming from individual perspectives. 3. Verification with Social Media Comments: We also verify our annotations by checking the comments on social media for each comic to ensure that our annotations align with the common interpretation of the comic. Despite these efforts, we acknowledge that our annotations may still carry inherent biases. We will further clarify this and discuss the potential biases in the Limitation section for better guidance of future usage.
Author Response (1/2)
Thank you for your valuable suggestions. We would like to first clarify that we have included a detailed ethics statement in **Section A of our Appendix**. However, recognizing the importance of this issue, we will move the ethics statement to the main body of the paper and revise it to ensure it receives the appropriate emphasis. We would like to highlight that all the data in our benchmark is publicly available, and we have implemented a thorough filtering process to exclude any data that may contain offensive, hateful, or sexual content. Additionally, all images are used **only for evaluation purposes and not for model training**. When releasing the annotations, we will also include terms and conditions for data usage to ensure that our dataset is **used strictly for research purposes only**. Below, we outline the key points and provide further discussion: ## Toward Data privacy, copyright, and consent We acknowledge the importance of this issue. All data samples are sourced from publicly available content on social media platforms, and there are no explicit licensing requirements beyond adhering to the platform's regulations. Specifically: 1. we use Twitter's API solely to access images for annotation purposes. These images are not stored, posted, modified, or released as part of our dataset; only the annotations are planned to be shared. 2. Our usage aligns with the Twitter Developer Agreement and Policy, which explicitly allows data collection. In addition, Twitter's Terms of Service (https://x.com/en/tos) grant Twitter the authority to "use, copy, reproduce, process, adapt, modify, publish, transmit, display, and distribute" posted content. This legal framework supports our data handling practices using Twitter's API. 3. We strictly adhere to copyright laws **by providing only the original links to the comics rather than distributing the images themselves, thereby avoiding any infringement on their copyright**. The comics are used only for evaluating and studying the humor understanding capabilities of VLMs, and the data are not utilized for any model training. When we release the benchmark, we will include the original links for each image. Finally, we recognize the importance of data privacy, copyright, and consent. We are committed to addressing these issues responsibly and will revise our discussion on ethical considerations. We will also provide detailed guidance on the appropriate usage of our annotations when releasing the benchmark.
Author Response (2/2)
## Toward subjectiveness and offensiveness of humor understanding To mitigate the risk of harmful content, we first conduct a thorough manual filtering process to exclude any comics that may contain offensive, hateful, or sexual material. We also acknowledge that GPT-4, in its role as an annotation assistant, may occasionally produce content that could be perceived as offensive. However, it is important to clarify that **GPT-4 does not replace human judgment** in our workflow. Instead, we employ a **rigorous** dual human-AI collaborative annotation process to ensure the quality and appropriateness of the annotations: GPT4 is only used to produce an initial result, which is then revised and verified by human annotators to produce the final annotations. Specifically, **every output generated by GPT-4 is subject to thorough verification by multiple human annotators**. Any content flagged as potentially offensive is promptly modified or removed by the annotators. To further reduce biases and ensure fairness in our annotations, we have implemented the following measures: 1. Diverse Annotator Backgrounds: Our annotators come from different genders and diverse cultural backgrounds, including North America and East Asia, providing a range of perspectives. This diversity helps to mitigate cultural and gender biases in humor interpretation. 2. Consensus Among Annotators: The annotation process incorporates multiple quality checks and verifications to ensure consensus among different annotators. Any comics with controversy and potential bias are filtered out. During the cross verification stages, annotations identified as biased by any annotator are properly modified. This process helps reduce biases stemming from individual perspectives. Despite these efforts, we acknowledge that our annotations may still carry inherent biases. We will further clarify this and discuss the potential biases in the Limitation section for better guidance of future usage of our benchmark. > Please add to the evaluation section … where GPT4 created anything that could be considered as offensively humorous This is a good suggestion! We have conducted additional analysis following [1] to detect offensive content in GPT4-generated literal descriptions and contradictions using the Google banned word list [2]. No offensive outputs were identified. This is likely due to our rigorous filtering of comics with potentially harmful content before feedforwarding into models. Additionally, due to the time limitations, we have conducted a follow-up human evaluation by asking annotators to further evaluate the 30 random samples used in our human evaluations (as shown in Figure 3), and no offensive content was detected. We will include these in the evaluation section. In conclusion, we greatly appreciate your valuable feedback. We will revise our ethical statement section to include more detailed discussions on these issues and incorporate your suggestions into our revised version. [1] MEMECAP: A Dataset for Captioning and Interpreting Memes \ [2] https://github.com/coffee-and-fun/google-profanity-words
Thanks for the detailed response. I still have many concerns about this paper. - I still feel the novelty of the paper is quite limited. The ACL best paper [1], published in 13 Sep 2022 on arxiv, already have started human understanding from cartoon. And this submission just change it two a specific, narrow-domain situation: presenting a pair of pictures (i.e., Juxtaposition). Although this is a nice extension for studying VLMs' ability of reasoning on two images instead of one, such reasoning abilities of VLMs have already been studied with many other multi-image VQA tasks for a long time. I don't think the creation process of this dataset is novel neither. The authors also claim that the major difference from this submission to [1] is just from reasoning on single image to a pair of images. - the tasks are subjective by nature so a larger set of annotators per example is need but authors do not provide inter-human annotator agreement. In the example that I mentioned "The negative title "Graphs Don't Lie, People Do" in the example of Figure 2". I think this can already show the problem -- you think it is not "appropriate" but I feel it is --- intentionally showing partial or chosen factual information is a form of lie. We may not need to agree with each other on this, but this suggests that the authors did not handle the subjectiveness of the dataset very well. - concerns on the cultural bias is still there. the authors did not explain well what they did to ensure the content in the dataset only contain culture-insensitive tasks. - concern on the metric: "Metrics: The primary evaluation metrics include BERT score, ROUGE-2 (recall), and GPT-based evaluation scores for literal description and contradiction generation tasks. While these metrics are useful for assessing surface-level content generation, they may not fully capture the depth of understanding required for humor comprehension." The authors' response in the general rebuttal does not address this point very well. To sum up, I think the dataset is fun and interesting and may be a good contribution, but in my humble opinion it is a bit less suitable for the main track at NeurIPS. I have also read the comments from other reviewers. Based on my concerns, I'd like to lower my rating. [1] Do androids laugh at electric sheep? humor “understanding” benchmarks from the new yorker caption contest
We thank Reviewer 2fnt’s constructive reply and we regret to see the score decrease. While we appreciate the reviewer's efforts, we would like to briefly clarify a few points in the Reviewer 2fnt’s reply in case of any potential misunderstandings. > The novelty of the paper is quite limited & nonlinear reasoning abilities of VLMs have been studied for a long time We would like to clarify that our work is not to “**just from reasoning on single image to a pair of images**”. Instead, our focus is on understanding complex, nonlinear narratives that emerge through the juxtaposition, particularly in contexts that involve abstract, human-centered events as depicted in comics. This is important for VLMs, which have been shown to struggle with deep, nonlinear reasoning by recent work. By exploring how VLMs understand contradictions and abstract social concepts through sophisticated reasoning processes, we aim to uncover insights that are critical for the advancement of these large models and can guide future developments in the field. > authors do not provide inter-human annotator agreement We would like to clarify that our annotations are not generated by models, but rigorously produced by human annotators with the assistance of GPT4. Each sample is annotated, verified, and checked by **at least three annotators** (constituting 43% of our total annotator pool) to ensure that each annotation achieves a high degree of consensus. Therefore, it is not feasible for us to compute inter-human annotator agreement because annotators do not produce annotations **individually and independently**. > The example suggests that the authors did not handle the subjectiveness of the dataset very well We formalize title understanding as a “selection task” where the positive title is deemed more appropriate based on *common* interpretations, instead of a “binary classification” task about right or wrong. For this specific example, the positive title is “commonly” considered to be better than the negative options. > concerns on the cultural bias Cultural bias often arises from varying interpretations across different cultural backgrounds. To reduce such bias and focus on common interpretation, in the annotation process each sample is **verified by multiple annotators from different cultural backgrounds**. Their **consensus** was required for an annotation to be included; if no consensus was reached, the annotation was either modified or excluded from the dataset. This approach ensures that we retain only samples that minimize cultural bias. While this approach significantly reduces bias, it is important to acknowledge that no dataset can completely eliminate bias. We recognize that our annotator pool may not represent every cultural perspective, and we are committed to including a more detailed discussion of this limitation in our revised paper. > concerns on metric Our use of BERT score, ROUGE-2 (recall), and GPT-based evaluation scores is designed to assess whether the model-generated outputs accurately capture the key information of the narratives. Achieving this level of generation inherently needs “*the depth of understanding required for humor comprehension*”. While these metrics may appear surface-level, they are effective proxies for measuring the model's comprehension, as they ensure that the outputs reflect a nuanced understanding of the comic. Again, we thank Reviewer 2fnt for providing constructive comments and suggestions. We also thank Reviewer 2fnt for the reply of our rebuttal. We will revise the paper to better incorporate these insights and clarify the unclear parts.
Decision
Accept (oral)