Digital Memes have been widely utilized in people’s daily lives over social media platforms. Composed of images and descriptive texts, memes are often distributed with the flair of sarcasm or humor, yet can also spread harmful content or biases from social and cultural factors. Aside from mainstream tasks such as meme generation and classification, generating explanations for memes has become more vital and poses challenges in avoiding propagating already embedded biases. Our work studied whether recent advanced Vision Language Models (VL models) can fairly explain meme contents from different domains/topics, contributing to a unified benchmark for meme explanation. With the dataset, we semi-automatically and manually evaluate the quality of VL model-generated explanations, identifying the major categories of biases in meme explanations.
Paper
Full text
Multimodal Understanding of Memes with Fair Explanations
Semantic Scholar · Computer Science · 2024
Abstract
Digital Memes have been widely utilized in people’s daily lives over social media platforms. Composed of images and descriptive texts, memes are often distributed with the flair of sarcasm or humor, yet can also spread harmful content or biases from social and cultural factors. Aside from mainstream tasks such as meme generation and classification, generating explanations for memes has become more vital and poses challenges in avoiding propagating already embedded biases. Our work studied whether recent advanced Vision Language Models (VL models) can fairly explain meme contents from different domains/topics, contributing to a unified benchmark for meme explanation. With the dataset, we semi-automatically and manually evaluate the quality of VL model-generated explanations, identifying the major categories of biases in meme explanations.