Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning

Summary

This paper implements multimodal emotion recognition and reasoning by fine-tuning the LLaMA model with instructions. It is trained on a large-scale dataset, fine-tuned, and tested on three datasets.

Strengths

The global, temporal and local features of the video modality are considered, and LoRA fine-tuning (all tokens), Prompt fine-tuning (text modality) and supervised fine-tuning (linear layer) are used at the same time.

Weaknesses

The innovation is limited and only existing methods are used. The key problems are not solved.

Questions

1. The complex interaction relationship between modalities is not considered. 2. The process is complicated, and the computing resource requirements are high. 3. The effect of Prompt Tuning and Instruction Tuning is highly dependent on the design of prompt words and instructions. If the prompt words or instructions are not accurate or comprehensive enough, it may impact the performance of the model. 4. The lack of reproducibility: the anonymous Github link provided by the author is empty.

Rating

5

Confidence

4

Soundness

2

Presentation

2

Contribution

1

Limitations

Yes

Summary

The paper introduces a multimodal large language model, named Emotion-LLaMA for emotional state understanding. The authors use open-source tools to collect and annotate a dataset, named MERR for model pre-training. Then they perform instruction-tuning on downstream datasets for emotion recognition and emotion reasoning. Extensive experiments are conducted and demonstrate the promising performance of the proposed approach.

Strengths

1. Extensive experiments are conducted and the method achieves SOTA performance on various dataset for emotion recognition and reasoning. 2. Straightforward visualizations are presented.

Weaknesses

1. The paper claims the MERR dataset as a core contribution. However, there is no systematic evaluation for the label quality of the dataset. I understand that pre-training on this dataset improves the downstream performance and thus its validity can be shown to some extent. However, this may be due to the diversity of the unlabeled data instead of the automatically generated label. 2. Table 3, with audio and video inputs, Emotion-LLaMA’s performance is worse/close to the baseline (VAT).

Questions

1. How do you pre-train Emotion-LLaMA? Is it supervised learning with coarse-grained emotion labels? 2. Do you plan to release the MERR dataset? 3. Table 2 fine-tuning, why does Emotion-LLaMA get worse performance on Disgust than MAE-DFER and VideoMAE?

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Please refer to weaknesses

Summary

The paper presents a new multi-modal instruction tuning dataset for emotion recognition. The authors also present results of training on this dataset with a multi-modal architecture based on LLaMa 2. They show evaluation results on DFEW and MER2023.

Strengths

Evaluating multi-modal emotion recognition approaches based on LLMs are a very relevant research topic. The results shown by the authors are convincing w.r.t. the performance on emotion recognition datasets.

Weaknesses

------------- The argument presented in lines 31 and 33 is not convincing. Why does an inability in methods lead to a lack of datasets? Related Work: ------------- As the dataset is one of the claimed contributions of the paper, there should be a discussion on previous emotion recognition datasets. It is important to lay out the reasons why these previous datasets cannot be used (or not easily be used) for instruction tuning of LLMs. Methodology: ------------ What videos is the MERR dataset based on? I could not find details in the paper on the selection process. Judging from the screenshots, it seems to be movie clips. This has important implications on the concept of "emotion" that is addressed in the paper and needs to be clarified. The chosen methodology lacks justification. For example, what is the reasoning behind selecting the frames with the highest sum of AU activations across all AUs included in OpenFace? It seems to me that there would be the danger of having a strong bias towards moments when the person is speaking, as this often leads to high AU activations, especially AUs related to the lower half of the face. The mapping of AUs to facial expression labels needs more explanation. E.g. is "happy" assigned if the combination of AU06, AU12, AU14 is active, or is it also assigned if only some of those AUs are active? Figure 1 does not show a facial expression description according to Table 7 at all, only descriptions according to Table 8. Concerning Table 8, there are several descriptions given for each AU. Are they all used at the same time, or is only one of them chosen? In Figure 1 it appears that only one of them is chosen, but how is this decided? Several further steps are not clearly defined, e.g. how does LLaMA-3 "refine" the annotations by aggregating the multimodal description? How is the instruction following data constructed of which a single example is presented in Table 9. Is it done manually? If yes, how and how many samples are created? In 3.1 it is not clear, how the dataset is "auto-annotated" with emotion labels. It is also not clear how these annotations are refined (with human involvement)? In general the concept of "emotion" used in the paper remains unclear. Is it about emotional displays, about internal states,...? In later parts of the method section it seems authors are targeting (internal?) emotional states. It would be important to know how they were annotated. Do the instructions for multimodal emotion recognition (Table 11) refer to different tasks? Some of these instructions seem to target displayed emotions, some target internal states. Training details are unclear. With the description provided in the paper it is difficult to understand the training procedure. Evaluation: ----------- The authors employ chatGPT to evaluate emotion reasoning. It is not clear to what extent this approach leads to a valid evaluation. E.g. ChatGPT was shown to be biased on emotion-related tasks [1]. In the end, evaluating the emotion understanding capabilities of one language model using another language model is circular. The impact of using the proposed MERR dataset for pre-training needs to be evaluated. [1] R. Mao, Q. Liu, K. He, W. Li, and E. Cambria, “The biases of pre-trained language models: An empirical study on prompt-based sentiment analysis and emotion detection,” IEEE Transactions on Affective Computing, 2022.

Questions

-

Rating

3

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

Limitations are not discussed in enough detail. There is no separate limitations section. The authors also do not lay out the limitations concerning e.g. (citing from the checklist): "The authors should reflect on the scope of the claims made, e.g., if the approach was580 only tested on a few datasets or with a few runs. In general, empirical results often581 depend on implicit assumptions, which should be articulated." "The authors should reflect on the factors that influence the performance of the approach.583 For example, a facial recognition algorithm may perform poorly when image resolution584 is low or images are taken in low lighting. Or a speech-to-text system might not be585 used reliably to provide closed captions for online lectures because it fails to handle586 technical jargon."

Summary

The paper presents the Emotion-LLaMA model, a multimodal emotion recognition and reasoning system that integrates audio, visual, and textual inputs. - The authors constructed the MERR dataset, which includes 28,618 coarse-grained and 4,487 fine-grained annotated samples across diverse emotional categories, enabling models to learn from varied scenarios and generalize to real-world applications. - The Emotion-LLaMA model incorporates specialized encoders for audio, visual, and textual inputs, aligning the features into a modified LLaMA language model and employing instruction tuning to enhance both emotional recognition and reasoning capabilities. - Extensive evaluations show that Emotion-LLaMA outperforms other multimodal large language models, achieving top scores on the EMER, MER2023, and DFEW datasets. The main contributions are: - The MERR dataset, a valuable resource for advancing large-scale multimodal emotion model training and evaluation. - The Emotion-LLaMA model, which excels in multimodal emotion recognition and reasoning through the innovative use of instruction tuning. - Establishing Emotion-LLaMA as the current state-of-the-art model in public competitions for multimodal emotion analysis.

Strengths

- The paper is well structured and well presented. The author organizes the article into five sections: introduction, related work, methodology, experiments, and conclusion. They clearly describe how they conduct data annotation and model design, provide a good introduction to the experimental setup and analysis of the experimental results, and present a relatively clear conclusion. - Model details are thorough. The authors have provided a good description of their model and training methods, allowing me to clearly understand how the model is designed and trained. I believe their results are reproducible. - This work is valuable . The authors provided a paradigm for emotion annotation of multimodal data and an annotated dataset. They also offered a clear explanation of the annotation process, which will contribute to the development of the related field.

Weaknesses

- Insufficient experiments: Although the author has conducted some comparative and ablation experiments, it is obviously insufficient for such a complex multimodal LLM. - Missing details of experimental setup: The authors mentioned that they fine-tuned on several target datasets, but the details of the fine-tuning (including data volume, dataset division, and fine-tuning setup) were not included in the article. This can lead to a decrease in the credibility of their results. - Lack of result analysis: Although the proposed model surpasses existing models in many metrics, the authors only list their experimental results in the results section without further analysis and explanation of the results and some phenomena. The lack of proper explanation and analysis can make some results confusing.

Questions

1. In section 4.2, the author mentioned that they used the HuBERT-Chinese large model for audio modality input processing. Regarding this part, I would like to know the following questions: - Are the experimental results sensitive to language? I hope the author can provide more experimental results to illustrate this point. - In the ablation study section (Tab5), the author seems to have only conducted ablation on the visual encoder. Why didn't they further conduct ablation experiments on the audio encoder, considering that there are many alternatives to Hubert? - Based on the previous question, does the author believe that the audio modality is not important in this task? I would like to see more ablation results on the modality scale. 2. As I mentioned in the weaknesses part, could the author provide more details of fine-tuning on the target datasets? 3. Is Multimodal Emotion Recognition a classification task? How do the authors explain the Dis column in Table 2?

Rating

4

Confidence

3

Soundness

3

Presentation

3

Contribution

4

Limitations

Although the author mentioned in the checklist that they discussed the limitations of the article, they did not explicitly discuss them in the text. Meanwhile, I noticed that the author did not mention their data sources in the article, and I am concerned whether this might involve data copyright issues.

Response to Ethics Review

Thank you for your detailed and insightful review. Below, we address each of your concerns and outline the actions we have taken or plan to take in the revised manuscript. **Data Privacy and Consent:** The MERR dataset is sourced from the MER2023 dataset, consisting of over 70,000 unannotated samples from publicly available movies and TV series. We have signed the necessary End User License Agreements (EULA) and received permission from the original data providers. Since this data originates from publicly available media, and participant consent was obtained during the original production, additional consent for our research is not required. We will also contact the original dataset's authors to ensure compliance with ethical standards. A section will be added to the appendix detailing the video data licensing and participant consent context. **Fairness and Bias:** The MERR dataset is designed to cover a wide range of multimodal emotional cues while avoiding discriminatory content related to sensitive attributes. We acknowledge the potential for inherent biases and have implemented measures to assess and mitigate them. We provided detailed scoring criteria and results in Tables 14 and 15, confirming the absence of significant biases. We used ChatGPT as a neutral evaluator, focusing on textual annotations to reduce bias related to visual or demographic factors. Moving forward, we will explore additional validation methods to ensure fairness. **Reproducibility and Transparency:** The open-source MERR dataset includes automatically generated emotion description JSON files, excluding original video content due to licensing restrictions. Researchers can access the original video data by applying directly to the MER2023 providers and signing the EULA. The revised manuscript will clearly outline the terms of use and provide detailed reproduction instructions. Our GitHub repository now includes all necessary scripts, documentation, and a README file to guide the reproduction process, with access restricted to research purposes only. **Broader Impact:** We have reviewed the MERR dataset and Emotion-LLaMA outputs and found no evident discrimination or bias. However, we recognize the potential risks of reinforcing societal biases in real-world applications. We will investigate these impacts further and discuss them explicitly in the revised manuscript. Future work will incorporate advanced bias detection techniques and input from ethics and social sciences experts. In summary, we believe the steps outlined address the ethical concerns raised. We are committed to ensuring our research adheres to ethical standards and will incorporate these considerations into the revised manuscript. Thank you for your thoughtful feedback.

Detailed Response to Reviewer iNoC's Feedback (Part-1)

Dear Reviewer iNoC, Thank you for your detailed feedback and for engaging with our work on Emotion-LLaMA. We acknowledge that your additional questions and concerns were raised just a few hours before the rebuttal deadline. Unfortunately, due to these time constraints, we were unable to provide a comprehensive response at that time. We greatly appreciate the opportunity to now address each of your points thoroughly. --- **Q2: DialogueLLM uses MELD, IEMOCAP, and EmoryNLP. All these three datasets are not mentioned in the author's response but have been used for instruction tuning LLMs.** **A2:** It appears there may have been some confusion regarding the datasets used in our work. To clarify: 1. **Task Mismatch:** The datasets MELD, IEMOCAP, and EmoryNLP are utilized by DialogueLLM for emotion recognition in conversations (ERC), focusing on the text modality. However, our work, Emotion-LLaMA, is fundamentally different as it emphasizes multimodal emotion recognition and reasoning by integrating audio, visual, and textual features to provide a far more comprehensive analysis of emotional states. 2. **Requirement Mismatch:** The instruction data used in DialogueLLM includes only emotion classification labels, lacking the depth and richness required for a nuanced understanding of emotions. Our MERR dataset, in contrast, provides instructions that include detailed multimodal emotional descriptions, enabling Emotion-LLaMA to analyze both external emotional expressions and internal psychological states with significantly greater accuracy. 3. **Modality Mismatch:** DialogueLLM’s input is limited to textual data, with no incorporation of audio or video modalities, as referenced in the DialogueLLM paper. This is a critical limitation that our approach directly addresses by incorporating comprehensive multimodal data into the instruction-tuning process, filling an important gap in current methodologies. --- **Q3: The selection procedure is still unclear - i.e., how was the dataset sampled from MER2023?** **A3:** We have previously addressed the question regarding the videos used in the MERR dataset in response to Q3-A3, where we provided detailed information about the video sources. We will now address the new question regarding the dataset sampling procedure from MER2023 and MERR: 1. The samples from the MER2023 dataset, which includes movies and TV series, were selected by cropping based on timestamps from the subtitles. The data collectors used a variety of open-source tools, such as YuNet and face.evoLVe, to ensure that each visual frame contained only one person. However, the dataset also includes a significant number of unlabeled samples that require further exploration. 2. The MERR dataset is derived from the MER2023 dataset. Initially, we filtered the samples based on the activation of specific Action Units to obtain coarse-grained descriptions. We then used LLaMA-3 to generate fine-grained descriptions, which were reviewed by emotional experts. This process resulted in 4,487 detailed annotated samples. --- **Q4 & Q5: No references are given for the supposed connection between AUs and emotion expression. It is unclear how stable this connection is in different contexts.** **A4 & A5:** The connection between Action Units (AUs) and emotion expression is not speculative; it is well-established and supported by extensive research: 1. **Validated by Research:** The Facial Action Coding System (FACS) has extensively validated the relationship between AUs and emotions, and this connection is considered stable and reliable across various contexts. This is a foundational aspect of emotion research. 2. **Practical Tools:** In practical applications, tools like OpenFace are widely used to extract AU information for precise emotion analysis, reinforcing the reliability of this connection in diverse scenarios. 3. **References:** To further substantiate this, we will include the following references in our revised manuscript: - [7] Facial Action Coding System, Environmental Psychology & Nonverbal Behavior, 1978. - [8] Affectiva-MIT Facial Expression Dataset (AM-FED): Naturalistic and Spontaneous Facial Expressions Collected In-the-Wild, CVPR 2013. - [9] Multi-Task Learning of Emotion Recognition and Facial Action Unit Detection with Adaptively Weights Sharing Network, ICIP 2019.