MDPE: A Multimodal Deception Dataset with Personality and Emotional Characteristics

Thank you very much for your guidance and valuable suggestions. We are very grateful for your comments, which are very helpful for us to sort out the structure and motivation of the article.  1. We acknowledge that our dataset, which includes only Chinese speakers, may limit its generalizability to other cultures and languages. To adapt the MDPE dataset for future research in different linguistic and cultural contexts, Future studies could involve collecting similar datasets from diverse cultural backgrounds, allowing for comparative analyses that explore how cultural factors influence deception detection. In fact, we have started collecting and processing an extended version of MDPE data, which will greatly alleviate bias issues caused by population and cultural differences.  2. In fact, we have a description of population statistics (section 3.2). In the updated manuscript, we provide a more detailed introduction to the detailed demographic distribution of participants, taking into account factors such as age, gender, and other relevant characteristics, in order to have a clearer understanding of the composition of the dataset.  3. We recognize the need for a deeper analysis of how emotional characteristics influence deception. In the revised manuscript, we will expand our discussion to include: A thorough examination of the emotional traits present in the dataset and their potential impact on deception detection. And emotion recognition is presented with multi-labels.  4. Regarding the relevance of personality and emotion in deception detection across different demographics, we believe that while the underlying psychological constructs may remain relevant, their expression and interpretation can vary across cultures. We will address this point in the updated manuscript, emphasizing the importance of cultural considerations in future research.  5. We appreciate your suggestion to explore additional multimodal modules for future research. In the revised manuscript, we will include a discussion of potential multimodal approaches that could enhance our understanding of deception detection.  6. While the primary focus of this study is deception detection, we acknowledge the creativity in the proposed comparison with other emotion recognition or personality detection models. We believe such comparisons could serve to validate the performance of personality and emotional traits in our context. Consequently, we will include a comparative analysis section in the new manuscript, assessing the performance of our approach against existing models.   We hope that these revisions address your concerns and strengthen the contribution of our paper. We are grateful for the opportunity to improve our work based on your valuable feedback. Thank you again for your comments and acceptance of our views.

Thank you very much for your guidance and valuable suggestions. We are very grateful for your comments, which are very helpful for us to sort out the structure and motivation of the article.  In this study, we introduce a comprehensive multimodal deception detection dataset that encompasses visual, audio, and textual data, along with personality and emotional characteristics. This dataset is the largest of its kind and offers a unique opportunity to explore how personality and emotional traits can augment deception detection. Our work not only lays the groundwork for future research in affective computing but also suggests potential avenues for improving emotion recognition through the integration of personality insights and enhancing personality recognition performance with emotional cues. We have conducted extensive experiments that preliminarily demonstrate the value of incorporating personality and emotional traits in deception detection tasks. While we acknowledge that our application of machine learning methods may not be as advanced as desired, we believe that deception detection is inherently challenging, and the performance of various machine learning approaches has been modest. This indicates a need for further in-depth feature exploration within this dataset by the research community. Consequently, we have chosen to present a straightforward machine learning approach as a baseline for our analysis.  We apologize for the typos and organizational issues in the previous version of the manuscript. We have thoroughly proofread the paper and corrected all identified typos. Additionally, we have restructured Section 4.2 to improve clarity and readability. We believe these changes will make our methods more accessible and better understood by the readers.  The 24 questions used by the interviewer for specific topics were specifically designed for this study (line 215), 5 psychology researchers mainly referred to the Fraud Triangle Theory and Rational Choice Theory. The question design integrated knowledge from psychology, criminology, and sociology to comprehensively capture multiple aspects of fraudulent behavior. We used the Delphi method to collect and integrate expert opinions, and refined the issues through focus group discussions. The design of some issues reflects consideration of the new trends in current fraudulent behavior, aiming to capture aspects that traditional methods may overlook. In fact, after designing the initial interview questions, we conducted pre experiments, collected feedback from interviewees, and revised the interview questions after further discussion, the final interview questions were actually the third version.  We appreciate your suggestion of alternative conferences such as ICMI, CSCW, and ACM-MM. While we believe our work is still relevant for this conference, we will consider your suggestion for future submissions if our work is not accepted here. We hope that these revisions address your concerns and strengthen the contribution of our paper. We are grateful for the opportunity to improve our work based on your valuable feedback. Thank you again for your comments and acceptance of our views.

Thank you very much for your guidance and valuable suggestions. We are very grateful for your comments, which are very helpful for us to sort out the structure and motivation of the article. 1. In the revised manuscript, we have condensed the "Introduction" section to enhance conciseness and shifted detailed background information, particularly from the second and third paragraphs, to the "Related Work" section. This reorganization improves the flow and maintains focus within the introduction. 2. We have clarified that the "effective incentives" mentioned (line 76) are monetary in nature and have provided details on their allocation (line 224) to effectively motivate participants in deceptive behavior. Additionally, we have meticulously revised the manuscript to address all linguistic inconsistencies and spelling errors, including those in Table 1, ensuring the accuracy and integrity of the presented data. 3. At the commencement of section 3.3, we have included a statement delineating the sequence of data collection activities. Participants initially complete the Big Five personality questionnaire to ascertain personality traits. Subsequently, half of the participants engage in emotional experiments while the other half partake in deception detection experiments. This balanced approach to experimental sequencing is intended to mitigate any potential influence of the sequence on the outcomes. 4. We have corrected the abbreviation "DDC" to "DCC" and offered a clear definition to prevent any ambiguity. 5. Furthermore, we have expanded upon the roles of the interviewers, detailing their background and responsibilities throughout the data collection process, thereby underscoring their pivotal role in our research. 6. To ensure participant anonymity, we have provided explicit details regarding the measures taken to remove or alter identifying information. The experimental videos provided for analysis exclude any segments containing personal details such as names, contact information, place of residence, education, occupation, and so forth. 7. The statement regarding CEVS videos being "outdated" and unable to effectively induce corresponding emotions refers to content that is no longer relevant or relatable to contemporary viewers. For instance, some videos intended to elicit happiness contained jokes that required historical context unfamiliar to most participants. As a result, these videos were replaced. 8. The 22 online videos were selected and annotated by 12 data annotators based on the CEVS criteria and evaluation methods (line 205). For further details, please refer to [1]. 9. In section 4.4, we have revised the second paragraph to elucidate the structure of the specialized emotion recognition models and the specific emotional cues they utilize. We have also detailed the role of personality and emotional traits in fraud detection tasks. 10. The 24 interview questions were meticulously designed for this study (line 215), with input from five psychology researchers drawing primarily from the Fraud Triangle Theory and Rational Choice Theory. The questions were crafted to integrate knowledge from psychology, criminology, and sociology, aiming to comprehensively capture various facets of fraudulent behavior. The Delphi method was employed to gather and synthesize expert opinions, and the questions were refined through focus group discussions. Some questions were designed to reflect current trends in fraudulent behavior, capturing aspects that traditional methods might overlook. After an initial design phase, we conducted pre-experiments, collected feedback from participants, and revised the interview questions accordingly, resulting in the final set being the third iteration. 11. We have thoroughly revised the manuscript to enhance writing quality and address all language issues, including uncertainties and unclear descriptions as highlighted by your review.  We hope that these revisions address your concerns and strengthen the contribution of our paper. We are grateful for the opportunity to improve our work based on your valuable feedback. Thank you again for your comments and acceptance of our views.

Thank you very much for your guidance and valuable suggestions. We are very grateful for your comments, which are very helpful for us to sort out the structure and motivation of the article.  1. The concept of 'true emotional expression' pertains to the congruence between an individual's outward emotional display and their internal affective state. Our emotion induction procedure establishes a baseline for emotional responses, which is crucial for comparison with the emotional responses observed during the deception experiment. Experimental results have indicated that emotional features significantly contribute to fraud detection tasks. 2. In this study, we present a comprehensive multimodal deception detection dataset, encompassing visual, audio, and textual data, as well as personality and emotional features. This dataset, being the largest of its kind, offers a unique opportunity to investigate how personality and emotional traits can enhance deception detection. Our work not only lays the foundation for future research in emotion computing but also suggests potential methods to improve emotion recognition by integrating personality insights and enhancing personality recognition performance with emotional cues. Through extensive experiments, we have preliminarily demonstrated the value of integrating personality and emotional traits in deception detection tasks. While our application of machine learning methods may not have achieved the desired level of advancement, we believe that deception detection is inherently challenging, with various machine learning methods showing limited performance. This suggests a need for the research community to further explore the features and methods within this dataset. Thus, we propose a simple machine learning method as a baseline for our analysis. In the revised manuscript, we have conducted a more in-depth analysis of the impact of machine learning on new datasets, including remaining challenges and insights into the limitations of current architectures, and how this dataset can contribute to advancing research in this field. 3. In the updated manuscript, we have included information on the public availability of existing datasets. 4. Building upon the six emotions covered by the Chinese Emotional Video System (CEVS)—happiness, sadness, anger, fear, disgust, and neutrality—we have added relaxation and surprise to our dataset. Relaxation serves as an emotional baseline, allowing for the comparison of other emotional states. Deviations from a relaxed state during specific questions or situations may indicate deception, as inconsistent relaxation levels could suggest an attempt to deceive. Surprise, a spontaneous and difficult-to-fake emotion, can indicate truthfulness or deception; a genuine surprise response to an unexpected question or accusation may suggest unpreparedness and honesty, while a lack of surprise may indicate a prepared deceptive response. 5. Our experiments were conducted at least five times with randomly selected samples, and the average results were taken to ensure statistical significance. 6. Regarding ethical issues, we ensured that all experimental procedures were explained to the subjects, who provided explicit consent for the recording and publication of their conversation and video data in scientific conferences or journals. Our data collection and dissemination adhere to the principle of informed consent and comply with relevant laws, regulations, and ethical review requirements, all approved by our institution's Human Subjects Institutional Review Board. We have implemented privacy protection measures, did not publish any personally identifiable information, and restricted dataset access to users who agree to use it solely for scientific research. The release of our deception dataset aligns with international standards and industry best practices, positively impacting scientific research, technological progress, and public safety. We have added a new section in the revised manuscript to address these ethical considerations, ensuring a comprehensive and responsible approach to handling the deception detection dataset. 7. We have thoroughly revised the manuscript to improve the quality of writing and address all language issues, including specific errors such as "role.Although," "new multi-modal deception dataset," and reference formatting, as pointed out by the reviewers.