Response to Reviewer Yjoc (1/N)
Thank you for your recognition of our dataset, **ShareChatX**, our writing, and our contributions to the field of spoken dialogue systems. Please allow us to address your questions in detail:
**Q1: Evaluation Benchmark (Weakness 1)**
**A1:**
Yes, we specifically included experiments on DailyTalk to evaluate the model’s performance in out-of-domain and real-world scenarios. Testing on out-of-domain datasets like DailyTalk demonstrates that OmniChat’s performance does not rely on in-domain data but instead reflects strong generalization capabilities.
Regarding information retrieval tasks, this is indeed an excellent idea, and we initially considered including such tasks. However, compared to ASR-based models, the most distinctive feature of spoken dialogue systems lies in their ability to understand acoustic information that cannot be conveyed solely through text in complex conversational scenarios (e.g., emotion, audio, and music). Thus, we ultimately decided to focus our research on this more challenging and open-ended problem (How to help spoken dialogue model understand acoustic information). That said, we believe that training with synthetic data can effectively enhance the information retrieval capabilities of spoken dialogue systems.
**Q2: Evaluation Metrics (Weakness 2)**
**A2:**
Thank you for your discussion on evaluation metrics. For our metrics, we referred to established baselines in spoken dialogue tasks [1,2], which commonly use BLEU, ROUGE-L, and BERTScore. However, we were concerned about the ability of these metrics to fully capture spoken dialogue capabilities, so we supplemented them with GPT-based metrics and human evaluations.
Please note that ShareChat-X data is entirely AI-synthesized. We refrained from testing GPT-eval on ShareChat-X initially due to concerns about the potential alignment of evaluation results with the LLM’s inherent biases (i.e., “rating its own answers”), which could reduce the evaluation’s objectivity.
**Q3: How does each type of synthetic data individually affect model performance? (Weakness 3)**
**A3:**
Please refer to Section 5.6 and Figure 4, where we analyze the performance of models trained on subsets such as ShareChat-Emotion and ShareChat-Audio. These results demonstrate the importance of spoken dialogue data (e.g., ShareChat-Emotion) for specific scenarios and the value of special-purpose dialogue data (e.g., ShareChat-Audio and ShareChat-Music).
**Q4: Model and Experimental Settings (Weakness 4)**
**A4:**
We provided detailed experimental settings in Section 5.1 and Appendix Section C. Additionally, we will release all code after the paper’s acceptance. If you find any missing experimental details, please let us know, and we will provide further clarifications.
**Q5: Real-Time Performance (Weakness 5)**
**A5:**
This work focuses on how synthetic data can enhance spoken dialogue systems during training. Our data synthesis methods and conclusions can benefit the training of various systems, including those aiming for either high performance or fast response times.
The model proposed in this work is designed to validate the impact of synthetic data on performance and follows a simple architecture similar to SALMONN, without optimizing for complex real-time processing.
**Q6: Pretrained Model Versions? (Question 1)**
**A6: Whisper**: Whisper large-v3. **Emotion2Vec**: Emotion2Vec-large. **BEATs**: Fine-tuned BEATs_iter3+ (AS2M) (cpt2).
**Q7: Human Validation and Evaluation Methods (Question 2)**
**A7:**
**Data Validation Process**:
After generating all synthetic data, we randomly sampled 2,000 examples from each dataset subset to check if the generated results met human standards. We found that 97% of the data adhered to logical human dialogue standards.
**Human Evaluation Process**:
We invited 10 evaluators to rate all generated responses on a scale of 1 to 5 based on the following criteria:
**1**: Contextually irrelevant and emotionally inappropriate.
**2**: Partially relevant to the context but feels unnatural, with emotionally inappropriate expression.
**3**: Partially relevant to the context, with emotionally appropriate expression.
**4**: Contextually relevant and natural, mentioning the attribute, with emotionally appropriate expression.
**5**: Contextually relevant, smooth, natural, and with emotions that perfectly align with the scenario.
The final score for each response was obtained by averaging the scores from all evaluators.