Thank you for reviewing our work, highlighting critical issues over various dimensions of the proposed dataset and benchmark, and giving precious suggestions. Although we have not finished revision paper writing yet, we just want to provide detailed responses to the concern about issues in the following:
$\textbf{Dataset Details}$ We get to realize the importance of complete case study for a new dataset. At first for article length limitation we only put part of that in appendix, but as reference we put two of the cases here.
Following is the first case of evaluation data in our dataset, containing two query (prompt and golden response as one query data).
$\text{
"from":"A", "to":"B", "text":"Hey, did you see what happened to John? He just fell during the game.";}$
$\text{ "from":"B", "to":"A", "text":"Yeah, I saw. He's on the ground and doesn't seem to be moving.";}$
$\text{"from":"C", "to":"D", "text":"Should we call an ambulance? I think it's serious.";}$
$\text{"from":"D", "to":"C", "text":"I'm already on it. I've called the campus emergency number.";}$
$\text{"from":"E", "to":"A", "text":"I have some first aid training, let me check his pulse.";}$
$\text{"from":"F", "to":"B", "text":"Is there anything I can do to help?"}$
Here we have golden response form LLM agent be $\text{"from":"Agent", "to":"C", "text":"C, since you've initiated the emergency call, I will assist you in coordinating with the emergency services."}$ while limited generated response be $\text{"from":"Agent", "to":"F", "text":"F, thank you for offering help. Could you please find a blanket for John to keep him warm?"}$ for ensuring that the most time-sensitive and important information is addressed promptly.
Similarly following the above conversation history we have
$\text{"from":"A", "to":"E", "text":"E, please let us know how he is as soon as you can.";}$
$\text{"from":"B", "to":"F", "text":"F, could you find a blanket or something to keep him warm?";}$
$\text{"from":"C", "to":"Agent", "text":"Agent, they said the ambulance will be here in 5 minutes."}$ as background conversation and here we set $\text{"from":"Agent", "to":"E", "text":"E, while you're checking John's vitals, please also ask if he's conscious and if he can feel his limbs."}$ as golden response rather than generated response $\text{"from":"Agent", "to":"A", "text":"A, please stay calm and keep an eye on John until the ambulance arrives."}$.
For more detail here we have another explicit example case in class meeting scene:
$\text{"from":"ExcellentParents", "to":"Agent", "text":"Hello Agent, I'd like to share some insights on how I helped my child become an excellent student."}$
$\text{"from":"AverageParents", "to":"Agent", "text":"Agent, I'm worried about my child's grades. Can you give me some advice?"}$
$\text{"from":"Teacher", "to":"Agent", "text":"Agent, I have some classroom strategies that could help improve student performance."}$
$\text{"from":"TopStudent", "to":"Agent", "text":"Hey Agent, I've been using a specific study method that has really boosted my grades."}$
$\text{"from":"AverageStudent", "to":"Agent", "text":"Agent, I'm struggling with math. Any tips on how to get better at it?"}$.
In this case when the tested model is required to choose next target and generate proper sentence as next word, we mark $\text{"from":"Agent", "to":"TopStudent", "text":"TopStudent, your study method sounds interesting. Could you please elaborate on it?"}$ as better(golden) response than \text{"from":"Agent", "to":"AverageStudent", "text":"AverageStudent, I understand you're having a tough time with math. Let's talk about some strategies to improve."} for the strategy to ask "TopStudent" fist for useful study methods that the agent itself may not know, instead of responding to "AverageStudent" with limited knowledge.
Then for more detail the characters involved in conversational scenario are greatly various for different scenes. For example, we mainly extract scenes from sports, politics, education, weather, research and business news set. In sports topic main characters contain athletes, refugees and audience. In education part they may be teachers, students, parents and so on while in business they may be modified to shoppers, customers, government workers and business manager. For better knowledge we provide additional statistics about the dataset. It contains $N_0=8305$ samples in total. On average in each sample there are $N_f=6.722$ unique characters and $N_c=15.81$ turns of conversation. But in fact the most significant difference of our dataset compared to existing multi-party chat datasets should be explicit comparison of golden response and tested response of the "agent" character directly designed for LLM to play, in precise conversation position. Finally our dataset currently are under examination to avoid risk of leaking by sponsor company, but after the pipeline we will public entire version before year 2025.