Response to reviewer-9my4
Thanks for your great efforts and time in reviewing our paper.
### 1. *Results of fine-tuned Video-LLaVA*
Firstly, we want to state that the few-shot evaluation (In-Context Learning) for LLMs is widely recognized as a strong baseline for reasoning and causal discovery tasks [1-6], effectively reflecting their performance on downstream tasks. Therefore, comparing LLMs in a few-shot setting is a common and accepted practice in the field.
Secondly, neither GPT-4 nor Gemini-Pro offer interfaces for fine-tuning, consequently, we reported the few-shot results of LLMs and investigated the impact of varying the number of few-shot examples, demonstrating the adequacy of our prompt (cf. supplementary Sec. G, lines 572-578).
Furthermore, as you and reviewer-2Vef suggested, to ensure a fairer comparison, we conducted a strongly-supervised experiment on the open-source method Video-LLaVA. Specifically, we fine-tuned Video-LLaVA using LoRA under its official implementation on our MECD training set. As shown in the table, Video-LLaVA gains a 4.6% improvement from fine-tuning on our MECD. **However, it still falls behind our proposed method**. Since fine-tuning LLM-based baselines is time-consuming, we will include more results of fine-tuning VLLMs for a comprehensive comparison in our final version.
|Setting| Acc|
|------------------------ | -------- |
|Video-LLaVA (few-shot)| 62.5|
|Video-LLaVA (fine-tuned) |67.1|
|Ours|**71.2**|
[1] Language models are few-shot learners. NeurIPS 2020.
[2] Rethinking the role of demonstrations: what makes in-context learning work? EMNLP 2022.
[3] Large language models are latent variable models: Explaining and finding good demonstrations for in-context learning. NeurIPS 2023.
[4] CLADDER: assessing causal reasoning in language models. NeurIPS 2023.
[5] Causal inference using llm-guided discovery. AAAI 2024.
[6] Open Event Causality Extraction by the Assistance of LLM in Task Annotation, Dataset, and Method. ACL 2024.
### 2. *More details*
We're happy to provide more details here to clarify your concerns. However, we need to state that **these details can be also found in the supplementary in our initial submission**. To further relieve your concerns about our implementation and reproducibility, we provide an **anonymous GitHub project** (https://anonymous.4open.science/r/NeurIPS-4887-MECD), which contains our *data annotation*, *training and evaluation codes*, and details about how we evaluate LLMs and how we fine-tuning the Video-LLaVA. Additional details can be found in the README.md file.
- **Training procedure**: All the experiments are conducted on 1 NVIDIA A40 GPU. We train our model for 20 epochs with a learning rate of 16e-5 about 6 hours. Our optimizer is consistent with the BertAdam optimizer, with 3 epochs of warm-up. We report the average results during all experiments under three random seeds (2023, 2024, 2025).
- **Hyperparameters**: $\lambda_C$, $\lambda_R$, $\lambda_V$,$\lambda_S$ are set to be $1.0, 4.0, 0.25, 0.05$.
- **Encoder & Decoder architecture**: Our encoder $Enc_{v}$, $Enc_{c}$, and multi-modal video decoder $Dec$ are built upon videoBERT, a joint model for video and language representation learning.
- **Evaluation of LLMs**: We prompt the GPT-4 with the following few-shot prompts to conduct evaluation (Details can be found in the anonymous GitHub project):
```
# Task: Each video consists of n events, and the text description of each event has been given correspondingly (separated by " ",). You need to judge whether the first n-1 events in the video are the cause of the last event, and the probability of the cause 0(uncausal) or 1(causal) is expressed as the output, Let's think step by step through the chain of thought.
Here are several examples of judging whether the first n-1 events in the video are the cause of the last event:
<start>
First example:
Text description of n events:
...
The probability output should be:
...
Second example:
Text description of n events:
...
The probability output should be:
...
Third example:
Text description of n events:
...
The probability output should be:
...
<end>
```
As a long-term maintenance benchmark, we confirm that the dataset and code will be public after being accepted.
We hope our responses can address your concerns and raise your rating of our paper.