We sincerely thank reviewer ZTvP for the constructive suggestions. Here we address and clarify the questions you posed.
**Paper writing and code release**
We agree with the reviewer that the writing for the general setting could be more concise and that the method can be explained in greater detail, we plan to address these aspects more comprehensively in the final version as structural changes to the writing require a relatively large amount of time.
We will first clarify some of your questions. All components within the central box, including the internal factors and assignments in the conditional probability tables for a given scenario, are automated and inferred through LLM interactions without knowing the specific downstream conditions. Later, the computations of observations (i.e., the condition-factor mapping) are also done automatically with LLMs. We include an extra instance in Fig. 2 that supports the opposite outcome, illustrating that BIRD can reliably compute probabilities and make decisions under any condition within the same scenario.
We will release the complete codebase, including the code for all detailed methods of prompting the LLM, the learning algorithm, and the data.
**Q1. Design choices of model parameters**
We agree with the reviewer that the design choices for model parameters can be better elaborated. To address this, we have added detailed explanations of these design choices in Appendix A.3 and provide the same detailed explanation here.
For most of the chosen values, we adopt a neutral and unbiased assumption. Consequently, we assign equal weights $w_j = 1, j = 1, ..., N$ and assume $P(O_i) = 50\\%$ in Section 3.2. The value of $\mathbb{P}_{\rm init}(O_i|f_j)$ in Equation 5 Section 3.3 is assigned as it represents random initialization.
To determine the mappings of rankings to probabilities in Section 3.3, we consulted two psychology experts and adopted the Likert scale theory. The finalized mapping between verbalized probabilities and numerical probabilities reflects typical human behavior and effectively distinguishes between verbalized probabilities in an unbiased manner. Additionally, slight adjustments to the mappings (~5%) do not noticeably affect overall performance.
We randomly sample 128 instances from the LLM as training data to manage costs in Section 3.3. This represents the minimum number of instances required for effective training. Ideally, increasing the number of sampled instances would improve the model's alignment with the underlying LLM. All training hyperparameters for the algorithm are optimized using a grid search with a hold-out validation set from Plasma.
We also conduct a preliminary ablation study on the loss function for decision making, as detailed below, and demonstrate that both components of the loss function are crucial for the model's performance.
| Llama-3.1-70b-Instruct | w/o MR Loss | w/o MSE Loss | BIRD |
|-------------------------|-------------|--------------|-------|
| TODAY | 65.5 | 65.2 | 74.3 |
| Plasma | 60.0 | 65.7 | 73.0 |
| Common2sense | 77.2 | 86.8 | 92.3 |
**Q3. 'Unknown' predictions**
We verify that 98% of the unknown instances stem from shortcomings in the factor generation method, as these instances cannot be mapped to any factor value. While our framework is theoretically robust, challenges in implementation may arise due to the lack of comprehensiveness or sufficient granularity in the generated factors. To address this, we plan to explore and develop more effective factor generation methods in future work.
At the same time, since BIRD does not rely on task gold labels to determine whether a prediction is unknown, we can leverage baseline methods (e.g., CoT) to generate predicted labels for these unknown cases. This approach ensures that our current comparisons, particularly regarding absolute performance gains, remain consistent even if we do not remove the unknown cases.
**Q4. Spurious biases of LLMs**
Language models operate as induction machines and thus may rely on spurious information about relevant concepts internally. This enables CoT-like inferences to make educated guesses that are correct in many cases but not necessarily for the right reasons to directly solve tasks. (as discussed in https://arxiv.org/pdf/2305.14825, https://arxiv.org/pdf/2311.09702, https://arxiv.org/pdf/2410.05229). BIRD provides an advantage by decomposing tasks into components with clearer individual objectives that leverage LLMs' strengths, such as abduction for factor generation, coarse estimation under complete information, and textual entailment. By structuring tasks in this way, each instance of LLM prompting becomes more controllable, mitigating the impact of spurious biases. Furthermore, BIRD incorporates a symbolic structure with explainable factors, encouraging more reliable and interpretable predictions.