LLM4Causal: Democratized Causal Tools for Everyone via Large Language Model

Summary

The authors propose a fully end-to-end pipeline generating responses to user querying an input dataset. The query consists of a causal question which need statistical causal analysis. The goal is for the proposed system to 1) understands the natural-language expressed query to format it in the correct way so that 2) the appropriate inferred tool is run through a predefined list of tools available online whose output is 3) formatted itself in a response phrased in natural language and answering the initial query The proposed approach, LLM4Causal, describes how the authors fine-tuned a modern pre-trained LLM on two datasets with the aim of improving the LLM output with regard to subtasks 1) and 3). The semi-automated methods to create these datasets are presented in details. The authors propose automatic and manual evaluation metrics and compare their system to the function calling feature of a chat-GPT (GPT4-turbo) LLM. Their experimental results demonstrate the superiority of their approach over the benchmark method.

Rating

8

Confidence

3

Ethics flag

1

Reasons to accept

The method is clearly described, experiments are sound. The task addressed is an effort toward leveraging natural language understanding and generation capabilities of LLM for actual assistant responding accurately to queries in the domain of causal analysis of quantitative variables. The work appears to be novel, useful for future steps in this direction, well designed and correctly evaluated.

Reasons to reject

Some details are omitted or not clearly explained, such as how exactly are the tools selected: does LLM4Causal’s LLM handle all the logic or do you implement standard logic once the causality-related task is identified by LLM4Causal? In other words, once LLM4Causal predicts that the task is CEL-ATE, how is DR selected over DML, TMLE, or IPW?

Questions to authors

Minor comments: - Table 1: overlapping caption - 3.1: adopt → adapt? - 3.1: publicity → publically? - 3.2: Appendix ??

Summary

This work explores the capability of large language models(LLMs) to perform causal tasks. The authors started with the motivation that existing LLMs do not handle specific structured data and knowledge well. The objective of this work is to build LLM4Causal by fine-tuning existing open-source LLMs capable of identifying causal tasks executing the corresponding functions and outputting the results. To fine-tune the training, they constructed two supervised datasets with GPT-4 prompting.

Rating

4

Confidence

4

Ethics flag

1

Reasons to accept

This work proposes a complete pipeline for boosting the performance of LLMs on specific tasks. However, constructing specific datasets via GPT-4 prompts and then fine-tuning open-source LLMs has been used in many previous works.

Reasons to reject

1. The work did not really improve LLMs' abilities in causal decision-making and reasoning, but rather taught LLMs to use causal analysis tools. 2. The experiment was well-designed and predictable. 3. Larger language models with larger scale parameters generally have stronger reasoning capabilities. It is suggested that further exploration and experimentation be conducted. 4. Provide implications in calling different causal tools and algorithms.

Questions to authors

The appendix provides insufficient details on how to use the GP4-turbo.

Summary

This paper presents an end-to-end framework for causal decision-making tasks, which addresses the weakness of the current LLM applications on causal tasks. A three-step data generation pipeline is proposed, it first interprets user requests into a JSON data for causal task , and then automatically selects the causal tool to execute the corresponding algorithm, finally it generates a natural language output. Two benchmark datasets, Causal- Retrieval-Bench for causal function calling and Causal-Interpret-Bench for causal interpretation are constructed.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

The article proposes a scheme to transfer diversity causal tasks into a unified structured JSON data. This approach can effectively integrates tasks and facilitating the use of LLM. The experimental results shows that the method has significant advantages. The paper provides detailed attachments to demonstrate the effectiveness of the work.

Reasons to reject

The paper proposes the use of structured JSON to represent a variety of tasks and demonstrates that this integrated approach outperforms end2end GPT-4 method. However, the paper does not validate whether this method is superior to handling each task individually, particularly in cases of individual fine-tuning them. Moreover, controlling the quality of JSON data needs additional manual consumption, which diminishes the practical significance of this integrated approach.

Summary

The paper presents LLM4Causal, an end-to-end, user-friendly large language model designed for causal decision-making tasks. The model addresses the limitations of general LLMs in handling causal tasks by fine-tuning them with two specialized datasets: Causal-Retrieval-Bench for identifying causal problems and extracting variables and Causal-Interpret-Bench for interpreting causal results. LLM4Causal operates through three major steps: interpreting user requests, assigning and executing causal tools, and providing easy-to-understand interpretations of the outputs. The model's performance was evaluated across three main causal decision-making categories: causal effect estimation, causal structure discovery, and causal policy learning. In total, there are 5 tasks in these three categories. The results demonstrate superior performance compared to baselines like ChatGPT and GPT-4. This work highlights the potential of fine-tuning large language models to enhance their capabilities in specialized domains like causal inference, making them accessible and effective for general users.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

1. This work presents a very comprehensive framework for causal inference. Here, the authors fine-tuned LLM to interpret uses' causal query, utilize external tools to calculate causal relations, and interpret the logic behind causal relations. 2. The framework covers five tasks and conducts a lot of experiments, showing its generalization ability on various causal tasks. 3. The problem studied in this paper is significant, which is how to use LLM to make causal inference more accessible to normal people.

Reasons to reject

1. There are some typos, like in Section 3.2, "See Figure 5 in Appendix ??" 2. Since the paper just fine-tunes Llama2 on those causal tasks instead of instruction tuning. Can we just fine-tune a small model, like T5, BART, or GPT2, to do this task? 3. Two fine-tuning datasets collected in this paper are quite small. The Causal-Retrieval-Bench only contains 1,500 pairs. Causal-Interpret-Bench only contains 400 examples. It seems that there will a very sever over-fitting problem for Llama2 (7B). 4. Is this framework possible to be extended to causality between textual events, like the following references? 5. There are also some missing reference: 1) Rashkin, Hannah, et al. "Event2Mind: Commonsense Inference on Events, Intents, and Reactions." Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2018. 2) Ning, Qiang, et al. "Joint Reasoning for Temporal and Causal Relations." Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2018. 3) Wang, Zhaowei, et al. "COLA: Contextualized Commonsense Causal Reasoning from the Causal Inference Perspective." Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.

Questions to authors

No

Thank you for the clarification

Thank you for the clarification. I now understand that, after identifying the task, the method is selected by default. In my opinion (also raised by Reviewer UqM4), it would improve readability if you specified that each task is currently assigned a default method, and future steps could include automatic method selection for a given detected task (out of this work's scope).

Follow-Up

Dear Reviewer, we sincerely appreciate the time and effort you invested in evaluating our submission. We hope our responses have adequately addressed your question and clarified any concerns regarding our paper. Should you have any further feedback, questions, or comments, please do not hesitate to let us know before the rebuttal period concludes. Your insights are invaluable, and we're keen to address any remaining issues.

Follow-Up

Dear Reviewer, we sincerely appreciate the time and effort you invested in evaluating our submission. We hope our responses have adequately addressed your question and clarified any concerns regarding our paper. Should you have any further feedback, questions, or comments, please do not hesitate to let us know before the rebuttal period concludes. Your insights are invaluable, and we're keen to address any remaining issues.

Follow-Up

Dear Reviewer, we sincerely appreciate the time and effort you invested in evaluating our submission. We hope our responses have adequately addressed your question and clarified any concerns regarding our paper. Should you have any further feedback, questions, or comments, please do not hesitate to let us know before the rebuttal period concludes. Your insights are invaluable, and we're keen to address any remaining issues.

Response to Rebuttal

I have read the response