Summary
The paper explores the challenges associated with using machine learning, particularly deep learning, to estimate causal treatment effects from high-dimensional data, such as images, in Randomized Controlled Trials (RCTs). The authors point out that standard practices in machine learning, such as selecting models based on accuracy and discretizing predictions, can lead to biases in causal estimates. To substantiate their claims, they provide both theoretical analyses and empirical results. They introduce a novel real-world benchmark dataset that involves the grooming behavior of ants, which they assert is the first benchmark designed for causal inference downstream tasks. The authors stress the necessity of incorporating causal downstream tasks into benchmark design and offer guidelines to enhance causal inference in scientific applications through machine learning.
Strengths
1. The paper addresses an important and under-explored area of causal inference, which will facilitate further research.
2. The authors provide some theoretical analysis to identify potential sources of bias in treatment effect estimation using deep learning models.
3. The paper is well-written and well-organized.
4. The experiments involve fine-tuning thousands of machine learning models with various encoders, enhancing the reliability of the conclusions.
Weaknesses
I am not very familiar with this topic, so I may adjust my rating based on further discussions with the authors and reviewers.
1. The main theoretical analysis focuses on binary classification, which may not generalize to many real-world settings. On line 251, the authors claim that Theorem 1 is validated on their dataset. Could you provide more details on how Theorem 1 is validated? I found it difficult to follow.
2. The setting of this paper differs from many causal representation learning algorithms, making it challenging to directly evaluate those algorithms on this benchmark. This may limit the applicability of the proposed benchmarks. Could you apply some existing causal representation learning methods to the proposed datasets and compare the conclusions? The experiments only test different pre-trained models, but it would be more compelling to demonstrate that this new benchmark can help answer causal questions with training in a causal manner.
3. The authors frequently mention causal inference downstream tasks. Can you further elaborate on why the proposed dataset is superior to other benchmarks for answering causal questions? The experiments primarily show that bias can arise, but I am not fully convinced that the proposed dataset is better than the existing ones.
4. While the authors acknowledge limitations such as keeping the backbones of models frozen and not incorporating semi-supervised learning, these constraints might affect the generalizability of their findings.
Questions
Please see the weaknesses below. I may reconsider my score after further discussion with the authors and reviewers. My main concern is the significance of the proposed benchmark and its utility in evaluating different causal representation algorithms. Why is this dataset better suited for answering causal questions and reducing bias compared to existing datasets?