Author Response #1
Thank you for the detailed review and insightful suggestions.
**W1 Counterfactual generation results**. We are sorry that our presentations might have confused you. We believe that our counterfactual generation results are sufficient to support that our DiTi learns a more disentangled representation compared to baselines. Specifically, we have two main types of experiments: 1) By interpolating a feature subset between a pair of images (Figure 5), and 2) by manipulating a feature subset guided by the weight of an attribute classifier.
- For 1), only our DiTi allows meaningful interpolation between image pairs as shown in Figure 5 and A11 in Appendix. Hence, we include the baseline results in Appendix (Figure A12-A13), where the generated counterfactuals are either identical to the original image, or contain substantial artifacts, validating the baseline features are still entangled.
- For 2), a disentangled representation should enable manipulating a single attribute without affecting other ones. As shown in Figure 2(a), 6 and A14-A16 in Appendix, baseline methods typically entangle multiple attributes (e.g., manipulating age and gender together), and are still prone to generating visual artifacts. In contrast, our DiTi mainly manipulate only the target attribute. Note that the results can be quite subtle when the attribute is fine-grained, e.g., “wearing lipstick”. Following your suggestion, we will adjust the claim about bedroom experiments to reflect the aforementioned point.
- We have other supporting experiments to verify disentanglement quality, e.g., visualization of classifier weight (explained in the last paragraph of Section 3.1), strong inference results as noted by other reviewer (Section 5.2), and the results of interpolating the whole feature instead of a feature subset (explained in the last paragraph of Section 5.3).
Overall, our results are from multiple evaluation methods, on different datasets, with many examples, and we also compare with diverse baselines (e.g., including conventional disentanglement methods based on VAE and GAN). Hence we believe that they are sufficient to validate our disentanglement quality.
**W2 Mitigating spurious correlation by disentangled representation**. Great suggestion. We evaluate our method and baselines on a few-shot learning task for the fine-grained attributes in CelebA. This is particularly challenging, as the classifier needs to isolate fine-grained class attributes from visually prominent yet spurious ones. As our method disentangles fine-grained attributes on the feature subset corresponding to a small $t$ (e.g., early $t$ in Figure 5), we can perform human intervention to only use that subset to train a classifier, which discards the spurious attributes. As shown in Table r1 and r2, using the full feature of DiTi for few-shot learning does not bring much improvements, as the classifier can still exploit the spurious correlations on the disentangled coarse-grained attributes. However, after intervention, we observe significant improvements over baselines.
Table r1: $N$-shot accuracy on "smiling" attribute averaged over 10 runs.
| Method | 5 | 10 | 20 | 50 |
| ----------------- | ---- | ---- | ---- | ---- |
| Diff-AE | 56.1 | 60.2 | 62.2 | 67.8 |
| PDAE | 55.5 | 59.5 | 61.8 | 66.5 |
| DiTi | 56.0 | 60.4 | 62.6 | 68.4 |
| DiTi+intervention | 58.5 | 62.7 | 67.4 | 71.8 |
Table r2: $N$-shot accuracy on "wearing lipstick" attribute averaged over 10 runs.
| Method | 5 | 10 | 20 | 50 |
| ----------------- | ---- | ---- | ---- | ---- |
| Diff-AE | 55.2 | 57.4 | 60.3 | 62.3 |
| PDAE | 55.0 | 56.6 | 59.7 | 62.1 |
| DiTi | 55.3 | 57.1 | 60.2 | 62.7 |
| DiTi+intervention | 56.5 | 59.2 | 62.4 | 65.6 |
In fact, we have performed similar follow-up experiments using a text-conditioned DM (Stable Diffusion), where our method beats few-shot learning methods using OpenCLIP by up to 13%. See Cjby-Q2 for details.