Disentangled Counterfactual Learning for Physical Audiovisual Commonsense Reasoning

Summary

This paper presents a groundbreaking Disentangled Counterfactual Learning (DCL) approach for physical audiovisual commonsense reasoning. The main objective of the proposed method is to infer objects' physics commonsense based on both video and audio inputs, effectively mimicking human reasoning abilities. To address the limitations of existing methods in utilizing the diverse characteristics of multimodal data and lacking causal reasoning abilities, the authors introduce the DCL method. Contributions: 1. introducing a novel DCL approach that leverages disentanglement and causal reasoning to improve multimodal data utilization and achieve remarkable performance. 2. propose a novel Counterfactual Learning Module to model physical knowledge relationships. 3. the proposed DCL method is designed as a plug-and-play module, making it adaptable for integration into various baseline models.

Strengths

The paper introduces an innovative framework for tackling the challenging task of physical commonsense reasoning. The authors meticulously design a disentangled sequential encoder and a counterfactual learning module, both of which contribute to the success of the proposed model in addressing this unique visual question answering (VQA) problem. Importantly, the model's modular nature allows for seamless integration with various baseline approaches, thereby enhancing its versatility. The experimental evaluation conducted in this study is relatively enough, demonstrating the effectiveness of the proposed method. The results showcase a significant improvement in accuracy on the PACS dataset, validating the model's ability to handle complex reasoning tasks. Moreover, the authors present qualitative results that vividly illustrate how their approach enhances material reasoning performance, further reinforcing the practical relevance and value of their work. In terms of the paper's presentation, the English writing is commendable for its clarity and accessibility. The authors effectively convey their ideas, making it easy for readers to grasp the core concepts and understand the technical details without unnecessary complexity.

Weaknesses

(1) The main experiment is limited to the PACS dataset and a specific physical audiovisual commonsense reasoning task. While the proposed method is presented as a plug-and-play module, it would be valuable to explore its generalizability to other VQA tasks and unseen data distributions. Considering the versatility of baseline models like CLIP, it would be worthwhile to investigate whether this module can be effectively applied in diverse scenarios. (2) The ablation analysis is not fully convincing. As a plug-and-play module, it is crucial to clarify that the observed improvement is attributed to the unique design of the proposed method rather than an increase in the number of parameters. To strengthen the argument, additional quantitative experiments could be conducted, such as replacing the Disentangled Sequential Encoder (DSE) with a trivial naive module and comparing the results. (3) The tables presenting the results of the quantitative experiments are not well displayed. To enhance clarity and readability, improvements could be made in the formatting and organization of the tables. Additionally, the t-SNE visualization in Figure 3 could benefit from displaying more distinct clusters and additional samples. The current arrangement of Figure 3 lacks persuasiveness, as the clusters on the 2D space are not significantly distinguishable from each other.

Questions

You can conduct more quanlitative experiments according to the weaknesses.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Limitations have been discussed in the paper

Summary

The work proposes an approach to separate object and action information from videos to improve the model's reasoning capabilities. The proxy-task of audio-visual question answering is used to train the model commonsense concepts of the physical world. Their contribution focuses on learning from introduced counterfactual examples. They aim to maximise mutual information between input data and static and dynamic scene factors while minimizing the mutual information between those two factors.

Strengths

The paper presents beyond a good intuition also a detailed mathematical definition of all concepts. Both can be followed well. A fair amount of relevant and current baseline methods were used for comparison. The approach seems to be lightweight enough to be trained on one single GPU within reasonable time. The ablation studies seem adequate and useful to understand the success of the disentanglement and the performance boost over baselines.

Weaknesses

Compared with datasets in other fields the used dataset seems just about big enough to test the approach however a test set in the PACS-Material dataset of 152 objects could potentially lack robustness. Other selected 152 objects may change the result by a large margin. However, in this specific domain it is probably hard to generate larger datasets and comparable work uses the same datasets. K-fold cross evaluation could help here to evaluate the result better. There is a clear trend that the method performs well in Table 1, however given the error the results could be more impressive. Again, K-fold cross validation with different splits could help. The paper mentions a few times that it can be used as a plugin to improve performance on multimodal fusion tasks, however due to the Q&A being an integral part I don't see how this can generalize to e.g. benefit tracking. Due to their claim it would be nice if the authors could illustrate examples of potential tasks that could be improved with this method.

Questions

In Figure 1. What is the referenced object? There are screws and a glass. What if there are several objects in the video but the question is ambiguous because it refers to only one object? Extracting audio as non-sequence seems odd. How long is the audio instance? What about objects that change their sound characteristically? E.g. think about a laundry machine and a tumble dryer. Both may make some sounds which are over a longer time similar but then the laundry machine starts spinning and you can tell them apart. With such sounds it's also hard choose the part of the sound to encode. How is the length choosen? Does it just cover exactly all the video frames? How does the approach deal with different lengths of videos? Is this just not regarded? It is hard to find an information of how long the video sequence or audio sequence is in general. In the supplementary material, details are given for the implementation of LateFusion and there it can be found that 8 evenly spaced frames are used but does that mean that every single frame is used or are there skipped frames? And what is the framerate? In line 175 it should say standard and not stand. The object in Figure 2, b) left side is hardly recognizable. It says the material is wood but is this a woodchip or a fake wood coin? It's very hard to see. And what kind of sound does this make? Are those woodchips dropped on a hard surface? It would be good to either choose another object from another image which is more illustrative or to write down somewhere what the reader actually should see. In the "Analysis of dynamic factor" it is said two times, audio information is often the main basis for human reasoning or dynamic information is often the main basis for human reasoning. I would just leave out those broad claims which can not be answered by training a network. It reads to me like a very general remark without any citation or investigation to back it up and should probably left to the field of HCI. In Table 1, how was the error calculated? Were different training runs performed with different random seeds?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The small dataset test set could probably be mentioned or defended somewhere for people who are not right inside the domain. The broader impact section feels like being included just as exercise but could be removed imo. The argument "if our algorithm is implemented on a robot and malfunctions this could be bad" is too general to be useful.

Summary

This paper proposes a novel Disentangled Counterfactual Learning (DCL) method for physical audiovisual commonsense reasoning. DCL consists of two main modules: Disentangled Sequential Encoder and Counterfactual Learning Module(CLM). Disentangled Sequential Encoder decouples videos into static (time-invariant) and dynamic (time-varying) factors in the latent space. The causal learning module augments the model’s reasoning ability by modeling physical knowledge relationships among different objects under counterfactual intervention. The experiments show that DCL can be flexibly integrated into baseline methods and improve their performance.

Strengths

+ The designed DCL is a plug-and-play module that can be incorporated into baseline methods. + The proposed method reaches SOTA performance. The authors also conduct ablation studies to illustrate the effectiveness of each component in their method.

Weaknesses

- This paper is poor-written. Some expressions are not consistent, e.g., "causal learning module" in the abstract and "counterfactual learning module" in the introduction. - Why causal learning can help commonsense reasoning? Which concrete problem in audiovisual commonsense reasoning can causal learning address? Although the authors introduce causal learning to augment the model's reasoning ability, no deep analyses are provided in this paper. The authors can provide an example or toy experiments to illustrate their motivation. - Figure 3 only shows the t-SNE visualization of dynamic factors. Can the authors show t-SNE of static factors and the comparison with baselines? - Why do the authors only use disentangled sequential encoder (DSE) in videos? Can audios be processed by a similar operation?

Questions

Please see weaknesses.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

1 poor

Contribution

2 fair

Limitations

The authors present the broader impact in this paper.

Summary

The paper introduces a novel approach for physical audiovisual commonsense reasoning by Disentangled Counterfactual Learning (DCL). The authors propose a disentangled sequential VAE to separate static and dynamic factors in the visual latent space with an additional contrastive loss term. In addition, a causal learning module that leverages counterfactual intervention between different objects is used to enhance the learning of physical knowledge relations. The proposed modules could be easily plugged into existing baselines. The experiments on PACS dataset demonstrate that the proposed method could improve baseline methods.

Strengths

1. The idea of modeling implicit physics knowledge relationships between various objects from audio-visual data is interesting and well-motivated. 2. The proposed method is a general module that could be plugged into any baseline. I could also see these modules are definitely not limited to being applied for audio-visual commonsense reasoning. 3. The paper is well-written and straightforward to understand. 4. Extensive experiments and analyses have been done to demonstrate the contribution of each proposed component.

Weaknesses

1. While I like the general idea of the proposed modules, the final performance shows a minor increment (1.4-3.2%) compared to the baselines. With all these sophisticated designs of additional components, I would expect a larger gap in terms of performance, even though I believe audiovisual physical commonsense reasoning is a challenging task. If I understand correctly, the questions in PACS are binary (e.g., object 1 or 2). As I look into the results, the accuracy is about 60% which is not much better than a simple random guess. In the Supplementary material, there is a model size comparison between models with/without DCL. With the DCL, about 12M more parameters are introduced. Is it possible that the improvement is simply from these additional parameters? A simple verification would be adding the same amount of parameters via MLPs on top of the existing baselines and reporting the outcomes without using DCL. 2. In Section 3.2, Line 137, the authors mention the full proof could be found in Appendix, but it is missing. 3. The T-SNE plots in Figure 3 and the Supplementary material could be more informative. The description of each color in the plot is missing. 4. See Questions.

Questions

1. In the current formulation, only the video features are represented as sequence data, but the audio feature is not. What is the reason for doing that? Based on the results, AudioCLIP actually works better than CLIP. This means probably audio plays a more important role in the reasoning. I guess having a disentangled sequential encoder for audio features and using contrastive loss could be possible. 2. Following the first question, the disentangled sequential encoder is for unimodal only (visual). I wonder whether adding a cross-modal mutual information term would be helpful if both audio and visual features are considered sequential. 3. As I mentioned in the weakness, the final performance is still around 60% accuracy, even with the proposed method. While I see some examples in qualitative results and analyses, like inaccurate labeling, I would like to ask what the main failure cases would be and the challenges to overcome. 4. Currently, the method is end-to-end training. I wonder whether it would be helpful if the disentangled sequence VAE is pre-trained first so that the static and dynamic factors are well-learned before applying counterfactual intervention.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

N/A

Summary

This paper proposes a disentangled counterfactual learning (DCL) approach to solve physical audio-visual commonsense reasoning. This approach first decouples videos into static and dynamic latent features, and then uses a causal learning module to augment the model's reasoning ability. Authors show that this module can be used to augment any existing baselines.

Strengths

The proposed approach is novel and inspiring for similar reasoning tasks. The approach section has detailed each sub-module of the approach. I also like the fact that this approach can be plugged into any existing work to improve its performance. The qualitative examples in Fig. 2 show how baseline models augmented with the proposed approach performs better than those without.

Weaknesses

The performance improvement against baselines is pretty marginal, especially it did not run Merlot Reserve with the additional module. I understand that it was because the computational resource is restricted. But still, since Merlot Reserve has a 10% improvement over the second-best baseline, it remains a question whether this proposed module still brings benefits to Merlot Reserve. Some other comments: 1. Random performance (guessing by chance) should be shown in Table 1. 2. The reported baselines are baselines in the benchmark but that does not represent the current SOTA in this domain anymore. There have been many other works coming out after CLIP for example. 3. Late Fusion w/ dynamic [42] should be written as Late Fusion [42] w/ dynamic. And the same applies to other rows in Table 1. 4. You need a more concrete example to motivate the physical knowledge relationship module. It is unclear in the paper what kind of correlation between different samples are shared.

Questions

My main question about the paper is regarding the performance margin and I'd appreciate authors' clarification on that.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

The limitation of this work has not been discussed but the societal impact has been discussed in the last paragraph.