Graph-based Unsupervised Disentangled Representation Learning via Multimodal Large Language Models

Summary

This paper introduces a bidirectional weighted graph-based framework, to learn factorized attributes and their interrelations within complex data. The authors proposed a β-VAE based module to extract factors as the initial nodes of the graph, and leverage the multimodal large language model (MLLM) to discover and rank latent correlations, thereby updating the weighted edges. Experiments demonstrate evidence of effective performance in disentanglement and reconstruction.

Strengths

This paper integrates the VAE with multimodal large language model, which provides strong interpretability via the open-world knowledge of the large language models/multimodal models.

Weaknesses

* As a generative model, β-VAE is extensively discussed with many useful variants. While this work heavily borrows the existing advances of β-VAE, I would respect ground-breaking findings or improvements on this model (e.g., closed-form derivation of the feature distribution z after employing a graphical model). Otherwise speaking about absolute performance, I expect a standard diffusion model can easily outperform β-VAE and most of its variants (And authors should include comparison and discussion with diffusion models, even it is less likely that the proposed method can outperform, but insights on what β-VAE is lacking would be insightful, although it’s less meaningful in the era of diffusion model). * Related to above, the technical novelty and motivation of this work is limited since the current model can be easily understood as the combination of β-VAE and Large Multimodal models. There is less strong justification while this combination is optimal rather than other obvious choices (e.g., integrating a vision-language model, or diffusion+MLLM (since diffusion models are also trained with variational inference)). Author should either provide strong justification (better if theoretical) or more empirical validation showing the proposed framework is optimal than other obvious variants to enhance their motivation and novelty. * Line 148, it is obvious that z dos not follow multivariate normal, and author proposed Eq. 4 as the solution. Surprisingly, the author only showed the derivation of this equation (which is trivial in my view) in the appendix, with no justification why this is optimal. Why α=1? Or how strong is this assumption? Is the resulting objective still convex? What is the distribution of z under the new objective? Overall the authors imposed made many conditions in the ease of derivation with many justifications missing. * The authors only used GCN to learn the graph representation, which ruled out many other prominent choices (E.G., GAT, gatV2, GIN, GTN, GraphSAGE). authors should conduct thorough ablation study to select the optimal GNN architecture. * As an applied paper, it is less acceptable that codes are not released in the current phase. As many papers release their codes in anonymous GitHub repos and recently there are higher requirements of reproducibility and codeavilability in top venues. We also need to check the overall soundness of the empirical implementation.

Questions

* For the bidirectional weighted graph, it is unclear why this is a good choice. From my knowledge, a causal DAG would be a better choice to represent the interrelation of the entities, and there are many works (e.g., [1]) available deriving the closed-form distributions of the features under this assumption. [1] BayesDAG: Gradient-Based Posterior Inference for Causal Discovery. NIPS 2023.

Rating

6

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

The authors addressed some limitations of their works.

Summary

Researchers introduced a bidirectional weighted graph-based framework to explore factorized attributes and their interrelations within complex data. They proposed a -VAE module for extracting initial factors and utilized a multimodal large language model (MLLM) to uncover latent correlations and update weighted edges. Integrating these modules enabled their model to achieve superior unsupervised disentanglement and reconstruction performance, inheriting interpretability and generalizability from MLLMs.

Strengths

1. The paper introduces a graph-based approach to model interrelationships within complex data, aiming to integrate background knowledge into Deep Reinforcement Learning (DRL). I find this idea novel and intriguing. 2. The paper presents a rigorous framework with clear exposition and straightforward methodology, making it accessible and easy to understand.

Weaknesses

1. The paper lacks significant innovation at the neural network and algorithmic levels. While the introduction of DisGraph and its optimization methods is proposed to be effective, insufficient explanation is provided regarding why they work. Strengthening this aspect of the description would enhance the paper's persuasiveness. 2. The paper could provide a brief explanation of some methods used, such as the Somers' D algorithm, even if included in an appendix. Currently, this aspect appears somewhat incomplete. 3. An explanation of the update mechanism for the entire model at the end of Section 3 would be beneficial.

Questions

1. How is the effectiveness of DisGraph ensured? What problems could arise if errors are introduced into DisGraph?

Rating

6

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

Yes

Summary

To achieve fine-grained, interpretable and unsupervised disentangled representation learning (DRL), this paper proposes a new framework by integrating $\beta$-variational autoencoder ( $\beta$-VAE), multimodal large language model (MLLM) and graph learning into a single pipeline. Experimental results show that the proposed framework can achieve a better trade-off in the capability of disentanglement and quality of reconstruction than the evaluated baselines under different datasets.

Strengths

The strengths of the paper are listed as follows. 1. The paper is well-motivated. Figure 1 clearly illustrates the limitations of the existing works in DRL and the advantages of the proposed framework, which clearly shows the motivation of the paper. 2. This paper provides a thorough and insightful summarization of the related work in DRL, which highlights the contribution of the proposed framework. 3. It is great that the authors could provide detailed and diverse qualitative results in the section of experiments. It can give the readers a more clear view to understand the effect brought by the proposed framework.

Weaknesses

The weaknesses of the paper are listed as follows. 1. It would be better if the authors could first formulate the problem as an optimization problem mathematically before introducing the details of the method to solve the problem. In the problem formulation, the input, output, constraints and objectives should be clearly defined. 2. In the proposed framework, the input to the decoder $D_{rec}$ is not from the normal distribution but from the variable extracted from DisGraph. It is a critical point that needs to be highlighted in Figure 2. However, it is missing in Figure 2. 3. Some technical details are not clear as listed below. A. The input to the decoder $D_{rec}$ is not from the normal distribution but from the variable extracted from DisGraph. But how to make this process differentiable for end-to-end training is unclear. B. It is not clear how to train the graph learner in an unsupervised way. What is the loss function? What is the dimension of the node feature? Moreover, how to update the adjacency matrix of the DisGraph given the updated weights of the GNN? C. In Figure 2, it is not clear about the usage of a set of extra images input to the landmark function. 4. It would be better if the authors could use some metric to quantify the disentanglement capability of the DRL algorithms and show the quantitative results to evaluate the proposed framework and the baselines. 5. It would be better if the authors could provide a more intuitive explanation of the loss function they designed. What is the usage of each term within the loss function? Why can they address the limitations of the existing challenges of DRL?

Questions

The questions of the paper are listed as follows. 1. Is Equation (2) instead of Equation (1) the fundamental objective of vanilla VAE? 2. What are the differences among $p_{\theta}(z|x)$, $p_{\theta}(x|z)$ and $p_{\theta}(z)$? Why can they share the same parameters $\theta$? 3. Why can the MLLM have the capability to provide accurate attribute scores given the multimodal input provided by the framework? 4. If it has been proved that unsupervised DRL is impossible without extra prior, why does the proposed unsupervised framework work? What is the extra prior or inductive bias here? 5. Could you please give more explanation about why is the proposed algorithm to obtain the impact scores between the pair of attributes reasonable? 6. What are the potential use cases of the disentanglement capability from DRL?

Rating

6

Confidence

3

Soundness

3

Presentation

2

Contribution

3

Limitations

The limitations are shown in the section of Weaknesses and Questions.

Summary

The paper presents a novel framework that integrates β-VAE with multimodal large language models within a graph structure, DisGraph, to enhance disentangled representation learning. This approach allows for effective handling of complex and interdependent data attributes in an unsupervised manner. The model dynamically updates relationships between attributes, improving disentanglement and interpretability compared to traditional methods. Extensive experiments demonstrate its superior performance in both disentanglement and reconstruction.

Strengths

1. The paper successfully integrates β-VAE with multimodal large language models (MLLMs) within a graph-based framework, which is a novel approach. 2. The use of a graph-based approach to model the relationships among attributes addresses a significant gap in existing DRL methods. 3. The experiments are well-designed, covering various aspects of the model’s performance.

Weaknesses

1. The paper would benefit significantly from clearer writing and better organization. The paper occasionally uses technical terms and concepts without adequate definitions or explanations. 2. While the paper addresses the unrealistic assumption of statistical independence in many DRL methods, the solution proposed still relies heavily on the disentanglement abilities of β-VAE, which itself often presupposes some level of independence or weak dependence among latent variables. 3. Also, the reliance on pre-trained multimodal large language models might introduce biases from these models into the disentangled representations. 4. A deeper theoretical analysis of why and how the inclusion of interrelations leads to better disentanglement would be valuable.

Questions

1. How are the generated node embeddings T generated by DisGraph associated with the losses introduced in Section 3.1?

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

Yes