Customized Multiple Clustering via Multi-Modal Subspace Proxy Learning

Summary

This paper incorporates a multi-model subspace proxy learning (Multi-Sub) to design a novel end-to-end multiple clustering approach and utilizes the synergistic capabilities of CLIP and GPT-4 to align textual prompts expressing user preferences with corresponding visual representations. The main contributions of Multi-Sub can be summarized as follows: 1. Capturing user’s clustering interest: Existing works struggle to adapt to diverse user-specific needs in data grouping. To overcome this limitation, Multi-Sub explicitly captures a user’s clustering interest by learning the desired clustering proxy under a user’s interest and aligning textual interest with corresponding visual features. 2. Simultaneous optimized framework: The previous methods separated the representation learning and clustering stages. Different from them, Multi-Sub obtains both the desired representations and clustering simultaneously, which significantly improve the clustering performance and efficiency. 3. Extensive experimental validation: Extensive experiments on all public multiple clustering tasks demonstrate that Multi-Sub outperform other methods. Moreover, a series of ablation studies further verify the effectiveness of Multi-Sub.

Strengths

1. In real world, data may have multiple aspects that they can be grouped into different clusters. However, existing methods solely consider a single partition. So, it is meaningful to propose an effective algorithm to overcome this problem. 2. The authors leveraged large language models (LLMs), including GPT-4 and CLIP, to align image and textual representations in the same subspace. Then, multi-modal subspace proxy learning is introduced to allow for the customized representation of data in terms specific to the user’s interests. 3. Experimental results on public datasets show that the Multi-Sub method has a significant improvement, indicating the effectiveness of the propose method.

Weaknesses

1. To change the two-stage learning approach of previous works, Multi-Sub aims to learn representation and clustering simultaneously. However, Multi-Sub employs a two-phase iterative approach to align and cluster images in training process, including (1) Phase I: Learning and Alignment; (2) Phase II: Clustering and Optimization. I wonder if this is another form of two-stage task. 2. The description of Clustering Loss is not very clear in Section 3.4, how to determine that samples belong to the same class? By pseudo-label? Where did the pseudo-label come from? 3. In this paper, the authors introduced large language models (LLMs) to learn representations and bridge the gap of textual and image features. But does the direct use of a pre-trained large language model introduce a priori information about the category, which can lead to unsupervised scenarios being corrupted?

Questions

Please refer to the Weaknesses.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have discussed social impacts and limitations.

Summary

This paper presents an innovative approach for addressing the limitations of existing multiple clustering methods. By leveraging the synergistic capabilities of CLIP and GPT-4, Multi-Sub aligns textual prompts with visual representations to cater to diverse user-specific clustering needs. This method introduces a novel multi-modal subspace proxy learning framework, which automatically generates proxy words from large language models to represent data in terms specific to user interests. The experimental results demonstrate that Multi-Sub consistently outperforms existing baselines across various datasets. Overall, I believe this paper makes a substantial contribution to the field of deep clustering and holds significant practical application value.

Strengths

The paper offers several notable strengths that contribute to its overall impact and significance in the field of multiple clustering: 1. The integration of CLIP and GPT-4 for multi-modal subspace proxy learning is novel and effectively addresses the limitations of traditional multiple clustering methods. 2. Multi-Sub excels in capturing and responding to diverse user interests, providing tailored clustering results without requiring extensive manual interpretation. Moreover, the performance gains come at a low cost and seem relatively easy to achieve. 3. The writing is clear and easy to follow. The figures are well-drawn, allowing for a quick understanding of the research motivation and methodological design. 4. Extensive experiments on a wide range of publicly available datasets demonstrate the robustness and generalizability of the proposed method.

Weaknesses

Despite its strengths, there are some areas where the paper could be improved to enhance its clarity and applicability: 1. Although the paper mentions the hyperparameters used, a more detailed analysis and discussion on the sensitivity of the method to these parameters would be beneficial. 2. Given the method's iterative nature and the use of large models, there is a risk of overfitting, especially on smaller datasets. I am curious whether regularization techniques were used to address this issue? 3. Table 3 compares the impact of different text encoders on performance. Clearly, there are significant performance differences when using different encoders, and the authors have indeed analyzed this issue. However, I believe the reasons behind this phenomenon could be explored in depth. Intuitively, given that the input text is quite simple, the overall performance should not be particularly sensitive to the choice of text encoder.

Questions

1. Although the paper mentions the hyperparameters used, a more detailed analysis and discussion on the sensitivity of the method to these parameters would be beneficial. 2. Given the method's iterative nature and the use of large models, there is a risk of overfitting, especially on smaller datasets. I am curious whether regularization techniques were used to address this issue? 3. Table 3 compares the impact of different text encoders on performance. Clearly, there are significant performance differences when using different encoders, and the authors have indeed analyzed this issue. However, I believe the reasons behind this phenomenon could be explored in depth. Intuitively, given that the input text is quite simple, the overall performance should not be particularly sensitive to the choice of text encoder.

Rating

7

Confidence

5

Soundness

4

Presentation

4

Contribution

4

Limitations

The limitations are discussed in the paper by the authors. There is no potential negative societal impact

Summary

The paper is about Multiple Clustering, which is an interesting topic. The authors propose a novel end-to-end multiple clustering approach that incorporates a multi-modal subspace proxy learning framework. The paper is well written and well organized. However, there are several concerns in the current version of the paper that addressing them will increase the quality of this paper.

Strengths

1 The authors' idea of using large models to aid clustering is novel. 2 The paper is clearly structured and easy to understand. 3 The paper has sufficient experiments to support its point of view.

Weaknesses

1 The authors point out that different clustering results can be given for different customization needs of users. Then it will bring several associations (not necessarily accurate): a. What should be done if the user's demand is exactly opposite to the potential clustering distribution? b. The experiments do give different clustering results for different demand types, if the user proposes a new type of demand, can the model also adaptively adjust? 2 Figure 2 is well drawn but could be further improved, some icons and fonts need to be adjusted. 3 The authors point out that their model is capable of outputting clustering results directly, and then there should be a corresponding formula to represent this. In addition, it is hoped that the authors will discuss further why, if it is not a difficult task to output clustering results directly, few previous methods have done so. 4 Authors should add details about the dataset, such as data size, feature types, etc.

Questions

Considering that the authors did not add an appendix, are there any other discussions or experiments?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes.

Summary

This paper introduces an end-to-end multi-clustering method that integrates a multimodal subspace proxy learning framework. It combines text prompts expressing user preferences with corresponding visual representations to achieve clustering based on user interests.

Strengths

1.The clustering task, driven by user interests, aligns better with user preferences and is more applicable to real-world scenarios. 2.The experimental results are promising, and the methodology is clear and logical.

Weaknesses

1.The contributions of the paper are not very clear. At first glance, it appears to merely combine CLIP and GPT, lacking innovative architecture. 2.The baseline methods chosen for comparison are neither cited nor introduced. 3.Section 5 discusses only limitations, lacking a discussion on broader impacts.

Questions

The evaluation metrics mentioned in the paper require comparing results with ground truth values. How were the multiple clustering ground truth values in the dataset obtained? How is the accuracy of these ground truth values ensured?

Rating

6

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

The authors only address the limitations of their work and do not discuss broader impacts.