Summary
This paper considers the problem of multi-view clustering from an information theoretic perspective. It focuses on representation learning, and optimizes said representation to improve down-stream clustering performance (with k-means). It introduces an Information Bottleneck based loss function, which considers consistency between views, redundancy and sufficiency of representations in addition to the traditional likelihood based reconstruction loss. It introduces 2 methods SCMVC and SUMVC based on some or all of these additional loss terms.
The paper then analyzes the model effectiveness by establishing a connection to the Bayes Error Rate, and also shows good experimental results across some multi-view datasets.
Strengths
- Introduces a novel approach for representation learning for Mmulti-View Clustering (MVC) based on information theoretic criteria.
- Proposes two separate methods, both of which show good experimental performance.
- Has mathematical rigor in both breaking down the loss function, as well as in the theoretical analysis that follows.
- Compares against multiple state-of-the-art MVC methods and shows superior performance on the chosen datasets.
- Conducts a wide range of experiments, including ablation studies and parameter sensitivity analysis.
Weaknesses
- The presentation of the mathematical parts of the preliminaries, discussion and analysis is quite opaque. Variables and terms are often explained only after they have already been used in equations. The equations tend to be quite cluttered and difficult to parse.
- The datasets chosen are less than ideal, and are mostly derivative from single-view datasets. There are other multi-view datasets such as NUS-WIDE and 3-Source News which are potential candidates for publicly available MV datasets.
- The assumptions (and thus applicability) of the methods are restrictive. I.e. assuming mutual redundancy across all views. As they mention in the supplementary, their methods perform poorly on heterogeneous MV data, limiting their applicability.
- The paper was not self-contained; the supplementary material was essentially required to understand the a lot of the details. For eg. the limitations of the methods were only mentioned in the supplementary material. The supplementary material must not be used as additional pages, and the paper itself should be able to stand alone.
Questions
Overall questions:
- This paper is more about representation learning for clustering than clustering itself, right? Similar to spectral clustering. Not that that's a bad thing, it's just a little misleading. You spend a lot of the paper talking about MVC but the actual clustering is just k-means on top of a learned representation. It might help to be clear about this up front.
- As I mentioned in the second point of the Limitations, it seems like this method is also weak to only partial information existing in views. How would you remedy this? Would ensemble variational methods work here as well?
- For the MNIST experiments, I am confused about your choice to use pairs of the same digit (but written by different people) as two views. The data distribution of both the views are very similar, then. Also, if you pair up digits, shouldn't your dataset size be 35000 (halved) and not 70000? Or are you using both (A, B) and (B, A) for each pair? If so, this doesn't really seem like a good multi-view dataset since the views themselves are basically indistinguishable overall.
- In table 3, the 3rd row seems to not have any useful information. It's clear that $L_{rec}$ would be required since you're using an auto-encoder. Instead, maybe you can have $L_{rec} + L_{suf}$ which you don't have here. That would be interesting to see.
I will leave line-by-line comments here (since there doesn't seem to be a better place for this). There are a few typos here and there, but I won't bother too much with those:
- [Line 32] Did you mean maximizing MI between representation* and output (not input and output)
- [Line 82] "... to quantify amount of" -- sentence is not complete.
- [Section 3] You use y, z without explaining/defining what they are. Clustering is an unsupervised task, so what does this mean in this context? I'm guessing it means cluster assignment but it isn't clear.
- [Line 164] I may be wrong but isn't $\phi^i$ the generational parameter set, and not $\theta^i$?
- [Line 180] What is a "pseudo-label?" A cluster assignment? You should define this earlier.
- [Line 164] What do you mean by unique distribution?
- [Line 186] This equation is very hard to parse. For cleaner appearance, you should consider using \left( and \right) to have larger parantheses. You could also remove the superscripts just for these long equations where there is no $j$, and just leave a note below. Also, shouldn't there be a conditioning on y in the first term? Lastly, $KL$ -> $D_{KL}$.
- [Line 198] maximizing*
- [Equation 9] There should be no expectation here, right? The $D_{KL}$ absorbs it, I think. Also, for the second term. the first distribution uses $z^j$ instead of $z^i$.
- [Line 243] Should it be $z^j_m$ here? Also, what is $P^{\otimes n}$?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
- [Author mentioned] Heterogeneity in the data (eg. in terms of dimensions of features) affects performance significantly.
- Restrictive assumptions are made on mutual redundancy between views. Multi-view data often has only partial information available in each view. I.e. you might need more than one view to get the complete picture. This also seems like a weakness of the methods.