Summary
This paper proposes DIVA, a novel method for clustering high-dimensional datasets using variational autoencoders. DIVA is a Dirichlet Process Mixture model fitted on the latent space of a VAE, and under variational inference assumptions can be optimised through its ELBO. Its key, claimed innovation is to alleviate the user from having to choose the number of clusters a priori, which DIVA learns adaptively from data.
Strengths
* This method provides a, to the best of my knowledge, novel method.
* The method solves an indeed open and important problem in the deep clustering literature, how to adaptively choose and learn the number of clusters from data, without requiring this to be set through prior knowledge of a user.
* The background provided is informative, and the mathematical formulation of the method is solid.
* The combined method in Algorithm 1 is non-trivial, combines several techniques for optimising the model, and is of interest to the community.
Weaknesses
* My main concern is that the method’s promise of alleviating the need to know the number of parameters a priori is only to some degree satisfied. While the number of Gaussian components are learned from the data, this number is determined by other hyperparameters of the model. In particular, I would like to ask the authors to comment on all hyperparameters that the method introduces and that need to be set, and specifically comment on the selection and effect of the concentration parameter alpha.
* The authors evaluate their method experimentally mainly using the clustering accuracy metric (ACC) to cluster as the supervised label, and qualitatively with t-SNE. The former evaluation is limited: Image datasets typically contain more than one valid partition [2], and focussing on the one imposed by the supervised label is in the best case limited, in the worst case inappropriate. – This paper requires further evaluation, for instance in terms of its generative capability and the diversity of generated samples (generating actual samples, see also my point below), but also provide more details into what clusters are actually learned. On the latter point, Figures 7 and Figures 6 and 7 in the Appendix only show a subset of the learned clusters. It remains unclear if they are cherry-picked, what all the clusters learn, how the shown examples are sorted, and how many clusters there are in total.
* The writing is sometimes hard to understand or unclear and could be improved. There are frequent grammatical issues.
* What is stated as “Generative performance” are not samples from the VAE, but merely reconstructions. This should be changed, or samples should additionally be provided, which would give additional insight into the model. – To see what is learned in each cluster, visualising the clustered input examples would be sufficient.
* A minor point: Very established VAE work has the acronym DIVA which is unfortunate, namely “Ilse, M., Tomczak, J.M., Louizos, C. and Welling, M., 2020, September. Diva: Domain invariant variational autoencoders. In Medical Imaging with Deep Learning (pp. 322-348). PMLR.” I suggest picking a different name for the method, or no name at all.
I am happy to reconsider my current recommendation upon receiving a response to the above points.
Questions
* Table 1 should include the results of VaDE [1] and MFCVAE [2], where available. For instance on MNIST, VaDE (94.46 %) “outperforms” DIVA (94.01%) and is likewise within range of standard deviation of MFCVAE, questioning the claim by the authors that their method “outperforms state-of-the-art baselines”. – In particular, I would like to ask the authors why they decided to not report VaDE even though they were aware of and mentioning it in the related work.
* DIVA does not support multi-facet clustering (in comparison to MFCVAE), which should be stated as a limitation.
* Eq. (1), left-most looks odd. Could you please explain? It seems inconsistent with the graphical model.
* I would be interested in the active number of clusters on the experimental datasets.
[1] Jiang, Z., Zheng, Y., Tan, H., Tang, B. and Zhou, H., 2016. Variational deep embedding: An unsupervised and generative approach to clustering. arXiv preprint arXiv:1611.05148.
[2] Falck, F., Zhang, H., Willetts, M., Nicholson, G., Yau, C. and Holmes, C.C., 2021. Multi-facet clustering variational autoencoders. Advances in Neural Information Processing Systems, 34, pp.8676-8690.
Rating
3: reject, not good enough
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.