Summary
The authors address the interesting question of how to disentangle the relevant manifold directions properly in the latent space of manifold learning normalizing flows (MLF). After a comprehensive theoretical background, the main motivation is presented in form of a toy example of a simple noisy line embedded in $\mathbb{R}^{2}$. When varying the latent dimension $z_{i}$ of a RNF and setting the remaining latent variables to $0$, the corresponding contour lines are partially aligned. Doing the same with the proposed method, CMF, leads to orthogonal contour lines and thus to a meaningful latent representation where one latent variable encodes the line and the other the (noisy) off-manifold direction. This intuition leads straightforwardly to enforcing the orthogonality of the flows Jacobian matrix. To do so, the authors propose a new cost function for learning manifold structures using an NF by adding a penalty term which ensures that the off-diagonal entries of the Gram matrix (Jacobian-transpose-Jacobian) are $0$ at the one hand, and are sparse at the other due to the l1-norm. The effecitiveness of the method is tested in experiments on toy-data, tabular data and 32x32x3 image data.
Strengths
The paper is overall well written. The theoretical background is exhaustive and well-explained. The example in 4.1 serves as a good illustrative motivation. The experiments are comprehensive.
Weaknesses
The contribution is somewhat marginal. Though citing [1], they don't mention that the corresponding authors propose a very similar penalty term. To my understanding, the only differences of the penalty term in [1] are
a) that in [1] the diagonal entries are penalized to be $1$ whereas in this work the diagonal entries are unconstrained.
b) [1] used the L2-norm whereas in this paper the L1-norm is used.
Given the great similarity, the penalty term suggested by [1] should be included in the comparison. What do we gain by not constraining the flow to be an isometry? In addition, the authors use the M-flow for benchmarking although a better method for overcoming the limitations of M-flows (namely that the log-determinant term is not considered while training) was already proposed in [2]. At least citing and ideally comparing with [2] should be part of the paper. The same is true for comparing with [3]. It would be nice to see how the contour lines differ from the method introduced in [3].
Furthermore, I am not sure to understand the reasoning behind Definition 4.1. To my understanding, every differentiable manifold is a canonical manifold as one is free to choose the basis of the tangent space to be orthogonal. Thus, such a basis always exists which is the only requirement to be a canonical manifold. Definition 4.1. is simply an existence statement and not useful as such. I think what the authors wanted to define is a manifold equipped with a chart s.t. the chart induces an orthogonal basis of the tangent space. However, this is very much the definition of the principal component flow [3]. Please enlighten me in case I totally misunderstood something.
I am also confused about the comparison with rectangular NF (RNF) in Figure 2. By definition for RNF we have that $d<D$. However, in Figure 2 the author uses $d=D$. This is a contradiction. If $d=D$, a standard NF can be used rather than an RNF. In addition, one claimed strength of the proposed CMF is the sparisity induced by the l1 norm. Thus, the relevant dimensions should be found automatically which elevates, in theory, the necessity to estimate $d$ a priori. Then, why not always set $d=D$ and then apply the proposed penalty term? The number of prominent latent dimensions should correspond to the true intrinsic dimensionality of the data.
Finally, the notation for the manifold, data manifold, and learned manifold is confusing. In line 74 or line 90, $\mathcal{M}$ is a lower dimensional manifold. In line 83, $\mathcal{M}_{\theta}$ is introduced without explaination. In line 212, $\mathcal{M}$ is referred to as the learned manifold and in the same line, the data manifold is introduced without further explanation. In addition, a new notation for a canonical manifold is suggested. I find the various versions for a manifold confusing and don't see the added benefit of it (especially given that Definition 4.1. is not useful in my opinion).
[1] Eike Cramer, Felix Rauh, Alexander Mitsos, Raúl Tempone, and Manuel Dahmen. Nonlinear 350 isometric manifold learning for injective normalizing flows. arXiv preprint arXiv:2203.03934, 351 2022.
[2] C. Horvat and J.-P. Pfister. Denoising normalizing flow. In Advances in Neural Information Processing Systems, volume 34, 2021.
[3] Edmond Cunningham, Adam D Cobb, and Susmit Jha. Principal component flows. In Interna410 tional Conference on Machine Learning, pages 4492–4519. PMLR, 2022.
Questions
1. Does Definition 4.1 really makes sense (see above)
2. Is the comparison with RNF in Figure 2 fair (see above)? By definition of RNF, it must be $d<D$. Please elaborate.
3. Why not always set $d=D$ and then apply the proposed penalty term (see above)?
4. Figure 1. Why is the density varying greatly when only varying one dimension? Shouldn't be uniform if the density is properly learned?
5. What is the quality in terms of density estimation? For instance, what is the KS statistics compared to the true density, or, at least, does it sum to 1?
6. Why not benchmark with PCA flow? It would be interesting to see if this method can overcome some of its limitations (see above).
7. Moebius band with RNF seems to be poorly learned. Why?
8. Why prominent latent dimensions are the ones with the greatest $G_{ii}$? Is this a mathematical statement/proposition or intuition?
9. line 305: in which case would an orthogonal basis not be an optimal representation?
10. What happens if we use the L2 rather than L1 norm?
11. How are prominent latent dimensions defined? Do they correspond to the true intrinsic dimensionality? Is there a clear cut-off in the magnitudes of $|G_{kk}|$ in e.g. MNIST? If not, a somewhat arbitrary threshold needs to be defined, isn't it?
*Minor*:
+ line 70-72: the statement is only true for large $n$
+ line 83: What is $\mathcal{M}_{\theta}$?
+ line 83: "to encorage $x\in \mathcal{M}_{\theta}$" do you mean $g_{\phi}(z)$ rather than $x$?
+ line 114: typo: "due to lack..."
+ line 145: noise is not perpendicular to line
+ line 217: mention reason (maybe in a footnote)
+ line 225: $\gamma$ not crucial but had an impact on training time? What is crucial for you?
+ line 244: bad wording "the expected the canonical manifold learning"
+ line 297: Inconsistent acronym: M-flow vs. Mflow
+ line 299: inferior lok-likelihood calculations of Mlow...where shown?
+ line 309/310: wrong statement: there are diffusion models with lower dimensional latent space. However, by default indeed the latent space has the same dimensionality
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Limitations
Major limitations are addressed in Section 6. However, some questions are unresolved such as: how to find/define the prominent latent dimensions without an intrinsic dimensionality estimator?