Summary
This work addresses the problem of aligning unpaired samples from multimodal data, whereby the problem is to find sample $x_i$ from modality $i$ that is best “related” to sample $x_j$ from modality $j$, when those two samples come from the observation of the same phenomenon according to two different modalities (e.g. sensors).
This is an important problem that arise in many endeavors including biology, and that has been somehow overlooked in the literature of multimodal representation learning, whereby large quantities of paired data (images and their corresponding captions) are available.
In a nutshell, the idea proposed in this work is as follows. Provided the following assumptions are valid, that is there exists a common latent space $Z$, which is entangled with a “perturbation signal” or label $t$, but that it is independent of modality specific noise $U_{i,j}$, then it is possible to estimate a common space for matching samples through the “bridge” across modalities offered by the perturbation signal $t$.
In practice, this amounts to training a classifier to predict $t$ given one of the available modality, which allows the creation of a transformed dataset endowed with a Euclidean distance. Such distance can be used to compute a matching matrix $M$, whereby entry $(i,j)$ represents the likelihood of $x_i$ being matched to $x_j$. Matching matrix $M$ can be obtained efficiently through the lenses of optimal transport matching with entropic regularization.
It should be noted that $M$ can then be used to construct pseudo-samples for any learning task that requires paired samples $(x_i, x_j)$ by replacing them with $(x_i, \hat{x}_j)$,
where the second term is obtained through the application of the matching matrix to a structural equation $f_\theta$, which is a parametric function with learnable parameters.
Several experiments on synthetic data, and a dataset whereby ground truth pairing is available, as well as experiments where no ground truth is available, indicate that the proposed method, in combination with optimal transport matching, provides an improvement over state of the art matching methods.
Strengths
* This work presents a simple method to produce a matching matrix that can be used to pair multimodal data. The algorithm only requires training of two classifiers and the execution of the Sinkhorn’s algorithm to compute the optimal transport matching.
* Experiments show that the proposed method using propensity scores in combination with optimal transport matching outperform the literature in terms of MSE and other specific metrics.
* Experimental result indicate, surprisingly, that the proposed method implicitly enforces invariant to modality specific information, which endows propensity score with an improved generalization, although this specific phenomenon is only hinted at and requires additional studies
Weaknesses
* The proposed method relies on strong assumptions, which are only superficially discussed. Relaxation of the conditional independence assumption A1 is discussed in appendix A, but only for the case of a small perturbation set $t \in \{ 0,1 \}$, which allows the authors conclude that under exact optimal transport, this is equivalent to operate on order preserving effects of $t$. The validity of the method also relies on A2, which allows computing propensity score from data alone, and not the shared latent space $Z$.
* The overall perturbation method, as well as the existence of a shared latent representation $Z$ is only vaguely described. I have looked up the key reference [Yang et al, 2021], but the lack of detail is uncomfortable. It is possible to get an intuition of how the method works, but I would prefer such important technical details to be spelled out clearly.
* There is no mention of the scalability of the proposed approach. Building a matching matrix can be a daunting task for some domains whereby a large amount of unpaired training data is available. In this case, the matching matrix could be either a big square matrix, or a very skinny and tall matrix that could pose some computational challenges. It would also be important to discuss about what would be the best operating scenario for the proposed method in terms of the number of samples per modality. In very unbalanced scenarios, would the classifiers using one or the other modality really behave similarly as per Equation (5)?
Questions
* Would it be possible to explain (and eventually add to the appendix) in more detail the setup from [Yang et al, 2021] which is used in this work? More precisely, it would be great to have the definition of the multimodal autoencoder, the existence of a shared latent space, and the details of the perturbations $t$ and their effects on $Z$. This would really help formalizing better the use of $t$ as a bridge between modalities.
* Can you provide a thorough discussion about the scalability of the proposed approach, the impact of imbalance in the size of the multimodal data (e.g. $n_1 >> n_2$), and the possible application to other domains with more abundant data?
* Can you elaborate more on the baselines you compare your method to? Lines 235-239 are not detailed enough to fully appreciate the differences w.r.t. the proposed method.
======= POST REBUTTAL UPDATE =======
Dear authors, thank you for the rebuttal. I have raised my score.
Limitations
Yes, the authors acknowledge that their work relies on strong assumptions, but suggest that empirical results indicate their method is robust to failures. Nevertheless, I could not find the discussion on the experimental results that clearly indicates such robustness.