Response to weakness 4
W4. The aim and construction of contrastive loss in this work seem a bit confusing to me. It is suggested to explain it in more detail.
A4. Thanks for the comment, and we are sorry to confuse you about the contrastive loss. According to your suggestion, we further explain the design idea of contrastive loss as follows in the manuscript:
For any sample $i$, we pair features on $2N$ channels in pairs ($4N^2$ feature pairs in total), and classify these feature pairs into 2 categories, namely positive pairs and negative pairs, where positive pairs consist of shared features from different views, and the remaining “shared-private” and “private-private” feature pairs are negative pairs. Our goal is to minimize the distance between positive instance pairs (the denominator part), while maximizing the distance between negative pairs (the numerator part). Here we not only consider maximizing the difference between private features decoupled from the different view ($o_{i}^{(u)}$ and $o_{i}^{(v)}$, $u\neq v$), but also reduce the similarity of shared features and private features from both intra-view and inter-view levels ($s_{i}^{(u)}$ and $o_{i}^{(v)}$, including $u=v$). This approach fulfills the design objective of the dual-channel model, which is to encourage consistency between shared features from different views while maintaining a clear distinction between view-proprietary feature and shared or other view-proprietary features.
In addition, considering the efficiency of matrix calculation, we set the coefficient of the contrastive item of shared features and private features to 2 in the numerator part, so that the entire paired similarities can be obtained by a single matrix multiplication, i.e., $[s_i^{(1)}; s_i^{(2)};…; s_i^{(N)}; o_i^{(1)}; o_i^{(2)};…; o_i^{(N)}]\times [s_i^{(1)}; s_i^{(2)};…; s_i^{(N)}; o_i^{(1)}; o_i^{(2)};…; o_i^{(N)}]=Q\in \mathbb{R}^{2N\times 2N}$. The matrix $Q$ can be divided into four sub-matrices whose size is $N$ by $N$. The sub-matrix in the upper left corner stores the similarity values of the denominator in Eq. (1), the sub-matrix in the lower right corner stores the values of the second item in the numerator, and the remaining two submatrix, which are transposed to each other, store the similarity values of the first item in the numerator. Therefore, we set the coefficient of the first term in the numerator to 2. At the same time, the diagonal elements of $Q$ are removed, so that the number of elements in the numerator and denominator parts is $\frac{1}{3N^{2}-N}$ and $\frac{1}{N^{2}-N}$ respectively.
About limitation of the paper:
In the conclusion, we explain the limitations of our method and further improvement directions in the future, such as adding the consideration of label correlation to the multi-label classification problem in the model. At the same time, for missing views and partial multi-label, in addition to introducing prior information, more complex view completion and pseudo-label prediction can be performed to improve the discriminability of the model.