Interpretable Dimensionality Reduction by Feature Preserving Manifold Approximation and Projection

>1 Unclear Motivation: Why perform SVD on instead of Xi or X? The motivation is not introduced clearly. Thank you for your question. The motivation for performing SVD on $W_iX_i$ rather than $X_i$ lies in capturing the local structure of the data. $W_i$ applies weights to emphasize the local neighborhood of each data point $x_i$ , ensuring that the SVD captures the most significant local variation. This approach provides a more accurate tangent space approximation by considering the local geometry, which would be lost if unweighted $X_i$ were used. We have clarified this motivation in the revised manuscrip, which is highlighted in yellow in section 3.1. >2 There is an error in (5), since Vi/Vj is known variable calculated in (2). Thank you for you suggection. The objective of Eq (5) is to compute the optimal alignment $O_{ij}$, which is an orthonormal matrix ($\textit{O}(d)$), given the tangent space $V_i$ and $V_j$. Thus, $O_{ij}$ is regarded as an approximation of $V_jV_i^T$. We have clarified this point further in the manuscript highlighted in yellow in section 3.2. >3 What are the advantages of the method compared with the existing works, especially in terms of computational complexity, running times, and robustness? Thank you for your question. FeatureMAP provides several distinct advantages over existing methods, particularly with its unique focus on feature preservation, alongside considerations of computational complexity, running times, and robustness: Feature Preservation: Unlike many dimensionality reduction methods, such as UMAP or PHATE, FeatureMAP explicitly embeds tangent spaces, allowing it to preserve feature-level relationships in addition to the manifold structure. This makes FeatureMAP highly interpretable, as it highlights which features contribute most to the embedding, a capability lacking in most existing methods. Computational Complexity and Running Times: While FeatureMAP involves additional steps, such as tangent space computation and alignment, it achieves comparable running times to methods like densMAP and TriMAP by leveraging approximate nearest neighbor searches and efficient optimization techniques (e.g., stochastic gradient descent). For datasets of 50,000 samples, FeatureMAP requires slightly more time than UMAP (150 seconds vs. 30 seconds) due to its enhanced functionality, but this is justified by its ability to preserve both feature importance and manifold geometry. Robustness: FeatureMAP is robust to variations in data density and structure, as its tangent space alignment and embedding steps balance local and global structural preservation. By aligning tangent spaces through cosine similarity and optimizing embeddings to maintain agreement with the original geometry, FeatureMAP ensures stable and consistent results, even for complex datasets. We have included a comparative analysis of the major existing methods in Section A.1.7 of the APPENDIX. >4 What’s the significance/application value of the method in real-word applications, especially under the LLM era? Thank you for this insightful question. FeatureMAP provides substantial value in real-world applications, particularly in the era of large language models (LLMs), where interpretability and efficiency are essential: - Interpretability of High-Dimensional Representations: FeatureMAP preserves and visualizes feature-level importance in high-dimensional data. It helps uncover the key features or dimensions contributing to downstream tasks, enhancing interpretability in applications such as sentiment analysis, question answering, and content generation. - Feature Understanding in Fine-Tuning and Evaluation: FeatureMAP identifies which features are preserved, transformed, and lost during LLM fine-tuning. This insight aids domain adaptation and improves model generalization by enabling better understanding of feature interactions. >5 This method involves multiple independent steps and is not a complete optimization problem. Will this lead to difficulties in obtaining the global optimal solution and the problem of the previous step affecting the next result? Thank you for this important question. FeatureMAP addresses the challenges of combining multiple independent steps by using a unified loss function that jointly optimizes kNN graph embedding and anisotropic projection based on tangent space embedding. This approach minimizes the risk of suboptimal results affecting subsequent steps. The loss balances two key objectives: kNN Graph Embedding and Tangent Space Alignment and Projection. By optimizing these objectives simultaneously, the method aligns local and global structures. The use of stochastic gradient descent (SGD) iteratively minimizes the combined loss, ensuring convergence to a solution that integrates graph embedding and tangent space alignment effectively. We have revised the algorithm pseudocode in Section A.1.6 to enhance clarity.

>Weaknesses: The way to solve Eq. (11) is not described in detail. Thank you for raising this question. We have expanded the description of the computation for Eq. (11) in Section A.1.4 of the APPENDIX by including details of the SGD calculation. The updated section is highlighted in blue. > A.4 seems hard for me to implement the FeatureMAP. Thank you for your question. We have revised the pseudocode for FeatureMAP in Section A.1.6 of the APPENDIX to enhance readability. The main algorithm is now divided into seven steps, with each step presented in its own pseudocode.

>Weaknesses: little or no support is provided that the resulting dimensionality reduction better preserves important features of the source space, which is the main motivation of the method. Thank you for this valuable comment. The core motivation for proposing FeatureMAP is its ability to preserve and visualize important features of the source space, which existing methods, such as UMAP and t-SNE, fail to achieve because they only consider embedding the topological structure by kNN graph. Unlike these methods, FeatureMAP embeds tangent spaces, enabling it to explicitly preserve feature directions and feature importance in the reduced space. This capability allows users to not only visualize the embeddings but also understand which features contribute most to the low-dimensional representation. To validate this feature-preservation property, we conducted experiments on the MNIST, Fashion-MNIST, COIL-20 dataset and adversarial examples. FeatureMAP successfully highlights the directions and importance of features, demonstrating its robustness and interpretability in scenarios where understanding the role of individual features is critical. >the comparison to other methods is mostly qualitative and makes it hard to argue the new method is somehow superior to previously proposed ones. Visualization results (Figure 12) are subjective ("clearly shows distint clusters of different digit groups"), quantitative measures (Table 1) are inconclusive, and running time is usually worse (Figure 13). Thank you for your feedback. FeatureMAP uniquely visualizes feature directions and importance, enabling data interpretation in embedding space, unlike UMAP or t-SNE. This interpretability is our principal contribution. - Quantitative Performance: FeatureMAP outperforms competitors in local metrics (e.g., continuity, kNN accuracy) and ranks consistently high on global metrics (e.g., Shepard goodness, stress), as shown in Table 1. - Computational Cost: FeatureMAP’s higher runtime (e.g., 150s vs. UMAP’s 30s) is justified by its extended functionality, including tangent space embedding and feature visualization, which surpass solely kNN-based methods. - Visualization and Density: FeatureMAP preserves density like densMAP while producing distinct clusters. These visualizations (Figure 12) align with superior quantitative performance, reinforcing the method’s effectiveness. We have further clarified these points in the manuscript to strengthen our arguments and provide additional context to the comparisons in Section A.2.4 >Questions 1 Thank you for raising this concern. Yes, our method FeatureMAP is equivalent to UMAP when considering only the topology (kNN graph) and the first term of the loss function (distance distortion). We have clarified this in the manuscript by appropriately attributing the corresponding techniques to UMAP and have revised Section 3, with updates highlighted in red. >Questions 2 Thank you for your question. The second optimization term in FeatureMAP preserves both local density and source features, uniquely enabling the visualization of feature directions and importance—capabilities not available in UMAP or t-SNE. Evidence from the MNIST, Fashion MNIST, COIL-20 datasets, and adversarial examples demonstrates FeatureMAP's ability to highlight feature directions and their influence in the embedding space. Furthermore, as shown in Figures 8, 9, and 10, FeatureMAP outperforms UMAP in density preservation. >Questions 3 Thank you for raising this concern. We have added a reference to [Chari & Pachter, PLOS Comput. Biol. (2023)] and discussed its relevance to our method in Related Work (Section 2). It is worth noting the ongoing debate around UMAP and t-SNE in visualization applications, highlighted both in research publications and on social media. On one side, [Chari & Pachter, PLOS Comput. Biol. (2023)] argue that such embeddings can be arbitrary and misleading, comparing them to forcing data into an elephant shape. On the other side, studies like [Lause et al., PLOS Comput. Biol. (2024)], [Damrich et al., ICLR (2022)], and [Van Assel et al., NeurIPS (2022)] show that while these methods do not preserve high-dimensional distances, they can still provide biologically meaningful insights through spectral clustering and probabilistic graph coupling analyses. Our method aligns with the latter perspective. Specifically, FeatureMAP extends conventional UMAP by incorporating local tangent space embedding, thus improving the interpretability in terms of both feature importance and density. This augmentation provides new functionality with feature direction and importance to better interpret the embedding. Also, it preserves the denstiy, which enhances the understanding the local and global embedding.

Thank you so much for your prompt response, and thank you again for raising this important question. To address your concern, we have revised the manuscript to include experiments and figures in Section A.2.5 of the APPENDIX (highlighted in red), illustrating FeatureMAP's feature-preserving strength compared to UMAP, t-SNE, and PCA. Briefly, the source features we aim to preserve are analogous to the feature loadings in PCA's biplot, which explain a feature's contribution to the data distribution. However, this interpretation is not directly applicable to nonlinear dimensionality reduction methods like UMAP and t-SNE, as they do not provide feature loadings in the embedding. FeatureMAP addresses this limitation by introducing the computation of tangent spaces and feature gradients over the manifold, thus making the feature loadings avaiable to locally interpret the embedding. Specifically, FeatureMAP is able to visualize the feature contribution by vector filed (Figure 1f in Section 1 and Figure 15 in Section A.2.5), where the feature direction indicates how the feature changes along the data embedding. In addition, the magnitude of feature contribution, i.e., feature importance, directly indentify the key features for each sample, with applications on revealing edges of MNIST, Fashion MNIST and COIL-20, and explaining the misclassification of advarserial attack. Moreover, due to the feature-preserving machanism, the embedding also preserve the data density because we model the local data points in one hyperecllipse. In summary, source features in this dimensionality reduction context represent feature contributions to the data embedding in terms of both direction and magnitude. FeatureMAP achieves this through feature gradients, providing practical applications in visualizing feature directions and leveraging feature importance to interpret embeddings. We hope this updated revision addresses your concern. Please feel free to share any additional comments if there are further issues.

Dear Reviewer, I hope this message finds you well. I am writing to kindly inquire about the status of the feedback for the revision based on your suggestion. I greatly value your time and expertise in reviewing our work. If there are any clarifications or additional information required from my side, please do not hesitate to let me know. Thank you very much for your efforts in providing constructive feedback.