Index $t$-SNE: Tracking Dynamics of High-Dimensional Datasets with Coherent Embeddings

$t$-SNE is an embedding method that the data science community has widely Two\ninteresting characteristics of t-SNE are the structure preservation property\nand the answer to the crowding problem, where all neighbors in high dimensional\nspace cannot be represented correctly in low dimensional space. $t$-SNE\npreserves the local neighborhood, and similar items are nicely spaced by\nadjusting to the local density. These two characteristics produce a meaningful\nrepresentation, where the cluster area is proportional to its size in number,\nand relationships between clusters are materialized by closeness on the\nembedding.\n This algorithm is non-parametric, therefore two initializations of the\nalgorithm would lead to two different embedding. In a forensic approach,\nanalysts would like to compare two or more datasets using their embedding. An\napproach would be to learn a parametric model over an embedding built with a\nsubset of data. While this approach is highly scalable, points could be mapped\nat the same exact position, making them indistinguishable. This type of model\nwould be unable to adapt to new outliers nor concept drift.\n This paper presents a methodology to reuse an embedding to create a new one,\nwhere cluster positions are preserved. The optimization process minimizes two\ncosts, one relative to the embedding shape and the second relative to the\nsupport embedding' match. The proposed algorithm has the same complexity than\nthe original $t$-SNE to embed new items, and a lower one when considering the\nembedding of a dataset sliced into sub-pieces. The method showed promising\nresults on a real-world dataset, allowing to observe the birth, evolution and\ndeath of clusters. The proposed approach facilitates identifying significant\ntrends and changes, which empowers the monitoring high dimensional datasets'\ndynamics.\n

Paper

Similar papers

© 2026 NYSGPT2525 LLC