The concept of unsupervised universal sentence encoders has gained traction\nrecently, wherein pre-trained models generate effective task-agnostic\nfixed-dimensional representations for phrases, sentences and paragraphs. Such\nmethods are of varying complexity, from simple weighted-averages of word\nvectors to complex language-models based on bidirectional transformers. In this\nwork we propose a novel technique to generate sentence-embeddings in an\nunsupervised fashion by projecting the sentences onto a fixed-dimensional\nmanifold with the objective of preserving local neighbourhoods in the original\nspace. To delineate such neighbourhoods we experiment with several set-distance\nmetrics, including the recently proposed Word Mover's distance, while the\nfixed-dimensional projection is achieved by employing a scalable and efficient\nmanifold approximation method rooted in topological data analysis. We test our\napproach, which we term EMAP or Embeddings by Manifold Approximation and\nProjection, on six publicly available text-classification datasets of varying\nsize and complexity. Empirical results show that our method consistently\nperforms similar to or better than several alternative state-of-the-art\napproaches.\n