FASTopic: Pretrained Transformer is a Fast, Adaptive, Stable, and Transferable Topic Model

Summary

The paper proposes FASTopic, a topic model using pre-trained document embeddings and embedding transport between documents and topics as well as words and topics. The minimized objective function is then a combination of the DSR and ETP which is optimized via finding topic, and word embeddings.

Strengths

- Interesting idea which is well documented - Initial experimental results show potential - potential benefit of getting interpretable word-embeddings while training topic model

Weaknesses

- Overall, I quite like the paper and the idea. However, I have some reservations, primarily due to the aggressive tone of the presentation, starting from the abstract. The paper repeatedly criticizes comparable methods, while presenting its own method as the ultimate and superior topic model. Such claims need robust support through experimental evidence. Although the experiments provide some validation, the tone of the paper creates a discrepancy, making the experimental results appear insufficiently convincing. Overall, more results are needed to back up the strong claims made in the paper. Below are some suggestions that, in my opinion, would support the claims more effectively. If these suggestions are unjustified, I would appreciate an explanation. **Minor:** I think the high-resolution plots are causing some trouble. I cannot really scroll through the paper smoothly. 1. **Include further models** -> ETM [1], ProdLDA [2], CEDC [3], CTMneg [4]. While I wouldn't suspect ETM/ProdLDA to be better than any other of the models, the simple models CTMneg and CEDC have shown to outperform BERTopic. 2. **Incorporate other evaluation metrics**: since evaluation is very difficult [5]. E.g. use some presented in [3] or [6]. 3. **How do you measure training time for all of the models?** - 3.1) Are all steps, including the document encodings, part of the taken training time? - 3.2) Do you use the same encoding model for all comparison models (where applicable)? - 3.3) Given that FASTopic outperforms BERTopic in terms of speed, I wonder for how many epochs you are training your model, and what it is that takes so long in BERTopic? The document encoding steps are the same for both models and while dimensionality reduction takes some time using UMAP, I have a hard time believing that it is slower than training FASTopic for a reasonable amount of epochs. 4. **How did you choose the number of topics for BERTopic?** Did you use Kmeans instead of HDBSCAN or hierarchicaly reduced the number of topics? 5. **How many parameters does FASTopic have compared to the other neural models?** [1] Dieng, A. B., Ruiz, F. J., & Blei, D. M. (2020). Topic modeling in embedding spaces. Transactions of the Association for Computational Linguistics, 8, 439-453. [2] Srivastava, A., & Sutton, C. (2017). Autoencoding variational inference for topic models. arXiv preprint arXiv:1703.01488. [3] Adhya, S., Lahiri, A., Sanyal, D. K., & Das, P. P. (2023). Improving contextualized topic models with negative sampling. arXiv preprint arXiv:2303.14951. [4] Thielmann, A., Reuter, A., Seifert, Q., Bergherr, E., & Säfken, B. (2024). Topics in the haystack: Enhancing topic quality through corpus expansion. Computational Linguistics, 1-37. [5] Hoyle, A., Goel, P., Hian-Cheong, A., Peskov, D., Boyd-Graber, J., & Resnik, P. (2021). Is automated topic model evaluation broken? the incoherence of coherence. Advances in neural information processing systems, 34, 2018-2033. [6] Stammbach, D., Zouhar, V., Hoyle, A., Sachan, M., & Ash, E. (2023). Revisiting automated topic model evaluation with large language models. arXiv preprint arXiv:2305.12152.

Questions

- Are the created word embeddings semantically meaningful? Would clustering the word embeddings e.g. [7] give meaningfull topics? If yes, this is an additional benefit which I would suggest to include in the paper. [7] Sia, S., Dalmia, A., & Mielke, S. J. (2020). Tired of topic models? clusters of pretrained word embeddings make for fast and good topics too!. arXiv preprint arXiv:2004.14914.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Not needed

Summary

This paper introduces a fast, adaptive, stable, and transferable (FAST) topic modeling paradigm by using dual semantic-relation reconstruction (DSR) to model topic-document and topic-word relations. It enhances topic modeling by incorporating a embedding transport plan (ETP) method to address relation biases. Experimental results show FASTopic outperforms existing baselines.

Strengths

1. This work introduces FASTopic by integrating the DSR paradigm, which is straightforward and provides a fresh aspect on handling semantic relations in topic modeling. Authors further propose ETP method to avoid relation bias issue. 2. Comprehensive experiments show FASTopic's superiority in effectiveness, efficiency, and adaptability, stability, and transferability. 3. The paper is well-written, and the code is available.

Weaknesses

1. The method's performance may heavily rely on the document embedding model, which makes it harder to interpret the model results. It need to futher discuss that how specific changes in embeddings affect topic modeling. 2. The misplacement of tables and figures, such as Table~1, could potentially distract readers but do not detract significantly from the content's quality. But it does not matter.

Questions

1. In Figure~2, is it intentional to design the variation in line thickness ($\pi_{11}$, $\pi_{12}$ and $\pi_{13}$) ? 2. Could the authors provide more detailed motivation and experimental validation for the choice of using pretrained document embedding model? How if train embedding model together? 3. What are the performance metrics (both training and inference) when FASTopic is run on a CPU environment? 4. Although it is claimed that FASTopic performs well without extensive hyperparameter tuning, the paper does not discuss experiments varying hyperparameters. Could you elaborate on how sensitive FASTopic is to changes in its hyperparameters?

Rating

7

Confidence

3

Soundness

3

Presentation

4

Contribution

3

Limitations

No potential negative societal impacts.

Summary

This paper proposes a fast, adaptive, stable, and transferable topic model, FASTopic. Instead of using the VAE or clustering method, it incorporates a new model structure named Dual Semantic-relation Reconstruction (DSR). DSR learns topics by directly optimizing the semantic relations among topics, documents, and words. The semantic relations are further regularized by an Embedding Transport Plan (ETP) method as an optimal transport problem. Experiments demonstrate the effectiveness of the proposed model.

Strengths

1. The paper is well-written and easy to follow 2. The proposed DSR framework is simple and neat. 3. The experiments and ablation studies demonstrate the effectiveness and efficiency of the proposed FASTopic method.

Weaknesses

1. [76,64] should be compared as baselines in the experiments since they both incorporate optimal transport objectives in their model as FASTopic does. 2. As far as I know, the time complexity of solving the optimal transmission problem is very high. Is there any technique used in FASTopic to efficiently derive a solution? Furthermore, it would be great to theoretically analyze the time complexity of the current FASTopic framework. 3. Is it fair to compare FASTopic, which takes document embeddings as input extracted by sentence-BERT, with other baselines in the experiments? 4. As LLM has shown great performances in many NLP tasks, I believe it is necessary to include a discussion about the reasons and advantages of developing a topic model that is not LLM-based.

Questions

See weaknesses.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

Yes

Summary

The author found that existing methods (VAE-based or clustering) suffer from low efficiency, poor quality of topic words, and instability. To address these issues, this paper proposes a novel topic modeling paradigm called Dual Semantic-Relation Reconstruction (DSR) for efficient modeling of semantic relations among three types of embeddings: document, topic, and word embeddings. The author attributes the low quality and instability of previous methods to relation bias issues, leading to repetitive topics and inaccurate document-topic distributions. To tackle this, the paper introduces the Embedding Transport Plan (ETP) to regularize relations among the three embeddings. Combined, DSR and ETP form the proposed topic model, FASTopic, which is evaluated on six benchmark datasets, showing encouraging performance.

Strengths

1. The paper is clearly presented and easy to follow, with an explanatory and easy-to-understand mathematical formulation. 2. The proposed DSR objective is effective, significantly reducing training time compared to VAE-based methods such as ECRTM and CombinedTM. 3. The experimental evaluation is extensive, showing that FASTopic consistently outperforms multiple strong baselines on topic coherence and topic diversity, while also demonstrating advantages in terms of running speed and transferability. 4. The paper demonstrates the model's robustness under multiple numbers of topics (K=75, 100, ..., 200).

Weaknesses

1. Tables 6, 8, and 9 present the ablation study of FASTopic using ETP and parameterized softmax. ETP, compared to the ECR used in the previous VAE-based method ECRTM, adds semantic relations between topics and documents. The paper lacks proof the dual transmission effectiveness of ETP by not comparing it with ECR (which only includes semantic relations between topics and words) and another case that only includes semantic relations between documents and topics. 2. I am concerned that while the DSR training method improves efficiency compared to VAE-based methods, it may make it difficult to build meaningful relations between topic embeddings and document embeddings for some long-text corpora due to its singular objective. 3. It would be nice to showcase a comparison of topic words or transferability between FASTopic and ECRTM to demonstrate that FASTopic offers improvements in multiple dimensions, not just speed.

Questions

1. In Line 76, the author thinks that methods like Topicgpt, which uses LLM to describe topics, deviate from the original LDA setting. Can the author provide a more detailed explanation in this section? 2. In Line 132, it is mentioned that some studies think that repetitive topics and less accurate doc-topic distributions are due to a large number of topics (K being set too high). The paper, however, attributes these issues mainly to relation bias. Does a large number of topics also affect FASTopic in the same way?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

1. The proposed method relies on embedding transport semantic relations and may be limited by the max input length of pre-trained document embedding models. 2. See Weakness 3.

Rebuttal Acknowledgment

Dear authors, thank you for your answers. All of my questions have been adequately answered, or have been already answered within the original manuscript. Thank you for bringing that to my attention. I have adapted my score accordingly 5 -> 7