Scalable Multi-phase Word Embedding Using Conjunctive Propositional Clauses

We acknowledge the weaknesses highlighted by the reviewer, which provide critical insights into our work, and we sincerely thank them for their thoughtful feedback and effort. Below, we address these points in detail: 1. An additional experiment was included in the similarity evaluation table for the FastText model, which is considered an enhanced version of Word2Vec. Interestingly, in this specific application, FastText yielded weaker results. This observation suggests that the classification adopted, while crucial for performance evaluation, relies heavily on human-constructed databases. These datasets, though widely accepted, may lack objectivity for this particular application and are generally subject to human biases. However, the primary focus of our work lies in the performance differences between the new model and the previous one by Bimal. These differences are significant and merit detailed analysis, which has been provided in **Section 4.1, paragraph two**. 2. The algorithm has been relocated to the appendix for better organization and accessibility. 3. The tables in the manuscript have been updated and improved. 4. During the preparation of our responses, we attempted to incorporate an additional application—document clustering. However, after initiating the application and observing preliminary results, we realized it would be infeasible to complete this work before the deadline due to time constraints. Nonetheless, we appreciate the reviewer’s suggestion and recognize the importance of exploring this application in the future. 5. The experiment related to the two-phase TM was repeated with an increased vocabulary size, expanded from 10K to 20K. The earlier vocabulary was limited and insufficient, leading to weaker results and an unfair comparison with prior models. With the expanded vocabulary, the results showed significant improvement. Regarding the convergence of results between models, this could be attributed to the nature of the application and the advanced state of the models employed, which are already optimized to perform competitively. The source code for these experiments is attached in the Supplementary Material. Please note that all source codes will be openly available online once the anonymous review period concludes.

We sincerely appreciate the reviewer’s insightful comments, which have significantly contributed to refining our work. Below, we provide detailed clarifications addressing the highlighted weaknesses and questions: ### **Weaknesses** 1. Firstly, in the introduction, we specifically emphasized a primary limitation of the previous method of embedding generation using TM. The earlier approach is computationally intensive and produces embeddings with limited reusability, impeding integration with larger systems, such as large language models, where embeddings need to be adaptable for use across various stages and applications. This constraint is a key barrier to scalability, underscoring the need for a more versatile embedding framework. Additionally, in paragraph 3 of the introduction, we described an experiment designed to illustrate the limitations of the prior method in the context of embedding scalability. Unlike traditional deep learning techniques, the previous approach does not easily extend to applications that incorporate multiple layers, which informed our primary research objective: to develop a scalable TM-based embedding method. Our contributions outline how this two-phase approach facilitates the construction of scalable embeddings (first contribution). We demonstrate this in Section 4.1 with multi-word vector evaluation (second contribution), and further validate it through data augmentation applications in Section 4.2 (third contribution). 2. This question is indeed critical, as we encountered challenges in comparing the embedding collections across the entire dataset vocabulary used in the new approach versus those in the previous approach. The previous method lacks comprehensive extraction of contextual information for the vocabulary included in training; rather, it focuses on specific elements directly related to the training task, without maintaining the primary advantage of transparency inherent in Tsetlin Machines (as noted in Section 3.2). Therefore, the extended execution time of the previous method was highlighted to illustrate that, despite its lengthy runtime, the generated embeddings are limited in their applicability to other experiments, even with minor modifications such as altering a single word involved in training. This limitation underscores the lack of scalability in the previous model. In contrast, the proposed model necessitates comprehensive training across all vocabulary in the dataset, which indeed requires a longer initial training period, but this process is only performed once; the resulting embeddings can then be directly applied across various applications. 3. The research involved two primary databases. The first is the One Billion Words corpus, which was utilized to assess pairwise similarity (as described in Section 4.1) based on four human-annotated similarity evaluation datasets. The second database is the IMDB dataset, focused on sentiment analysis (Section 4.2), and was applied in data augmentation tasks. In the updated version, an additional comparison has been included in the first table, and the second table has been updated with higher-quality embeddings derived from the IMDB dataset. ### **Questions** In preparing the input \( X \) at each phase of the model, data is converted into sparse matrices, which streamlines the processing and execution of logical operations central to the TM framework. During this conversion, word frequency within each document was disregarded at both phases. This approach implies that the model does not factor in word occurrence frequency; instead, it leverages the principles of Tsetlin Automata to select features (words) that are representative of the trained content. Although this aspect may have been implicitly addressed within the foundational TM structure, the significance of disregarding word frequency was unfortunately not explicitly stated in either the previous or current methodologies. We appreciate the reviewer’s insight on this point and confirm that this clarification will be explicitly added to **Section 3.1**: > “It is important to note that the construction of \( X \) is independent of the frequency with which a word appears in a given document; its mere presence, even if mentioned only once, is sufficient.” Regarding the embedding’s structural depth and semantic richness, we consider the generated embeddings to be sufficiently deep and descriptive for comparison with prior methods, aligning closely with the examples provided in the previous approach. However, we acknowledge that the embedding may not capture a densely interconnected representation of word relationships, a limitation which has been noted in the appendix as a direction for future research.

We sincerely thank the reviewers for their efforts in helping us improve the quality of our research. The points raised also address several key weaknesses, and we provide our responses as follows: ### **Weaknesses** 1. Referring to the lack of analysis of the computational complexity (time and space) of the proposed method, we acknowledge some challenges encountered with our approach: - The source code includes several fundamental sequential operations, which hinder the possibility of parallel execution to leverage faster processing. Despite utilizing six GPU cores, the performance gain was minimal. We recognize this limitation and aim to address it in future work. - In the proposed model, hypervector concepts were employed in a basic manner. During the course of our work, we discovered that a research team had developed more advanced techniques to enhance the performance and storage efficiency of Tsetlin Machines using hypervectors. Incorporating such techniques in future iterations of our model could lead to substantial improvements (https://arxiv.org/abs/2406.02648). - Profiling operations revealed that a significant portion of the execution time is consumed by I/O logic used to store files in a pickled manner. By adopting more efficient data storage methods and reducing I/O operations, we anticipate performance improvements. This enhancement is also part of our future development plans. 2. During our preparation for this response, we attempted to include an additional application: document clustering. However, upon initiating this task and observing the preliminary results, we realized that completing this work before the deadline was unfeasible due to time constraints. Nonetheless, we appreciate the reviewer’s suggestion and acknowledge its importance. 3. Concerning the weakness related to the hyper-parameters of the TM Classifier, we confirm that this information is included in **Part 4.2, paragraph 3**. However, the hyper-parameters related to MLP were not discussed, as the settings were default across all experiments (with `random_state=42`). This can be verified in the accompanying source code. 4. Please refer to the response provided for **Question 1** below. 5. We have re-conducted the experiments, increasing the vocabulary size from 10k to 20k. Previously, the smaller vocabulary resulted in weak embeddings and an unfair comparison. With the expanded vocabulary, there was a significant improvement in the results. 6. Please see the responses to the questions points below. ### **Questions** 1. Since the hypothesis in our research revolves around the previous method of building embeddings using TM principles, the differences between the two approaches were discussed in depth throughout the manuscript. Below, we provide a concise summary of these distinctions: - **Introduction (Third paragraph onward):** We conducted an experiment focused on the primary research question: whether the long training time in the previous method can be mitigated to allow reusability. Based on this, we outlined three contributions of the research: 1. Solving the problem of scalability and enabling the reuse of training results. 2. Constructing embeddings for input word sequences to facilitate comparison with the previous method. 3. Highlighting one potential application of our proposed method. - **Section 3.2 (Final paragraph):** We identified another issue in the previous approach: the fusion of information and loss of TM's transparency, which is addressed in our method. - **Section 3.3:** The advantages of transitioning the old method into the first phase of the new model were highlighted in three points. An example using the word “car” demonstrated our method’s ability to retain interpretability after embedding, thus preserving the transparency of TM compared to the previous approach. - **Section 4.1 (Second paragraph):** While discussing the results, we provided a rationale for why the previous method outperformed the new method in certain similarity measures. 2. Please refer to the response provided under **Weakness 3** above. 3. Examples have been added in **Section 4.2** to clarify the embedding results from the two phases. 4. Additional points on expected future work have been included in the appendix. 5. Please refer to the response provided for **Weakness 1** above. 6. The source code will be publicly available online. However, due to conference anonymity requirements, it was not explicitly mentioned in the manuscript. To address this, we provide the full source code in the Supplementary Material for your review.

I would like to thank the authors for their response. Among the suggestions I provided, only 2 items are replied with updated discussions in the paper and comments, including comparison between TM-AE and two-phase TM-AE, and future research directions. While other items are not updated with substantial changes, especially, the following are still missing: 1. Computation complexity of two-phase TM-AE 2. Additional evaluation tasks beyond semantic similarity, such as analogy, categorization, and outlier detection 3. Although the authors mentioned they have updated Table 1 after increasing the vocabulary size from 10k to 20k, I did not numbers within Table 1 being updated 4. Experiment results of TM-AE within Table 2 5. Documentation of hyper-parameter tuning of MLP within section 4.2 The above updates provided by the authors are not sufficient for me to raise my rating from 3 to 5.