The Power of LLM-Generated Synthetic Data for Stance Detection in Online Political Discussions

Dear reviewer, We really appreciate your thorough and constructive feedback. We have taken your suggestions to heart and made changes to the best of our abilities. We first want to address your suggestions for improvement and then address general comments about the paper. > Add analysis of entropy/diversity metrics for synthetic data > What's missing here is any real investigation into what makes synthetic data actually useful... * We have now added a more thorough analysis of the synthetic data and what makes it useful: * To measure the diversity, we calculate the entropy of each comment (entropy over words) for both the real and synthetic data, and compare the corresponding distributions. * To gather insights about the diversity in the data, we compare average entropy and average word length for the interquartile ranges of the entropy distribution. The entropy reveals information about the content of the data, while the length acts as surrogate for the structure. * We have gathered our insights in Section 5.2 and Figure 5: * The results show, that the entropies between the real and synthetic data are similar. We also observe, that the average comment lengths of the synthetic data are much longer than the real data counterparts. Looking at the samples in Appendix C1, it becomes clear that the real comments have a concise writing style, common in online political discussions. The synthetic data comments are verbose, while still addressing the topic in a nuanced way. * Looking back at Figure 3, it becomes apparent that the BERT model seems to be invariant to the comment size, since the synthetic and real data align well despite the difference in comment length. The similar entropies also hints towards a similar amount of diversity in both datasets. * This also aligns with our second experiment which was already in Section 5.2, which we have now renamed as "content vs structure". We showed that fine-tuning with synthetic data that does not align with the topic does work well, which we see as validation for the topic of the synthetic data being important to achieve improvement. * Thus, one clear insight from this is that at least for the BERT model, we need synthetic data that is aligned with given question(topic). * Nonetheless, it would be interesting to explore in future work whether comments that reflect similar structure as the real comments improve performance even further. > Provide theoretical justification for why synthetic data helps beyond just "more data" * Taking the above into account, we additionally believe the synthetic data helps for a few reasons. We have rewritten the text in the results section to better reflect this: * The synthetic data is generally of high quality: from 10000 generated comments in total, only 2%-3% (261) were unusable, while the high quality comments contained nuanced opinions. * We thus can assume and also observe from Figure 3 that the synthetic data effectively interpolates between the real data, smoothening the decision boundary of the BERT model. * This explains why augmenting the synthetic dataset with the most informative samples works so well for increasing synthetic dataset size: the synthetic data reduces overall variance when learning the decision boundary with the most informative samples, helping the model to remain robust while incorporating the new real data. > Explore what makes certain synthetic samples more useful than others > Consider alternative methods for introducing genuine novelty into synthetic data * We believe to have answered this question above, however we also want to restate our thoughts from the introduction. We argue that the synthetic samples are useful on a distributional level: * They serve as prior to the model for the "favor" and "against" classes, since the BERT model is sensitive to content (topic) of the data. * They are useful for detecting samples on the decision boundary of the model such as the most informative samples. This is what inspired SQBC. * While analysing the most informative samples we noticed no clear discernible patterns in the data other than the embeddings being in between the two classes, thus we think a more interesting approach consists in looking at the data from a distributional or statistical learning perspective. * In this sense, we could leverage the structured embedding space provided by the synthetic data to design methods that generate samples outside of this reference distribution. This could be achieved through a latent space approach such as using the synthetic data as latent prior or through an adversarial approach where the generative model attempts to learn representations that are as far as possible from the reference distribution. * We believe that synthetic data can open up an avenue to designing such methods and have added these ideas to the discussion part (Section 6) of the paper.

> Compare computational costs vs. benefits more explicitly * Thank you for this suggestion! * We have added a paragraph to the discussion section "Potential impact" stating the advantages and contributions of our method to practitioners in the field of stance detection and online political discussions. * In short, good stance detection inference with BERT models is accessible to smaller organisations. The synthetic data generation can be outsourced to specialised infrastructures. * The ability to generate useful synthetic data allows to tailor the model to a specific topic, where no data is available and where data collection or labelling efforts are too costly. > The most interesting part is actually buried in Section 5.1, where they show that using Mistral-7B directly for stance detection fails miserably... * This is indeed a very interesting result. We have rewritten the section to better detail our thoughts regarding as to why the Mistral-7B model struggles at stance classification with the X-stance dataset: 1. The niche topics in the X-Stance dataset are likely not present verbatim in the training data of the Mistral-7B model. While this might also be the case for the SemEval or P-Stance datasets, political tweets about US politicians are more likely to be found online in a similar form. 2. The smaller parameter count of the model may affect capturing the overall context when providing it with both a discussion topic and a comment in its context. In contrast, when generating synthetic data, the model only needs to focus on the discussion topic in its context. 3. LLMs are usually not fine-tuned to give single word responses. In fact, to get the results in Table 2, we accepted responses that contained the words "favor" or "against". If we specifically matched for "favor" or "against" exclusively, only a handful of comments would fullfil this criteria. * Overall, while this result is interesting, it is a bit out of the scope of the paper, since our focus is on shifting stance prediction onto the simpler BERT model. > Yet another synthetic data paper that shows modest improvements but doesn't quite nail why or how to make synthetic data actually useful. Some interesting ideas buried under conventional methodology. * We are glad you found our ideas interesting albeit somewhat buried. Please allow us to restate the motivation behind this paper and why the synthetic data and SQBC approach are connected in the context of online political discussions. * We based our work on two main issues in stance detection for online political discussions. 1) Available datasets are usually small or non-existent given how complex and niche online discussion topics can be. 2) In the available datasets there are usually several samples that are hard to classify either because the comments are inherently ambiguous or are simply poorly written. This makes training a good model difficult. > The active learning component feels tacked on, though I'll admit the SQBC approach is clever. Using synthetic data as a reference distribution for selecting informative samples is neat, but again - why does this work? The paper handwaves at "ambiguous samples" without diving deeper into the theoretical foundations * Thank you for the suggestion! We hope to now have better highlighted the connection between the approaches in the paper. * Regarding the theoretical foundations, we have attempted to provide a bit more theoretical and technical insights in the results and ablations sections and also in the responses above. * In case this is not sufficient, we'd be happy to address any additional concerns in this regard. > Were there any computational/economic reasons for not scaling up your compute? I'm sympathetic to this as I understand the burden of even a single A100 - but if you do have more resources, why not use them? * Yes, unfortunately this was the only compute we had available for the project.

Dear reviewer, we would like to thank you for your thorough review and interesting thoughts. In the following, we attempt to answer your questions and comments: > It would be very helpful to explain why only a German dataset is used for the experiments. Also, if German text is used, have the authors considered using a different LLM that has good German language processing capabilities for the experiments? * You are correct! We will add our reasons for using the German X-Stance in the datasets section. * In short, we used the German X-Stance dataset because it is the most comprehensive dataset for stance detection in online political discussions. It contains over 48k comments divided into 140 different topics (also called targets). We felt this variety in topics was important to validate our method. * Moreover, the individual dataset sizes (per topic) are quite small (between 50-500 samples) which reflects the usual limited data availablility in these kinds of discussion processes. * Known English stance detection datasets such as SemEval 2016 or P-Stance have far less varied data. SemEval has only six topics(targets) while P-Stance has three topics(targets). These topics are also not as specific as in the X-stance dataset, since the SemEval and P-Stance datasets mostly contain tweets regarding US politicians and mainstream topics. Therefore, it is more likely the LLM has seen SemEval or P-Stance data in its training set. This is one of the reasons we think the Mistral-7B model performs so poorly on classification on the X-Stance dataset, as the comments there are far less likely to be encountered online. * Overall, since we observed during our test that the translation models were able to translate the comments with virtually no loss in quality, we decided to use the German X-Stance dataset. * We are also confident that our approaches would work with other languages if the pipeline is adjusted accordingly. > The novelty and impact of the work is still limited... * We have added information and insights about the synthetic data in Section 5.2 to further highlight the relevance and importance of our approaches: * Regarding SQBC, we believe the insight that the synthetic data distribution can act as a reference distribution is novel and can be useful for future work. As stated in the results section, it is particularly interesting that random selection works better than both SQBC and CAL when the synthetic data size is large (M=1000). Our intuition behind this is that with increasing synthetic dataset size, the smoother decision boundary makes the model less sensitive to outliers (which are introduced with the random selection method, but not with SQBC as seen in Figure 4). This robustness induced by the synthetic data allows the model to improve even when selecting random samples. We believe this an interesting insight and result. * It is also important to note that the stance data is relatively homogenous due to its discussion structure as we now show in Section 5.2. This means that if the data were to contain any severe outliers, the random selection method could select these samples and worsen the performance of the model. This would not happen with SQBC due to the k-nearest-neighbors objective. * We believe that we provide insights for future work with regards to when and how synthetic data is effective in improving stance detection. Apart from Figure 3 and the discussion around it, we have added further technical and theoretical insights in the results section. We have also added a new ablation where we inspect the entropy distribution of the data together with the corresponding average lengths of the comments in Section 5.2 (see Figure 5). We sum the insights up briefly: * The synthetic data is generally of high quality: from 10000 generated comments in total, only 2%-3% (261) were unusable, while the high quality comments contained nuanced opinions. * We observe that despite the synthetic comments being longer than the real comments, the entropy is very similar. Also as seen in Figure 3, the synthetic and real data align well, which indicates that the BERT model is invariant to comment length which can be seen as a surrogate for comment structure. * Together with the "Content vs Structure" ablation, this validates the observation that a BERT based stance detection model is more sensitive to the actual content of the data than the structure. * We have also added a section regarding the potential impact of our work for the area of stance detection in real world settings: * In short, good stance detection inference with BERT models is accessible to smaller organisations. The synthetic data generation can be outsourced to specialised infrastructures. * The ability to generate useful synthetic data allows to tailor the model to a specific topic, where no data is available and where data collection or labelling efforts are too costly.

> There is room for improvement in terms of presentation. In particular, the active learning method proposed can benefit from first presenting an overview of the high-level intuition behind the method before describing the method itself. * Thank you for this suggestion! * We have added to the beginning of the method section to better connect the concrete method to the high level intuition from the introduction section and to Figure 1.

Dear reviewer, we thank you for putting in the time to provide us with a thorough review. We are eager to respond to your questions and comments: > It’s possible I’m missing some key context here, but I’m having trouble following the ablation study in Section 5.2.. > In the ablation study for "Content vs. Size", I am not sure I understand why you call the shuffled dataset "misaligned". Could you please explain the reasoning behind this. * Thank you for pointing this out! * We do not shuffle instances within the synthetic dataset. We shuffle the entire datasets with regard to the questions, i.e., we shuffle the correspondences between question and (Q) and synthetic dataset (SD). * For instance, we have 10 Questions and therefore 10 corresponding synthetic datasets. Instead of having the natural correspondence of Q1->SD1, Q2->SD2 ….. and so on, we change the correspondences, e.g., Q1->SD9, Q2->SD5, …. * Overall, we have renamed the section "content vs structure" and attempted to describe the purpose of this ablation more clearly. The core idea is to analyse whether per-topic fine-tuning is really necessary. For this, as described above, we change the correspondences of the given question and the synthetic datasets for fine-tuning. * We have also added a new ablation in the same Section which shows the entropy distribution of the data and the average length of comments. We observe that despite the synthetic comments being longer than the real comments, the entropy is very similar. Also as seen in Figure 3, the synthetic and real data align well, which indicates that the BERT model is invariant to comment length which can be seen as a surrogate for comment structure. * Together with the "Content vs Structure" ablation, this further validates the observation that a BERT based stance detection model is more sensitive to the actual content of the data than the structure. > Even though authors acknowledge this as a limitation, fine-tuning a separate model for each question does not seem to be a scalable approach, especially when the main motivation for the research was in line with training robust models for online deployment. * We understand this goes against the current paradigm of training large general models. However, there are a number of benefits to fine-tuning separate models in an online political discussion context: * They are more efficient to deploy for inference, especially for smaller organisations that do not have many resources. * Considering participation processes in government, these are usually carried out on a per topic basis. * Concerning robustness, together with synthetic data we can tailor models to specific topics while outsourcing the synthetic data generation to larger infrastructure. * The smaller stance detection model remains interpretable and more predictable compared to larger general models as we show in Section 5.1. * The new ablation in Section 5.2 further highlights the importance of topic alignment between the model and the synthetic data. We added these points for further clarity to the discussion section. > The X-stance dataset is described as having around 48k annotated comments on various questions. However, an overview of the dataset’s statistics—such as the number of comments per question—would greatly enhance readability. When you mention selecting 10 questions from the test set, it would be helpful to specify how many comments correspond to each question. While I see some statistics are included in the Appendix, a high-level summary within the main text would improve clarity and context for readers. * We have added more information regarding the amount of topics and a table in the datasets section of the exact sizes and splits of our test datasets. > Section 4.2, General setup: Please review this section for more readability. Currently, it is a bit difficult to get a picture of what models are being tested and how the methods differ between them. * Again, thank you for pointing this out! We have improved this section accordingly. > Why choose translation over adapting prompts directly? Is Mistral unable to generate responses in German, or were other multilingual models considered? * Yes, we found that smaller models such as Mistral-7B struggle at generating grammatically correct and coherent sentences in German. At the time of performing our experiments, among the smaller models we did not manage to find a good German model to generate high quality comments. * In any case, we found that there is virtually no loss in quality when using the translation model.

> In the generated dataset, did you find any instance where the LLM failed to generate the requested content? For instance, generate statements not in favor when requested for "in favor" content or LLM refusing to generate any relevant content at all. * We have added more insights regarding the synthetic data in Figure 5(Right) and Appendix E1. * Only 2-3% of all synthetic comments were unusable, in these samples the LLM would either generate a very short comment or repeat certain content for a number of times until it would catch itself to finish writing the comment. Appendix C1 shows some of these samples withe minimum and maximum entropy. * We also did not find instances in the synthetic data where a comment with the opposing stance was generated. We believe the request of generating a comment in "favor" or "against" is a strong conditioning variable for the model. * Of course, with different models mileage may vary. With the rapid advancements and the plethora of models available, we chose not to focus our work on an ablation of the different models. Instead we wanted to showcase a small model that works well and is relatively efficient, while feasible to run.