Counterfactual generation lies at the core of various machine learning tasks, including image translation and controllable text generation. This generation process usually requires the identification of the disentangled latent representations, such as content and style, that underlie the observed data. However, it becomes more challenging when faced with a scarcity of paired data and labeling information. Existing disentangled methods crucially rely on oversimplified assumptions, such as assuming independent content and style variables, to identify the latent variables, even though such assumptions may not hold for complex data distributions. For instance, food reviews tend to involve words like tasty, whereas movie reviews commonly contain words such as thrilling for the same positive sentiment. This problem is exacerbated when data are sampled from multiple domains since the dependence between content and style may vary significantly over domains. In this work, we tackle the domain-varying dependence between the content and the style variables inherent in the counterfactual generation task. We provide identification guarantees for such latent-variable models by leveraging the relative sparsity of the influences from different latent variables. Our theoretical insights enable the development of a doMain AdapTive counTerfactual gEneration model, called (MATTE). Our theoretically grounded framework achieves state-of-the-art performance in unsupervised style transfer tasks, where neither paired data nor style labels are utilized, across four large-scale datasets. Code is available at https://github.com/hanqi-qi/Matte.git
Paper
Similar papers
Peer review
Summary
This paper proposes a framework for unsupervised style transfer by disentangling the content and style representations. Unlike previous research, they do not assume independence between the content and style variables in the generation process but rather only a lower influence of the style variable compared to the content variable on the generation process. Based on this assumption, the paper then introduces two theories with corresponding proofs of the identifiability of both the style and content variables under these assumptions. That is, the content information can be preserved without needing style. Given the theoretical discussion, the paper then proposes a framework based on variational autoencoders (VAE) to address the task of style transfer through style and content disentanglement. The framework is then evaluated empirically on the task by comparing it to other state-of-the-art baselines. In the results, the paper demonstrates the gain from their implementation in most measures considered. The ablation study further illustrates the importance of every component added to their model.
Strengths
- This paper contributes to the style transfer task by eliminating the independence assumption between style and content. - It supports its claims by providing both theoretical backing and empirical evidence. - It is clear in most of its parts.
Weaknesses
I do not see any major weaknesses in this paper. However, the authors can address some clarity issues in the questions section.
Questions
- The submission title differs from the title of the paper. Is that on purpose? - Line 153-156: If I understood correctly, then the described subspace should consist of all vectors whose entries are *not zero* for all indices in S. but in the paper says *zero for all indices in S* - In Section 5.3, the paper mentions that the s_(transfer) is computed from the average of randomly sampled style values of the *desired style*. What is the desired style here? - The paper considers the G-score the most important measure because of its comprehensive assessment. It would be nice if the paper clarified what it means to have a comprehensive assessment. - In section 6.1, It would be nice if the paper clarified the implemented model and the baselines. I learned later in the results section that the model is built on top of CPAVE - In Table-2 what are src1,2, and 3
Rating
8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
4 excellent
Presentation
3 good
Contribution
3 good
Limitations
- The central limiting aspect of this paper is the assumption that style has a lower influence on the generated text compared to the content, which doesn't hold for all text generation tasks. However, this limitation is acknowledged by the authors.
Summary
In the paper, the authors take varying dependence between content and style into account in the counterfactual generation process. The identification problem is addressed when using a VAE for the task. It is proved that the subspaces of latent variables for contents and style are identifiable. Then based on the theorems and assumptions the authors propose to build a VAE based model with sparsity regularization to solve the problem. The proposed framework is tested on four datasets and gets relatively high scores compared with other unsupervised baselines. Ablation study and case study are also conducted to consolidate the conclusion.
Strengths
1. Detailed mathematical proof is done for the conclusion. 2. The model performs well compared with other unsupervised baselines. 3. The proposed framework may be applied in various models in the future.
Weaknesses
1. Human evaluation might be necessary to prove the framework actually reaches the original expectation. The automatic metrics may not be adequate. 2. The MATTE does not perform well enough in the experiments. It performs poorer in accuracy and perlexity than beta-VAE. The higher scores based on word-overlap can mean little in such situation. 3. Performance of VAE-based models always varies according to random factors, and it’s better to note the mean and std. of the results in multiple trials.
Questions
Thanks for authors' detailed rebuttal, which addressed my concerns. I'll keep my score.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
2 fair
Contribution
3 good
Limitations
Limitation discussed in the paper.
Summary
This paper wants to do a controllable text style transfer by tackling the dependence between the content and the style variables. They adopt the concept of influence sparsity requiring the influence of the style variable to be sparser than the content variable. They evaluate their method on several NLP datasets to show the style transferred text generation.
Strengths
The paper aims to do controllable text style transfer which is an interesting application in NLP domain. In this paper the authors relax the independence between the style and context fills the gap in the literature.
Weaknesses
The main idea of this paper is to disentangle the content from the style. However throughout the whole paper, I don’t think the authors define clearly what is a “style”. In the literature, people usually define the sentence structure (e.g, dependency parsing tree) as the “style” and the semantics as the “content”. While in this paper, in Introduction (line 37), the author mentioned a positive sentiment is “style”, but later in Sec.3 (Line 113), they refer to the positive descriptions of something as “content”. The experiment only shows the style transfer on sentiment perspective and lacks comparison to many advanced baselines.
Questions
- The author mentioned their method could leverage the data from multiple domain. How is this reflected in your results? - In your experiment, the goal is to do a “style” transfer which means transferring the sentiment. However the accuracy (which means the intended attributes are expressed, Line 299) is very low, how can you guarantee your style transfer is successful? Also what is the label distribution for each data? What is the accuracy of a random guess in each case? - From Table 2, the transferred texts seem only change several tokens. So how does your method compare to a naive antonym replacement? - There are other unsupervised text-style transfer learning algorithms that are more up-to-date, please check: https://github.com/fuzhenxin/Style-Transfer-in-Text
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
2 fair
Presentation
2 fair
Contribution
2 fair
Limitations
N/A
Summary
The work "Controllable Text Generation with Identifiability Guarantee" presents a model for unsupervised counterfactual generation based on a model with disentangled style and content. The model is based on a variational autoencoder augmented with two flow-based models operating in the latent space. The flow-based models perform the disentanglement and, since they are invertible, allow to intervene on the style variable for counterfactual generation.
Strengths
Overall, I find the basic idea and the model itself interesting and novel. I like how the paper presents theoretical results that are then illustrated by the model rather than just an experimental evaluation. However, there are several reservations that weigh the scales towards rejection for me.
Weaknesses
First, the experimental evaluation is underwhelming, for several reasons: (i) the only setting provided is sentiment transfer, which significantly restricts the claims made for counterfactual generation in general; in fact, I would argue that the introduction and abstract are much more general than the actual results, and would advise to rewrite the introduction to mention that the model is only proven to work for sentiment transfer; (ii) I'm not sure that the BLEU metric and consequently the G-score make a lot of sense here since BLEU simply shows how much of the original wording is preserved; e.g., replacing a word with a synonym reduces BLEU but, all else being equal, arguably makes for better counterfactual generation since it makes generated sentences more varied and hence useful, e.g., as synthetic data; (iii) moreover, I'm not sure I understood the accuracy metric as presented: e.g., the IMDB dataset only has positive and negative sentiments, the authors claim that their classifier has 95% accuracy on original validation sets (this makes sense), but then Table 1 shows IMDB accuracies for counterfactuals ranging from 14% to 38% -- so that's much worse than chance for all methods including the supervised upper bound?.. it may be a misunderstanding on my part, but the paper does not clarify this at all; (iv) the qualitative results in Table 2 are also unclear: e.g., the authors claim that Optimus alters the semantics but doesn't MATTE also do it in Src 2? plus, I couldn't understand Src 4 at all, either the original or transferred versions, they don't make any sense. Second, I'm afraid that these days a text generation model has to compare itself with modern large language models, while the authors choose a GPT-2-based model from 2019 as their best baseline (and a baseline that serves as an unachievable upper bound since it's supervised). I wonder how well GPT-3.5 would do if you just asked it nicely to "please invert the sentiment while preserving content as much as possible in the following sentence that originates from the domain of movie reviews", maybe with a couple of generic examples in the prompt? Third, related work would benefit from a section on topic modeling that has developed joint sentiment-topic models such as ASUM or USTM that are quite similar in their basic assumptions to the presented model. Topic models obviously cannot serve for counterfactual generation but it looks like they were an influence.
Questions
Please see previous section
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
2 fair
Presentation
3 good
Contribution
2 fair
Limitations
The approach proposed in the paper has some technical limitations leading to a quality decrease. However, the proven identifiability guarantees lead to the mitigation of some of the potential societal risks. E.g. if the algorithm is used to change the style of some text, its semantics should be kept intact, to avoid the occasional creation of misinformation messages.
Summary
This paper discusses controllable text generation and tackles the dependence between content and style in the counterfactual generation task. Identification guarantees are proven and used to enhance disentangling of the variables.
Strengths
Style and content disentanglement, especially for scenarios with sparse data, is very challenging and important to language generation. A theoretical discussion on sparsity constraints is interesting. The following practice is motivated by the theory and the experimental results are quite positive.
Weaknesses
+ The writing requires improvement: The definition of some key concepts of this paper are missing, such as 'identifiability guarantee' and 'relative sparsity'. The connection between the generative model (described in Sec 3) to previous works is not discussed. Comparison to an existing counterfactual generation framework is appreciated. The paper confounds 'sentiment' and 'style', which gets worse when it is heavily used as the running example throughout the paper. Sentiment is more of the semantic aspect of the text. Many methods and technical choices are only expressed in math without enough explanation and motivation. + Some technical questions: u is used in Sec 3 but missed in Sec 4. How is it considered? What is the definition of T(z)? Assumption 1.i requires g to be invertible, but sparse matrices are usually not invertible, as they are not full rank. The connection between the proposed theory and practice is loose. One issue is the assumptions are strict, while they are used as losses (which means the assumptions may not hold during training). Considering there are many missing definitions and explanations, I cannot judge the correctness of the theoretical part. The latest best baseline methods [28,52] are published in 2020, you may consider later work such as [A,B] as baselines. There is no human study and style classification only relies on one model. There could be some biases in the evaluation.
Questions
+ Presentation Issues What is 'relative sparsity'? L21: 'state-of-art' -> 'state-of-the-art' 'grey shade' -> 'grey-shaded nodes' Not sure about the difference between ';' and ',' in Eqn 2. Some explanation of the assumptions using natural language would assist readers understand the work. + Missing reference [A] A causal lens for controllable text generation (NeurIPS 2021) [B] Variational Autoencoder with Disentanglement Priors for Low-Resource Task-Specific Natural Language Generation (EMNLP 2022)
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
2 fair
Presentation
1 poor
Contribution
3 good
Limitations
The evaluation is limited to 1) only automatic methods without human inspection and 2) using a singular model architecture in experiments. All these may lead to biased discussion.
Thanks a lot for clarifying my questions. I hope that these answers get integrated into the final version of the paper.
We will improve our draft according to your suggestions as indicated above—many thanks for the time and effort dedicated to our work!
Thanks a lot for your clarification with details. I wonder if you could update the paper properly within the limitation of the pages, since you give so many updates for the reviewers.
Thank you for the thoughtful question. We have incorporated the indicated updates as follows to ensure that the paper is informative and meets the page limit. - We included the human evaluation results and results from the tense-transfer task as two separate tables in Section 6.1 and Section 6.3 (a new subsection). - We included a discussion on the degradation issue of $\beta$-VAE in Section 6.1 (sentiment transfer performance), and we refer the readers to Appendix A.4 for the Diversity-2 measurement table. - All results now feature both mean and std. - We included remarks on comparing our principle-based generation approach with the anatomy-replacement method and ChatGPT at the closing of Section 5, and we refer the readers to Appendix A.5 for experimental results. We included the CTC score–BLEU alternative evaluation and multiple style classifier evaluation results in Appendix A.3 and allude to them in Section 6 (Evaluation metric). - We included suggested references in Section 2 (related work). - For short text edits (e.g., typos, minor remarks, more details, and footnotes), we made adjustments to the corresponding paraphrases. To abide by the page limit, we shortened and merged the two “contrast with prior work” paragraphs in Section 4.1 and Section 4.2 and placed the abridged version by the end of Section 4. We shortened the baseline description in Section 6.1 and deferred the detailed version to the Appendix. We condensed texts in Sections 1 (introduction) and 7 (conclusion). As uploading drafts is not permitted at this stage, we share a few revised paragraphs below. The discussion on the comparison with anatomy-replacement methods and ChatGPT (now in Section 5): >As well acknowledged, large language models (LLMs) have demonstrated an impressive ability to generate fluent text. That said, we view the principles for counterfactual generation as complementary to the development of LLMs, and we hope that our theoretical insights can further enhance LLMs. We supply examples in Table 11 (Appendix) that LLMs falter on sentiment transfer for overlooking overall and implicit sentiments, although they can precisely replace sentiment-related words. Thus, one may anticipate that LLMs would benefit from principles for representation learning, as developed in our work. Contrast with prior work (previously Section 4.1 and 4.2, now merged and placed in Section 4.2): >Zheng et al. enforce absolute sparsity constraints on latent component influences. In comparison, Theorem 1 requires relative sparsity between two subspaces, which could be better suited for language-related applications. Unlike Kong et al., which assumes subspace independence, our method acknowledges the interdependence between the two subspaces, a common scenario in language contexts. Assumption 2 enforces separate influences for content and style to facilitate style identification. Conversely, Kong et al. hinge their proof on content-style independence, limiting its applicability here. We refer the readers to Appendix A.8 for a detailed comparison. The degradation of $\beta$-VAE issue is added in Section 6, sentiment transfer performance: >Among LSTM baselines, $\beta$-VAE shows high sentiment transfer accuracy and fluency but poor content preservation. We have observed that many generated sentences follow simple but repetitive patterns, e.g., 2.2% transferred sentences in Yelp containing the phrase “I highly recommend” while only 0.6% original sentences do. These sentences are fluent and express the desirable sentiment but significantly differ from the original sentences, indicating generation degradation.~\footnote{The metric diversity-n (Li et al.) can also indicate repetitive generation pattern as it measures the ratio of distinct n-grams in all the n-grams in the generated sentences. We add the complete evaluation results in Table 9 (Appendix).} Thank you for your constructive comments, and please let us know if you have any suggestions – many thanks!
Thank you for your clarifications! After reading the responses, I feel this paper could be improved in the next round of modification, regarding (1) clarifying a series of fundamental concepts; (2) better explanations of the methodology and its assumptions; and (3) better organization and presentation of the paper.
Thank you for feedback and the effort for reviewing our work. As indicated in our previous response, we have explicitly incorporated your feedback into our manuscript, including explicitly defined concepts like identifiability, a discussion on the references you pointed us to, and suggestions on evaluation. As uploading drafts is not permitted at this stage, we share a few revised paragraphs below: We explicitly defined identifiability at the opening of Section 4: >The notion of identifiability describes the possibility of learning the true statistical model (up to certain equivalent classes) from its samples [4]. The identifiability of a variable $z$ indicates that the estimated variable $\hat{z}$ contains all the information of $z$ without mixing the information of other latent variables. Formally, there exists an *invertible* mapping $ h $, s.t. $ \hat{z} = h ( z ) $. This notion of identifiability is widely adopted in prior work [24,46,50]. Suggested references [A] and [B] have been added to Section 2 (related work), shown below: > A line of work adopts pretrained language models as the encoder and the decoder in their model. For instance, one of the most popular pretrained VAE models, Optimus [30], employs BERT as its encoder, and GPT2 as its decoder. To train a latent connector between BERT and GPT2, it is pretrained on the wikitext dataset via an unsupervised reconstruction objective. On top of Optimus, the model in [A] introduces a pretrained classifier conditioned on the style labels, as well as two counterfactual objectives. [B] focus on transferring learning after pretraining on the style-annotated training dataset. Our model also leverages the latent variables to model the data-generating process and conducts style transfer in the latent space. However, unlike [A,B], we do not need style labels in the training phase. The human evaluation schema and results have been added in two parts of Section 6 (Experiments), i.e., evaluation metric and sentiment transfer performance: >Evaluation metric: We conduct both automatic and human evaluation. For human evaluation, we invited three English-fluent evaluators to rate the sentiment reverse, semantic preservation, fluency and overall transfer quality using a 5-point likert scale (higher scores indicating better performance) and rank the generated sentences from different models (tied items are permitted in rank). >Sentiment transfer performance: Based on the automatic evaluation results, we randomly selected 100 examples from the four datasets and gathered the generated sentences from the top-performing baselines in each group, namely CPVAE, Optimus, and MATTE. The results in Table 2 (table is updated for human evaluation results) show that human annotators favour Optimus in terms of content preservation and fluency, after considering the style transfer correctness, Matte ranks the best-performing method with more than 58% support set. We were wondering whether your technical concerns had been properly addressed by our responses so far (If yes, could you please adjust the rating accordingly?). Please let us know if you have further concrete questions or concerns that we can address. Thank you for your engagement with our work.
Once again, we are grateful for your time and efforts. Since the discussion period will end in one hour, we are very eager to get your feedback on our response. We understand that you are very busy, but we would highly appreciate it if you could take into account our response when updating the rating and having a discussion with AC and other reviewers. Thanks for your time, Authors of # 1309
Thanks for the rebuttal. However, after reading other reviews as well, I feel there are quite some efforts needed for the paper to improve and solve all the reviewers' concern. I will keep my original score.
Thanks for your follow-up insights! As indicated in our responses to other reviewers, we have responded to all the explicit concerns raised by all the reviewers and updated our manuscript accordingly. As uploading drafts is not permitted at this stage, we share some revised paragraph examples below: To highlight the **effects of domain $u$ in our results**, we modified the paragraph **sentiment transfer performance** in **Section 6.1**. >Our model is built on top of CPVAE with the proposed causal influence modules and sparsity regularisations. Specially, we adopt $u$ to establish the domain-varied dependency between content and style, illustrated in Eqn(2), while all the baselines in the LSTM group first derive a domain embedding and then concatenate it to the sentence-level representation to differentiate their domain sources. We can see clear improvements over the best-performing baseline CPVAE, which can be partly explained by our novel method of incorporating domain information. Moreover, the first example "*The guy is very flavour*" in Table 2 shows that the simple concatenation of domain index in CPVAE fails to capture the domain-adaptive content-style dependency and leads to an unnatural content-style match. We included remarks on comparing our principle-based generation approach with the **anatomy-replacement method and ChatGPT at the closing of Section 5**, and we refer the readers to **Appendix A.5, Table 11 and Table 12** for experimental results. > **Comparison with large language model**. As well acknowledged, large language models (LLMs) have demonstrated an impressive ability to generate fluent text. That said, we view the principles for counterfactual generation as complementary to the development of LLMs, and we hope that our theoretical insights can further enhance LLMs. We supply examples in Appendix, Table 11 that LLMs falter on sentiment transfer for overlooking overall and implicit sentiments, although it can precisely replace sentiment-related words. Thus, one may anticipate that LLMs would benefit from principles for representation learning, as developed in our work~\footnote{We also provide the results of anatomy replacement in Appendix, Table 12, to further compare the interventions that occur in input-space and latent-space.}. We included the results of the tense transfer task in **Section 6.3 (a new subsection)**, with newly added **Table 5** and **Figure 6** as illustrations. >To further verify our theoretical insights, we apply MATTE to tense transfer between past and present, in which tense is the style variable with relatively sparse influence to sentence. We reuse the model trained on the above four datasets and do inference on the Yelp dataset. Specially, we follow Shen et al. to identify the tense of sentences by extracting the main verb using the StanfordNLP Parser. In order to transfer the tense from past to present, we collect 100 present sentences in the dev set to derive $s_\text{transfer}$ as a replacement of the original $s$ for the past sentences, and vice versa. >The tense transfer accuracy results on Yelp test tet (Table 5) show significant improvements, 6% from MATTE over the best-performing baseline. We compare the learnt style variable $s$ derived from CPVAE (left) and MATTE (right) in Fig 6. There is a clear bond between past and present sentences in MATTE, while some past (blue dots) are mixed in the bottom of the red district in CPVAE, which implies that MATTE learns a better disentangled tense representation. We hope the updated text could address your concerns. Please kindly let us know if you have further concrete questions or concerns. Thank you for engaging in our work!
Dear reviewer zc3n, Once again, we are grateful for your time and efforts. Since the discussion period will end in one hour, we will be online waiting for your feedback on the further response we provided yesterday. We would highly appreciate it if you could take into account our response when updating the rating and having discussions with AC and other reviewers. Thanks for your contribution to NeurIPS 2023! Authors of #1309
Dear reviewer pDz1, Once again, we are grateful for your time and efforts. We have been eagerly waiting for your feedback on our point-to-point response. We will be here waiting and hope to see it before the discussion period ends. We understand that you are very busy, but would highly appreciate it if you could take into account our response when updating the rating and having discussions with AC and other reviewers. Thanks for your time, Authors of #1309
Thanks authors for carefully addressing my comments. New experiments provided in the rebuttal dismiss most of my concerns. I raise the score to 6.
Thank you so much for providing valuable feedback and acknowledging our work! We will incorporate them carefully into the future version -- many thanks for your time and effort!
Decision
Accept (poster)