Goal Driven Discovery of Distributional Differences via Language Descriptions

Summary

This paper presents a new task called D5 (goal Driven Discovery of Differences between text Distributions via language Descriptions) ) where the objective is to identify and describe meaningful differences between two given text corpora. Authors created two datasets to evaluate their D5 task based: SYND5 and OPEND5, based on validity and relevance.

Strengths

The creation of OPEND5 and SYND5 datasets for training and evaluation shows originality.

Weaknesses

Task justification and applicability: The paper introduces a new task D5 (goal Driven Discovery of Differences between text Distributions via language Descriptions). However, the need for such a task is not clearly articulated. The authors do not provide sufficient examples or scenarios where this task could be beneficial, making it difficult for the reader to fully appreciate the significance of this work. It would be beneficial if the authors could provide concrete real-world applications or problems where this task could contribute to a solution or add value. If the main usage is as the example shown in Figure 3, the users can use GPT to generate answers themselves nowadays. Interpretation of discoveries: The paper acknowledges that the interpretation of the discovered differences is challenging and often requires domain experts. However, the study doesn't suggest concrete ways to address this problem, which could limit the practical application of the D5 system. Reliance on Turkers' judgments: The paper relies heavily on the judgments of Mechanical Turkers to evaluate the validity of discoveries. This approach might introduce another layer of bias or inaccuracies due to the potential lack of expertise among Turkers as authors mentioned in the section 6.2. Missing a conclusion section Minor suggestion: the right edge of Figure 2 is cut off which need to adjustment.

Questions

The above weakness are also my questions. Evaluation Metrics: The paper primarily reviews the task based on the metrics of validity and relevance, but also mentions evaluating subjective metrics like novelty and significance. Could the authors explain why validity and relevance were prioritized in the evaluation process? And can the authors provide a brief overview or key insights from the evaluations of novelty and significance, given that they are confined to the appendix due to space constraints? Related work: The author propose their contribution as the novel D5 task but also mentioned they mirrored Zhong et al. (2022)'s study. Could the author explain what is their novel contribution compared with Zhong et al's work?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

The authors have indeed discussed several limitations of their work which they couldn't address at this stage.

Summary

This paper proposes a goal-directed discovery and description of differences between text distributions (D5). Given two corpora and a research goal, the task is to generate a discovery in a natural language predicate. The paper creates datasets SynD5 and OpenD5, which can be used to evaluate validity, relevance, novelty, and significance.

Strengths

- The paper tackles an interesting task and formulates research discovery as an NLP task. - The paper creates new SynD5 and OpenD5 datasets from a variety of problems.

Weaknesses

- #1 The task is not well-formed or well-motivated. - #2 The dataset quality is not ensured. As stated in LL119-123, OpenD5 does not ensure that there is ground-trugh and thus is not clear hwo reliable it is as a benchmark. - #3 The paper is generally well-written but the key concept and main messages are hard to parse. First of all, it is a great effort to compile hundreds of datasets into a single benchmark but I’m afraid that I’m not very clear what we can conclude after seeing method A perform better than method B on this dataset. After reading the paper, I wasn’t able to answer a question “What functionality assessment can this test bed be used for?” by myself. Looking at examples in Figure 2, the OpenD5 data split looks synthetic and some of them rely on the ground-truth labels for the original task. (e.g., 10-star reviews and 0-star reviews.) In other words, how the original corpus was split into Corpus A and B is essential to answer the research question. Thus, if the split is given, the problem is partially solved already. In my understanding, the task essentially asks if the model can figure out how those two corpora were created from the original corpus with additional context information as the research goal. For Machine Learning problems, I believe analysis on each pair could be one research paper (e.g., the difference between SNLI and MNLI; LM’s output differences etc.) Then, again I’m not very sure what we can learn from evaluation results on the OpenD5 benchmark. To me, the problem the paper is tackling is rather automating Exploratory Data Analysis using LLMs (like the paper below. To be clear, it’s considered concurrent work and the paper doesn’t have to compare with it.) Pingchuan Ma, Rui Ding, Shuai Wang, Shi Han, Dongmei Zhang, Demonstration of InsightPilot: An LLM-Empowered Automated Data Exploration System, https://arxiv.org/abs/2304.00477

Questions

- As raised in the Weaknesses above. - Q. In 2.1 Task Format, is the goal description given for the task (and thus is included in the OpenD5 dataset)?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The paper discusses the limitations. For ethical concerns, I didn’t check the licenses of all the datasets (whether they can be redistributed after modification)

Summary

The paper introduces a new machine learning task - D5 - to model and discover differences between two large corpora by providing a task input as an exploration goal. The paper also introduces a synthetic dataset, SynD5, for evaluation of D5, as well as OPEND5, a dataset containing 675 problems that D5 can be used to train on.

Strengths

- The task tackled by the paper - generating discoveries based on corpora is impactful - The paper provides in-depth description of their methodology and datasets, as well as how they were built - The proposed framework is innovative

Weaknesses

- OPEND5, the dataset that was built specifically for D5, has been collected manually by the authors, which could introduce biases. - As the authors themselves mentioned, the extracted hypotheses can be merely correlations in the corpora and not necessary causations. However, D5 is built in such a way that it would treat them as 'discoveries'. Given the nuance around the types of things the model would extract, having clearer, more robust measures and datasets for evaluation of such a system should be in place before releasing it. Validity scores being quite low support this as well. - The paper selects 21 out of 3296 examples to showcase as discoveries, potentially showing only cherry-picked examples.

Questions

- I wasn't clear what was the purpose of the distractor (mentioned in 2.1) - I’m not sure I understand the process described in 4.3 where a discovery’s validity is meant to improve by fine-tuning a LM on discoveries with validity scores obtained from an LM. If we can’t trust the validity of the LM to begin with, why would we trust a model fine tuned on it?

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

I've mentioned a couple limitations in the paragraphs above. I believe the authors have adequately managed to address the limitations in the paper.

Summary

The authors formulate a new task, D5, that automatically discovers differences between two large corpora in a goal-driven way. The task input is a problem comprising a user-specified research goal (“comparing the side effects of drug A and drug”) and a corpus pair (two large collections of patients' self-reported reactions after taking each drug). The output is a goal-related description (discovery) of how these corpora differ (patients taking drug A “mention feelings of paranoia” more often). In addition to the dataset the authors also build a D5 system, and to quantitatively evaluate its performance, Finally, the authors discuss the limitations of the current D5 system, which discovers correlation rather than causation and has the potential to reinforce societal biases when misused.

Strengths

1. The authors propose new tasks and datasets that are interesting and intriguing, which is considered a contribution to the community. The idea of generating discoveries with controllable goal is a valid and useful task. 2. Overall the task looks well-formulated, and the dataset is covering multiple data domains. 3. The writing and presentation is of high quality. The paper is generally easy to follow and understand. 4. The qualitative analysis part is very well done and complete.

Weaknesses

1. More discussion could be done regarding the uneven distribution of data examples in different domains. For example only 10 from Health versus more than 200 in social sciences. 2. The authors designed a set of scores for evaluation metrics, and it would be better if there could be some sort of relevance study between the subjective scores with a set of human scorers. 3. Although using closed large language models are exciting because they generally provides better results, it would be nice to also add in any results with open-source models or smaller models, both for the sake of reproducibility (since OpenAI will deprecate the old APIs soon) and also to show the community where the current open models is at for your proposed tasks.

Questions

Above.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes, the authors state limitations as well as possible biases.

Summary

This work formulates a new task called D5, proposes a language model (LM) based approach to generate desired outputs in the context of this proposed task, and contributes a new dataset of open-ended problems to highlight how their LM-based approach can facilitate exploration of differences between two large English text corpora, with respect to a stated goal. The goals (for exploring differences) are stated in natural language, and the output 'discovery' of the differences is also generated in natural language. The work shows how discoveries generated by their D5 system can be evaluated on both a synthetic diagnostic dataset (where ground truth differences are known and recovery of these differences can be directly tested), and on the open-ended OpenD5 dataset. The evaluation in the latter case depends on one somewhat objective criterion of validity, and then relevance to the user-stated discovery goal is evaluated manually using subjective human assessments, where humans are either the authors themselves or crowd workers (not domain experts). This work as a whole seeks to demonstrate the potential of LMs to facilitate human or subject matter exploration of hypotheses for differences in two corpora, and these discovered hypotheses are meaningful towards a user-stated goal.

Strengths

The paper is well written and is clear in its definition of the proposed task as well as the objectives of the work. The goals carry real-world significance, as a valid and reliable method to enable exploration of goal-relevant differences between two corpora can help scientists and domain experts in generating useful hypotheses and potentially help generate insights that would have otherwise been too laborious or just not that obvious via a manual assessment (especially when the two corpora are big in size). The work takes a concrete step towards these goals of enabling human exploration and discovery. While exploratory differences between text corpora have been studied in NLP before, adding a goal, expressed in natural language, to this process of discovering differences and enabling systems to take this goal into account is a novel modification to this kind of exploration task. The OpenD5 dataset is a valuable data contribution if released and documented properly. Proposed evaluation criteria and robustness checks also serve as a key framework for designing similar tasks and making sense of the goals of the proposed system and task.

Weaknesses

- While proposed criteria and robustness checks are valid and sensible, when the goal is facilitating human exploration and providing a system and framework to end-users, it is important that people other than the authors create the prompts or fill in the non-fixed portions of the prompt template because it is not clear if even slight differences in how the ‘open-ended’ parts of the prompt text (and its framing) could affect the generated hypotheses and discoveries. It is understandable that recruiting people to also generate prompts and assess discoveries for the same goal and input corpora but with prompts written by different people is difficult, especially if a well-powered human experiment is to be conducted. However, it is critical to study the effect of how the exploration goal is phrased — which includes paraphrases. While the validity and relevance to criteria are necessary and well-motivated eval criteria, arguably another really important criterion here (since the system is intended to help human exploration) is stability (which can map to intra-coder reliability): is system output robust to tiny, non-meaningful changes in the stated exploration goal? In fact, this can be seen as an important component of system reliability – if we cannot expect consistently similar or the same discoveries for stated exploration goals or prompts that carry the same meaning and logically entail the same output, then the system is clearly not reliable for use by humans. Additionally, this robustness or stability is important to establish, given literature in NLP, for example, on adversarial robustness where minor perturbations in text (such as in text style which should be irrelevant to the task) could 'break' the NLP system, and more. See Qi et al (2021) for an example with text style transfer, Zhang et al (2020) for a relevant survey, and Wallace et al (2019) for some useful pointers on adversarial triggers in NLP. In addition to this important robustness criterion, it is also generally important to study the effects and properties of the language of the exploration goal. The following questions can be used to create a more tested system of prompt design: What must be included? What are the constraints on phrasing? Does the order of specifying things matter? Is there a guide that should be followed when constructing the exploration goal for the most 'valid' and 'relevant' results? Do different phrasings of the same underlying exploration goal affect the validity or relevance of the model output discovery? Even for the template portions of the prompt (which are not meant to be modified), it is not clear how this template was constructed and what happens when these templates are changed. Theoretically, could a different template achieve better results? Without such an analysis, both the usage of the system and the generalizability of the research work are negatively affected. - One of the stated aims of this work and the proposed system is facilitating the discovery of unknowns or the generation of novel, previously unknown insights. However, including what has been presented in section 6.1, it is not clear at all if this has actually been established even using the OpenD5 dataset. With only the authors judging the discoveries and no subject matter experts involved, such a claim cannot be evaluated or it needs to be reframed and scoped out appropriately. Novel to the authors does not seem to represent a significant advance and does not seem to represent a step towards “discovering unknowns” stated in line 35. However, this is a critical potential contribution of this work and not taking concrete steps to establish this potential negatively affects the significance of the work. Also, in lines 284-285, the authors acknowledge that the signal about validity is unreliable, and since validity is critical when it comes to hypotheses generation and exploratory differences, it is unclear if the proposed framework is ready for uptake and if it could provide potential value to practitioners as things stand. - The explicit acknowledgment and some discussion on some of the limitations are important and add to this work, however, the limitations that have been acknowledged are of great importance and represent harm. The exploratory analyses aspect and the aim of discovering the unknown make the harm of bias reinforcement even more critical, yet no path to mitigating these harms has been provided. The biases in authors’ subjective evaluations are also similarly really important here and it is not clear why at least other humans or recruited crowd workers cannot help in all the key human judgement-based evaluations, though the cost is certainly an understandable factor. Not incorporating domain experts and their knowledge in assessing discoveries is also a big limitation since the potential for this system to generate novel insights and find unknowns remains unclear. With the authors rightfully cautioning users about bias amplification and reinforcement towards the end of this paper, with no pathway to mitigate or understand when such cases arise, it is questionable whether the system should be used at all. There are also at least two other key limitations (discussed below) that also need to be addressed or at least comprehensively discussed in this work. In summary, the key weaknesses of the work as it currently stands include a) system reliability for usage by end-users is not yet fully established, with more robustness checks on prompt construction and framing needed; b) while stated as a goal, the step taken in this work does not seem to demonstrate potential for generating novel insights or discovering unknowns, and evaluation signal for the validity of the discoveries that do get generated also seems to be unreliable; and c) limitations, both the ones explicitly acknowledged and the ones that are not, represent real-world harms, and the mitigation of these harms has not been tackled and as things stand, these harms seem to outweigh any benefits of the proposed system and framework. References: Fanchao Qi, Yangyi Chen, Xurui Zhang, Mukai Li, Zhiyuan Liu, and Maosong Sun. 2021. Mind the Style of Text! Adversarial and Backdoor Attacks Based on Text Style Transfer. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4569–4580, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. Zhang, W. E., Sheng, Q. Z., Alhazmi, A., & Li, C. (2020). Adversarial attacks on deep-learning models in natural language processing: A survey. ACM Transactions on Intelligent Systems and Technology (TIST), 11(3), 1-41. Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019. Universal Adversarial Triggers for Attacking and Analyzing NLP. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2153–2162, Hong Kong, China. Association for Computational Linguistics.

Questions

1. Is it safe to say that at least for the paper as it stands, “discovering unknowns” is not something that has been evaluated or judged? Is there a robust plan to enable this system to reliably help domain experts discover true unknowns? 2. What can we expect about the robustness of the system to variations in the phrasing and framing of the stated exploration goal by different non-author users (see the first point in the weaknesses section)? Also, from an end-user as well as a research perspective, are any experiments conducted to test what must be included in the exploration goal or other non-templatic parts of the prompt, what should be constraints on phrasing, if the order in which various things (about the corpus, goal, etc.) things matters? Can we expect that different phrasings of the same underlying exploration goal will affect the validity or relevance of the model output discovery? 3. How was the prompt template designed, and do we know if a different template could yield better results? 4. As noted above in the third point in the weaknesses section, even within the scope of identified limitations, it does not seem to be easily identifiable for an end user when correlation might get framed as causation or when bias reinforcement might happen, and therefore, can the current system be used with those potent limitations and harms as a possibility? In other words, given these known limitations and harms, can we currently handle them effectively, or if not, should D5 usage be held until those limitations are overcome or at least mitigated effectively? 5. Section 6.1 is an important assessment of OpenD5, however a lot of details do not seem to be provided which hampers the review but also the clarity a reader needs. The manual selection process of the 21 discoveries is not clear: how was the ‘representativeness’ of potential use cases judged? Also, how were criteria 3 and 4 actually assessed (lines 254-255)? Apart from this section, some other key details and justification behind certain decisions are not provided which are also important, these are noted below. Suggestions (acknowledging possible space limitations for author response, questions above are separated from some questions below – addressing the below questions can improve the paper): - With respect to lines 49-50, goal-relevance being higher is expected (and good that it does happen) since one model has goal-conditioning and the other was not designed to handle stated goals; but it is interesting that validity or correctness is higher too – analysis of what invalid differences are now not generated in the proposed model or what valid differences were missed in baseline model would be interesting (unless already done, in which case a clearer pointer to this analysis should be included). - For the synthetic diagnostic test, it seems like the testing would benefit from either an additional or alternative approach to a synthetic test bed where the two corpora are not generated by language models (since systems being tested in this work are also language models, which calls into the question the strength of the proposed synthetic benchmark). Given that the properties for goal-relevant and distractor dimensions are around topic, genre, and language, a straightforward synthetic benchmark is using parallel corpora and then running actual topic models on them to generate and divide documents/texts by topic using the outputs of topic models: in this way, the corpora being compared remain real-world human-authored collections while the distraction vs goal-relevant test can still be conducted. OpenD5, while being a valuable real-world dataset contribution relevant to the task, does not test this distractor element present in SynD5. So something like the proposed alternative diagnostic with a) corpora authored by humans, and b) a known distractor element should provide a robust synthetic testbed arguably necessary to gain insight into the proposed framework for using LLMs. - Line 98: cannot just stay Claude-v1.3 – what actually is this, exactly, needs to be specified. - What courses on computational social science (line 110-111) were used? A list should ideally be provided in the appendix. - Line 112: what is the inter-annotator agreement for the exploration goals? - Line 179: why 60? - For the hypothesis validator (lines 183-184), how good are language models in terms of simulating Turkers' validity judgement? Is this a valid proxy for this hypothesis validation step? - Especially for synd5, it seems like there was an opportunity to test with a variety of open source open access models which is especially important since we should operate under the assumption that many users will not have accept to GPT-4. - Line 238: It actually is important that non-authors rate relevance in order to get a more robust assessment of relevant improvement when using goals on OpenD5. - Typo in line 294: the closing quotation marker should not be the beginning of a new line. - Line 329 in related work section: topic models deserve a mention since they are used for the discovery of human interpretable latent concepts, and in fact relevant to one of the experiments conducted in this very work where authors use the topic as one of the dimensions varying between two input corpora samples (see the suggestion above around a more robust synthetic test).

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

While some key limitations are already acknowledged in this work, there does not seem to be a robust discussion on managing and mitigating these in real-world usage of the system. With things like correlation being potentially framed or interpreted as causality, for example, when can users expect such things to happen – without any markers or indications for a system output, it is hard not to be skeptical of any discovery generated by the proposed system. In other words, the key question that seems to be underexplored here: if it is not easily identifiable for an end user when correlation might get framed as causation or bias reinforcement is happening, should the system be used with those potent limitations and harms as a possibility, and given these known limitations and harms, if we do not know if they can be handled effectively or not, should D5 usage be held until those limitations are overcome or at least mitigated effectively? Acknowledgment is an important system, but adequately addressing limitations needs more work, and there are also at least two more really important broader considerations not discussed in this paper that need to be addressed: 1. The work seems to operate in the English language alone, at least in terms of substantive testing and application in the paper itself. There is no discussion about theoretical generalizability to other languages and expectations around that, and at least there should be an acknowledgment of the lack of empirical generalizability of findings about the system to other languages. The languages in which the synthetic and contributed real-world dataset operates should be made clear by the end of the introduction itself. 2. Some important broader impacts/limitations are not being discussed and there should be more discussion in the paper about who gets to use the proposed system and potential double-use scenarios. For example, given that GPT-4 leads to better performance than the open-source alternative, if using this system for effective discovery relies on OpenAI API access and compute resources discussed in Section 8, many scientists and other end-users will probably not be able to use this method of exploration effectively, advantaging some communities over others — leading to many ramifications such as expanding existing inequities in access and resources. How can open-sourced tools be leveraged here, and can smaller models requiring fewer resources be incorporated effectively? This unequal opportunity question also ties to the previous point about language -- doctors, for example, can be operating in different cultural contexts using different languages to make their notes and we do not know if their experiences be used effectively for hypothesis generation same as those operating with English. Low-resource languages are also a potential limitation and concern that warrants discussion.