Explaining Datasets in Words: Statistical Models with Natural Language Parameters

Summary

The authors propose a general framework to generate explanations of different text datasets, by parameterizing the data distribution with textual predicates. The textual predicates, along with their weights, can be viewed as an explanation of the data. Three different tasks are used as examples: clustering, time-series modeling, and multi-class classification. To learn the textual predicates and their associated weights, the authors first iteratively learn the predicates in the embedding space along with the weights, and then use a pre-trained language model to discretize the predicate embeddings into textual space. Experiments are conducted on three datasets that have ground-truth predicates. A qualitative experiment with user-chatgpt dialogues is also presented.

Strengths

1. The proposed framework nicely generalizes from the classical Bayesian text analysis methods and makes them more relevant/useful in the LLM era. 2. Comprehensive experiments are performed on four different datasets/tasks. 3. The paper is well-written and easy to follow.

Weaknesses

1. Missing pure LLM baselines. It is fairly common nowadays that LLMs are directly used to generate explanations given the data, which can be applied to any dataset and any task. More evidence is needed to show the advantages of the proposed method compared to direct prompting. For example, for the clustering task, what if you perform clustering first, and then prompt GPT4 to directly explain each cluster? 2. The explanations generated are mostly topic descriptions, and the datasets used for qualitative results are quite simple. Can the proposed method generate more complex descriptions for more difficult datasets? For example, for a STEM problem-solving dataset (e.g. MATH), can the proposed method explain the dataset by the theorem used/subarea of the subject/similar solving strategy? In summary, I like the way the proposed method combines the classical Bayesian method with LLM prompting, but I'm not totally convinced that the proposed method is significant as there is no direct comparison with LLM prompting, or evaluation on harder datasets. I'm willing to raise my score if my concerns are properly addressed.

Questions

See weaknesses.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes.

Summary

This paper proposes to use "natural language predicates" to explain text datasets. Authors develop a model-agnostic algorithm that optimizes continuous relaxations of predicate parameters with gradient descent and discretizes them by prompting language models. With this method, the authors can characterize the dataset with a distribution over predicates. The authors apply it to taxonomize real-world user chat dialogs and show how the text data's features can evolve across time, demonstrating the potential of the method in understanding text data.

Strengths

1. The paper proposes a new method to explain text datasets. There are a few new concepts in the paper, such as "Predicate-Conditioned Distribution" and the corresponding optimization algorithm that involve both continuous and discrete optimization. 2. A practical application of the method is demonstrated on real-world user chat dialogs. 3. The problem is well-formulated and backed by enough mathematical and empirical evidence.

Weaknesses

1. The motivation of the paper is somewhat unclear. It's not well explained in the Introduction section why people want to "Explain text datasets". Is it for controlling the dataset quality for pretraining? Or tailor the domain of dataset for finetuning LM? Although the authors showcase the method on a dataset of real-world user-ChatGPT dialogues, the results look arbitrary and not very insightful. Only "Taxonomizing User Applications via Clustering" and "Characterizing Temporal Trends via Time Series Modeling" are still not fully demonstrating the usefulness of the method. 2. The method should be compared with more baseline methods. It's unknown what's the benefit of predicate-conditioned distribution over other methods. The authors should also explain why the "predicate" is the key to explain text datasets, instead of other factors such as just verbs, nouns, text representations, etc. 3. The datasets being tested are relatively small and not showing the scalability of the method.

Questions

1. Can you provide more insights on the motivation of the paper? Why do people want to explain text datasets? What are the potential applications of the method apart from the two demonstrated in the paper? 2. What is the benefit of using "predicate" to explain text datasets? How is the predicate better than other forms of explanations?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

Limitations are being discussed in the paper.

Summary

The paper introduces a a framework for a family of models parameterized by natural language predicates. This models allow for a language-based interpretation of text distribution. Such framework can easily instatiate models include clustering, time-series, and classification models. The authors develop a model-agnostic algorithm that optimizes continuous relaxations of predicate parameters with gradient descent and then discretizes them by prompting language models (LM). This approach is demonstrated to be effective in taxonomizing news dataset as well user chat dialogues while also characterizing their evolution over time.

Strengths

I find the paper to have strengths along several axes: - Novelty: The general framework is, to the best of my knowledge a novel approach to describe text datasets in a way that is interpretable and scalable. - Model-Agnostic Algorithm: The algorithm developed to optimize the models proposes is versatile and can be applied to various types datasets under different approaches, namely classification, clustering and time-series analysis. - Extensive experiments and interesting ablations: The authors demonstrated the practical utility of the approach by applying it to real-world datasets (user-LLM chats), as well as news data for comparison purposes. - The results show improvements over the baseline and when this is not the case Bills and NYT dataset for clustering in table 2, the difference is either minimal or compensated with the generality of the proposed method. Overall the paper is well written and easy to follow. The results well presented and support the claim made in the paper. The limitation as well as weaknesses of the work are presented and discussed.

Weaknesses

I did not find major weaknesses in this paper. - The main one would be in the addition computational and money costs for the proposed method compared to other classical approaches. However, in my opinion, the additional costs are counter balanced by performance and generality of the method. - Another weakness is the use of a single text embedding model. I do not think this addition experiment is necessary for accepting this paper, but I do think that testing the robustness of this method with various text embedding would make the presented claims stronger. - Additionally, given the experiments are averages across several seeds, I would report the variance of the results. ---- MINOR - Please adjust the citation format, in the text there's a number-based format but there are not numbers in the references - Very minor: then I tried to read table table 1 (pag 6), I was confused about what metric surface was. This was explained later in page 7. I would add a brief explanation of surface in the caption of the table, make a reference to the section when surface is explained or move the table closer the experiments section. - Typo in caption of table 2: relaxiation -> relaxation - line 277 general purpose -> general-purpose - line 648: you refer section G.1 from section G.1, did you mean section E.3? --- Recomendation Overall I consider this a strong a paper and I vote in favor of its inclusion to NeurIPS

Questions

- How are you selecting language predicates lengths? Given the NLG component in the LLM call, can predicate length evolve over time? Have you done an analysis of such change in your optimization process? - When you mentioned ranking by correlation scores: the correlation is computed between $[[\phi]]$ and the dot product between a predicate relaxation and a sample embedding. Are you correlating continuous vs discrete values? Is $[[\phi]]$ a binary value? Could you please clarify your correlations here? - In the no-refine case, am I reading the results correctly If I understood that you select random predicates and discretize after one full round of optimization (of $w$ and $\phi$ them once? - In the no-relax case, you are still discretizing using a language model, is that correct? Would this just mean that the optimization process would take longer given that you are skipping the correlation-based selection criterion?

Rating

7

Confidence

4

Soundness

4

Presentation

4

Contribution

4

Limitations

Yes, in section 6, E.3 and G.1.

Looking forward to your comment

We have run the experiments to address your reviews. Please feel free to let us know if you have any questions. Thanks a lot!!