The Goldilocks of Pragmatic Understanding: Fine-Tuning Strategy Matters for Implicature Resolution by LLMs
Despite widespread use of LLMs as conversational agents, evaluations of performance fail to capture a crucial aspect of communication: interpreting language in context -- incorporating its pragmatics. Humans interpret language using beliefs and prior knowledge about the world. For example, we intuitively understand the response "I wore gloves" to the question "Did you leave fingerprints?" as meaning "No". To investigate whether LLMs have the ability to make this type of inference, known as an implicature, we design a simple task and evaluate four categories of widely used state-of-the-art models. We find that, despite only evaluating on utterances that require a binary inference (yes or no), models in three of these categories perform close to random. However, LLMs instruction-tuned at the example-level perform significantly better. These results suggest that certain fine-tuning strategies are far better at inducing pragmatic understanding in models. We present our findings as the starting point for further research into evaluating how LLMs interpret language in context and to drive the development of more pragmatic and useful models of human discourse.
Paper
Similar papers
Peer review
Summary
The paper introduces a task to evaluate the ability of large language models (LLMs) to resolve conversational implicatures and go beyond the literal interpretation of the meaning of the language. The evaluation is based on a dataset of naturally occurring implicatures that are converted into a binary classification task using a set of templates. For instance, the dialogue _"Can you come to my party on Friday?”_ , _"I have to work"_ is converted into two sentences: _Esther asked “Can you come to my party on Friday?” and Juan responded “I have to work” which means no_ and _Esther asked “Can you come to my party on Friday?” and Juan responded “I have to work”, which means yes._. The two sentences are inputed in the different models, considering the one maximizing the likelihood as the model choice. Four categories of state of the art models are evaluated on the proposed task. The evaluation protocol takes into account potential variance induced by prompt templates. Different methodologies are also benchmarked to try to improve the performance of the models on the task, this includes: - Fine tuning on the task. - Instruction tuning-; - Chain of thought prompting Results demonstrate that the task is challenging for base models (BERT, RoBERTa, GPT2, GPT3, ...) with a performance slightly above random. Without any fine tuning, some models already reach a performance above 70% (text-davinci-00x, ChatGPT), with GPT4 only 5 points behind human performance. The improvement induced by instruction tuning and chain of thought prompting is further demonstrated for some models with GPT4 reaching human performance.
Strengths
The paper is well written with all the material provided to reproduce the results. The paper introduces an original evaluation protocol for the task of resolving implicatures and demonstrates that human performance can be reached by combining state of the art models (GPT-4) and the recently proposed chain-of-thought prompting.
Weaknesses
The task proposed is not new and the dataset proposed is the same as the one introduced in [BigBench](https://arxiv.org/abs/2206.04615). The results, in the mentioned paper, already demonstrated the ability of PaLM with k-shot-prompting to [perform](https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/implicatures) above average human performance. The main differences between the two papers include the choice to keep ambiguous cases for the task and adding other models along with a chain of thoughts prompting to the benchmark. In the benchmark, the variance in the performance of the models, displayed in Table 2, only takes into account the variability induced by different prompt wording. To compare the models, it would have been helpful to perform bootstrap on the test set to estimate to support the claim on the comparison of the performance of the different models.
Questions
Why has no fine tuning on the base models been performed? Do you have the increase in performance induced by chain of thoughts prompting for GPT4 for particularized (context heavy) examples? l107 _"Our analysis is novel in its approach; using ambiguous data that humans can easily resolve, and scope"_ Maybe reformulate, not very clear GPT4 doesn't seem to benefit from in context example. How do you explain that? Did you try to isolate the effect of data contamination? Especially for GPT-4, the model could have been trained on the data used in your evaluation protocol given that the original dataset was published in 2020
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
The proposed evaluation framework focuses on conversational implicature. If implicature is key for natural language understanding, evaluating models on conventional implicature is also important. If outside of the scope of this well conducted evaluation, it could be of interest for future work.
Summary
This paper evaluates several popular LLMs on a benchmark for evaluating pragmatic reasoning via resolving question-answer conversation snippets into the implied binary answer. The study explores the influence of different factors in LLM design, including training method (next-token prediction vs. instruction tuning), number of few-shot examples, model size, inference method, etc. Several conclusions are made about the results: instruction tuning seems to improve pragmatic reasoning, models are mostly robust to perturbations in the input prompts, scaling model size seems to have an effect in improving performance, CoT prompting is useful for some models, and remaining failures of the model appear to center on more difficult (particularised) implicatures.
Strengths
* The paper offers a very comprehensive / thorough / careful evaluation of the LLMs being evaluated. Particularly, the try a number of different prompt variations, try different numbers of few-shot examples (as well as having a randomization process for choosing these examples, to avoid confounding results with some fixed set of examples) * The finding that few-shot examples are useful mostly for conveying format of the response, rather than as a way of defining the task itself, is very interesting. I would be interested to see a more in-depth study of this (perhaps another paper) as it is somewhat surprising (but perhaps less surprising for instruction-tuned models).
Weaknesses
In general, I wanted to see a bit more discussion and analysis, since this paper is focused on analysis. * Moving the analysis for other types of implicatures into the main paper * More examples of what the model is generating, particularly for CoT examples * More discussion / analysis of human "errors" here -- why might there be errors? Genuine noise in the annotation process, or different interpretations of the same example? * Discussion on why instruction tuning might be influential in improving pragmatic reasoning * Discussion / analysis on why there is still a gap for particularised implicatures, and what we can do to address it Nits: * The first example (GPT's answer to the user's question about the phone) actually doesn't necessarily make sense, as it's definitely not literally true -- GPT can't have seen the user's phone * "fine-tuning on instructions at the example-level" is vague to me, particularly the "example level" part. What is this supposed to refer to? Is there a distinction between general "instruction-tuning" and "instruction-tuning ad the example level"? * Also, "context" in Insight 5 is vague -- I wouldn't think of this kind of commonsense/world knowledge as "context" necessarily, so I'd suggest renaming this * Formatting of Figure 2 is somewhat odd * I'd suggest adding a line for "human performance" in Figure 2 * I'd suggest adding the few-shot CoT example (or some examples of generated CoT answers) into the main paper
Questions
* Is there evaluation on this dataset with respect to an upper bound in performance given only the response, without the question? I'm curious whether answers like "I've gotta get up early" or "Some" have some spurious correlations with the labels, e.g., "no", in this dataset. * Is the accuracy actually decreasing relative to k in Figure 4? I.e. is this difference significant? If so, why might that be? Minor questions: * Are there cases where examples are ambiguous, and situational context might affect the label? Did you investigate cases where the human annotators were "wrong" to identify the source of those "errors"?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
Yes, mostly. More discussion on these non-binary implicatures (at the end of the discussion) would be interesting. I'd also be interested if the authors can perform analysis on the existing benchmark to see whether the labels can be predicted with only one half of the example (i.e., the response); i.e. whether there are spurious correlations within the benchmark.
Summary
The authors analyze the behavior of LLM pragmatic understanding (capability to add and omit imformation to efficiently communicate in context) using a new evaluation protocol. They compare LLM performance to human performance and argue that instruction fine-tuning with examples may improve pragmatic understanding. They propose using the assignment of higher likelihood to coherent utterances than similar but incoherent utterances as "resolving an implicature correctly." This includes both describing a situation where implicature understanding is necessary to essentially resolve an entailment question, whether an answer that on its face is irrelevant means "yes" or "no," or whether its a sufficient answer. For example, `Esther asked “Can you come to my party on Friday?” and Juan responded “I have to work”, which means yes.` is a failure case because the answer is wrong; the response means no. A model preffering `...which means yes.` resolves the implicature. They assess this problem is few and zero shot settings for LLMs. They control for prompt variation effects with 6 different prompt settings, testing a fairly comprehensive set of both generative proprietary LLMs and standard open pretrained transformers like RoBERTa. They find no model can beat human performance on their resolution task, but GPT-4 comes closest. They conclude with a detailed analysis, with 5 insights.
Strengths
Well-defined and scoped problem. Out-of-community definitions (eg, explaining pragmatics) well-motivated. Citations are sufficiently comprehensive in my opinion. Exciting direction to explore to further understnading and engineering of LLMs. Fully-specified approach and sufficiently broad evaluation. Clear statement of insights. Overal, I think this paper has clear value to the community.
Weaknesses
I think it would be possible to use more than 6 "curated prompt templates" (163-172). The "we control for prompt variability" is really not central to the story of the paper but I don't really find the idea that averaging in essence over 6 prompts is sufficient to argue that that has happened. Perhaps claim of robustness should be weakened. ~They claim task-specific instruction tuning improves implicature resolution, but there's no concrete A/B test for this. They don't instruction fine-tune a model themselves for the task (not that doing so would necessarily be possible given constraints). How do we know that IFT is really driving GPT4's superior performance and not just scale or some other factor? I find the evidence presented here unconvincing~ EDIT: In light of the author's response, I do think the case for IFT driving the improvement to be a bit more convincingly made, or at least as convincingly made as can be given the inherent constraints (OpenAI's closedness for ex). I hope to see the authors response to this point integrated in the camera ready, with maybe some weakening of claims in light of the incomplete degree of knowability of this issue
Questions
Maybe respond to the questions I raised implicitly in the weaknesses? I am open to being convinced.
Rating
8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
4 excellent
Presentation
4 excellent
Contribution
3 good
Limitations
I find limitations to be sufficiently addressed this time.
Summary
This paper investigates how well recent LLMs preform on resolving conversational implicatures. It presents a task on conversational implicature resolution built on top of the crowdsourced and human-annotated dataset of George and Mamidi (2020). The work presents an evaluation protocol that lays out how LLMs are evaluated on implicature resolution. The work further presents an array of experiments on various LLMs to understand whether they do well on this task and in what settings (e.g. 0-shot vs 5-shot). Results are analyzed in the form of insights and main takeaways that the authors highlight.
Strengths
- Addresses an important problem, namely studying the competence of LLMs at pragmatic inferences—here specifically implicatures - Extensive experiments on a wide range of recent LLMs to validate how well these LLMs perform pragmatic inferences involving implicatures
Weaknesses
- The work is poorly situated with respect to prior work on computational modeling of pragmatics and the related work section misses an important line of work which I discuss in the comments. - Further analysis is required in certain cases (see comments below)
Questions
Intro: Line 28: “utterances that convey something other than their literal meaning” At the first pass of this sentence and reading “other”, I read it as suggesting implicature conveys something else different/unrelated to the literal meaning, and I had to reread it to fit it within the definition of implicature. I suggest replacing “other” with “beyond” as I feel that fits better the description of implicature. Section 2: As a work that is, in essence, motivated by investigating how good LLMs are at “interpreting language in context—incorporating its pragmatics”, I feel the related work section leaves out an important line of work on the computational modeling of pragmatic phenomena including implicature and the closely related phenomenon of presupposition and on understanding the limitations of modern NLP models in capturing these phenomena starting with the work of Cianflone et al 2018 on modeling adverbial presupposition, Schuster et al 2020 & Li et al 2021 on scalar implicature, the ImpPress dataset (Jeretic et al 2020) on presupposition and implicature in the context of NLI and related experiments on how good NLI models (including BERT-based models) are at capturing these phenomena, Kim et al 2021 on presupposition verifiability in the context of QA, Parrish et al 2021 with the NOPE presupposition corpus, Kabbara et al 2022 on the limitations of NLI models in resolving presupposition cases. These first come to mind. Despite these efforts focusing on (what is now) smaller LMs models (with the exception of the first mentioned work which precedes the BERT era), I believe they are important efforts that set the stage for the present work and need to be cited and imo are more relevant to the work than some of the information mentioned in the first paragraph of Section 2 (e.g., how some models are toxic, unhelpful, etc). Appendix D has a nice background on implicature from a linguistics/philosophy of language but still misses all of the work I mentioned here on modern computational modeling of pragmatic phenomena. Section 3: A general concern of mine is the benchmark contribution of the paper given that the same dataset has been introduced in BIG-bench. The authors are clear about how their dataset is different given that it encompasses instances that are discarded in BIG-bench (for being too “ambiguous”) but regardless the overlap between the two is obviously substantial and so I felt this is something that could benefit from a discussion here. Section 4: Regarding the random label experiment and looking at the related section in the appendix (I.6), two points/questions come to mind: 1) Given that there are two labels in question, did you verify how often the label was not changed? That could eclipse the results we’re seeing here. 2) This experiment and the close results in some cases (for 5-shot text-davinci-001 and cohere-command-52b) brings up the following question: to what extent the models are learning any notion of pragmatics here? Did you carry out some qualitative analysis in a systematic way to see whether the models are in some cases getting it right for the wrong reasons? For example, Kabbara el al 2022 showed that for presupposition-based NLI, RoBERTa and BERT were getting high accuracy performance on certain presupposition types but upon further investigation and adversarial testing, it turns out the models are often exploiting superficial cues that are not related to notions of pragmatics. I wonder to what extent this could be applicable for the task you’re exploring here. Regarding the analysis in Appendix I.4 (varying k in-context examples), apart from the conclusion that is in the body of the paper (Section 4, Insight 2), I somewhat feel the results are so noisy for Cohere-52B and OPT-175B that it’s hard to really make sense of the results. InstructGPT3-175B is the only case where the results follow our intuition: the more examples a model sees, the better it gets at understanding the notion of implicature and so the higher the accuracy. For Cohere, we see a very noisy trajectory across the board between k = 1 and 15 (before the results start stabilizing and moving upwards) and for OPT, the results are mostly on a downward trajectory except for a chunk in the middle and end up still going downward for k>15. Would like to see if you have any thoughts on this. Table 3: Regarding the 5-shot CoT performance We see a variance between the text-davinci models. Generally, although specific details are not public, we can assume models got better as newer versions were released. So it makes sense to see the performance improve for the 5-shot and 5-shot COT scenarios across the 3 davinci-models. However, it’s still unclear why CoT would help marginally in one case (002) and in a rather noticeable way (003) but hurt the performance in the 001 case. Any insight there? This is even more pronounced when the drop is not trivial (7%) and is the only case among 6 (if we were to count the -0.1% as roughly not helping but not hurting) Appendix G: Human evaluation I’m somewhat confused by the choice of the authors to run human experiments using on prompt only. They could’ve simply chosen to run multiple experiments with at least a couple of different prompts. The authors justify this by the fact that humans are less likely (than models) to be sensitive to variations in the prompt. I feel this is at best speculative and, if anything, human experiments and pilot studies often show that human subjects can be primed to think or react in a certain way given certain wording or structure, etc. Minor presentation note: Line 25: I’m not sure if there’s a writing style guide that allows this form “I have to work.”. [essentially the two dots back-to-back]. Seems rather odd to me. I would think the one within the quotations need to be dropped. See here for more details: https://www.hamilton.edu/academics/centers/writing/style/essentials/punctuation-of-quotations#:~:text=The%20final%20period%20or%20comma,the%20period%20follows%20the%20citation.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
4 excellent
Presentation
3 good
Contribution
3 good
Limitations
Yes
Thanks a lot for answering all the concerns mentioned. Once again, I think it's a well conducted paper with a comprehensive evaluation providing interesting insights on the ability of LLMs to resolve conversational implicatures. - About instruction tuning, you are completely right and sorry for the confusion. And you nicely demonstrate the benefit on this methodology to resolve implicatures. - About the BIG Bench task, I understand the two limitations you are mentioning: overestimation of the human performance and the other one about the choice made to keep only the subset of unambiguous examples in the BIG Bench benchmark. About the human performance, you performed a thorough evaluation, involving 5 annotators for each subsets of 150 examples with metrics on annotator agreement. This is way more detailed that the BIG BENCH paper. About the choice, made in Big Bench of keeping the subset of *unambiguous* cases, it seems that the annotators, in your paper, are able to perform significantly above chance level. So, indeed, they might have been too conservative when selecting the subset but I don't think it makes the task more interesting. You also mention that the *BIG bench task uses only base LLMs and no state-of-the-art fine-tuning methods*. It's important to note that they reach above human performance relying on it. - About the effect of data contamination, thanks a lot for providing the performance with the question only for GPT-4. With those numbers, it seems indeed that data contamination is unlikely. I think adding this in the Appendix could indeed be helpful. Could you also specify how you evaluated GPT-4 with question only? Finally, I want to thanks the authors for addressing all the comments. The paper is well written with a well conducted benchmark, ensuring reproducibility with all the details provided. My main concerns were about the novelty of the work done and the potential moderate impact given the BIG BENCH paper. To be more specific, the task seemed already to be [solved](https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/implicatures) with the big bench paper, where they demonstrated that relying on PaLM with k-shot-prompting performed above average human. Given that the Big Bench paper is not peered reviewed and that evaluation is way more detailed in your paper, I have raised my score.
We thank the reviewer for thoughtfully responding to our rebuttal, and revising their score. We are curious as to what would hold the reviewer back from stronger support for the paper. We believe it might be down to some outstanding misunderstandings, and would appreciate working with you both to ensure that the writing can be amended where needed such that other readers do not make them, and in the hope that you will consider bringing your level of support for the paper in line with the other reviewers. **BIG bench** We believe (correct us if we’re wrong) that what is standing between the reviewer and stronger support for the paper is related to a perceived limited impact due to the BIG bench result, as the reviewer also mentions they think the paper is *"well conducted paper with a comprehensive evaluation providing interesting insights on the ability of LLMs"* and say that we are *"ensuring reproducibility with all the details provided"*. Below, we aim to clarify that the difference with the BIG bench is not *just* that our work was peer reviewed and theirs wasn’t, but that—beyond the novel analysis and model comparisons in our paper—**our version of the task is also harder**, with differences in the data distribution and experimental protocol, which we believe makes the evaluation **fairer and more informative**. We expect PaLM's few-shot score to be lower on our dataset due to the difficult additional examples in our benchmark comprising 30% of the total data. We expect BIG bench human performance to be underestimated due to low-quality human evaluation, and actual human performance to be higher, like on our benchmark. Therefore we cannot be sure that PaLM would reach above human level performance on our benchmark, and with it, solving the task -- in fact it seems quite unlikely. Taking all this into account, we would argue that you simply cannot say much about implicature performance of contemporary LLMs based on the BIG bench result at all, and in turn our paper is **the first to actually provide this contribution**. **Spurious correlation experiment**. We re-ran the benchmark fully but removing the responses from the examples. We did this 0-shot and 5-shot, where we did the latter to give 5 examples of how to potentially utilise spurious correlations in the benchmark. The results are as follows: ``` | Question-only | 0-shot | 5-shot | |---------------|----------------|----------------| | ChatGPT | 54.3% +/- 3.3 | 41.7% +/- 12.4 | | GPT-4 | 48.9% +/- 10.5 | 53.7% +/- 0.5 | ``` The result for chatGPT and GPT-4 are mostly random, meaning that it's unlikely these models will utilise spurious correlations in our benchmark in a few-shot setup. **********************Conclusions********************** We hope that clarifies the fact that the contribution of this paper is indeed ******************************************substantive and novel******************************************. To further clarify the writing regarding this, we will revise the related work section to get the above across more clearly. Specifically, we will focus more on the claim that we still don’t know LLM’s performance compared to humans from the BIG bench result because it likely overestimates LLM implicature understanding and underestimates human performance. We would be thankful if you had any further suggestions regarding where we could make this distinction clearer, and hope you will be in a position to strengthen your support for the paper.
Thanks for the answer to my questions. As explained in my first comment, I acknowledge the quality of your work but I am still questioning the novelty. As apparently I made *some outstanding misunderstandings*, I am going to try to rephrase why I think the paper is interesting but with limited novelty. - One of the main arguments you state is that your version of the task is harder. More specifically, you kept *the difficult additional examples in our benchmark comprising 30% of the total data*. I want to emphasize that some of the filtering performed for the BIG BENCH task seems legitimate. For instance, they provide an example of factual mistake in the original dataset: *Speaker 1: Are you busy?* *Speaker 2: I’m drowning in work.* *Original answer: No.* To me, removing this example and others that are too ambiguous to reach human agreement mostly changes the expected human performance but doesn't make the task more fair or informative. - One important strength of your evaluation protocol compared to the BIG BENCH Benchmark is the addition of the prompt sensitivity analysis over the prompt templates. One again this demonstrates the soundness of your approach and I want to highlight once again the quality of your work. However, based on the results, we can observe that the variability induced by the prompt template is quite low. Therefore, I disagree with your statement: *"you simply cannot say much about implicature performance of contemporary LLMs based on the BIG bench result"*. I want to reiterate that the paper is very well written, reproducible with experiments that are well conducted. Your work is clearly more detailed than the one from BIG BENCH. However I disagree with the statement that you are the first to assess implicature performance of LLMs. As I understand it, it mostly come from our disagreement on the importance of including what you call the *the difficult additional examples*. I hope it clarifies my view on the novelty of the paper and happy to discuss if you need further clarification.
We thank the reviewer for the swift response and engaging so actively with our rebuttals. We genuinely appreciate this. We do however still (respectfully!) disagree with the reviewer’s comments about the BIG bench result, and elaborate further below. **BIG bench** While it is true that our protocol keeps all the data of the original dataset collectors, and therefore has some annotation errors, the fact remains that on this entire subset of 30% of the data humans achieve 72%. Even if there are some annotation errors, humans still get significantly above random accuracy on the examples. So before we can say PaLM achieves human accuracy, we should verify the accuracy it gets on these more challenging examples. Two examples that BIG bench also discards: Utterance: *"Can you lend me hundred dollars?"*. Response: *"Is this supposed to be some kind of a joke?"*. Implicature: *"No"*. Utterance: *"Do you know, how long is Uncle Arthur staying with us?"*. Response: *"Ask your father."*. Implicature: *"No"* These are valid implicatures that humans get correct. They are indeed ambiguous, but that’s the point of implicatures. As the reviewer says, the annotation errors don’t make the task more fair or informative, but the ambiguous examples do. **Prompt sensitivity** The reviewer also mentions a strength of our protocol is the estimation of prompt sensitivity and that based on our result we can say variability is low. Now, we might guess that for PaLM prompt sensitivity also won’t be an issue (although we do not know this for sure), but without our result we wouldn’t know this. To summarise; the reason we say we can’t say much about LLM implicature performance from the BIG bench result is because it overestimates LLM performance by also discarding valid examples, likely underestimates human performance, doesn’t estimate prompt sensitivity, doesn’t evaluate SotA fine-tuning techniques, and is not reproducible or analysed in any paper.
Dear reviewer, Thank you again for your responses and the engagement with our rebuttal. In light of the nearing end of the rebuttal period, we’d like to kindly remind you of the above response to your clarification on the novelty of our work.
Thank you for the reply! In terms of discussion about non-binary implicatures, apologies for a vague suggestion. I do think discussing evaluation would be interesting (both metrics, and how to collect this kind of data), though don't want to draw from the focus of the paper. Thanks also for the additional analysis and examples. Will update my score as this reply has addressed many of my concerns!
Thanks for the response!
We thank the reviewer for the response and the increase in rating. We will allocate some of the additional page in a final draft of this work to discussing non-binary implicatures in more depth. We are confident that with your thoughtful review we have been able to improve the manuscript; thanks for your engagement!
Answer to rebuttal
Thank you for the detailed response to my review! 1) Regarding the references that I mentioned, this is the complete list of papers (according to the order of appearance in my review): - Cianflone et al (ACL 2018): Let’s do it “again”: A First Computational Approach to Detecting Adverbial Presupposition Triggers - Schuster et al (ACL 2020): Harnessing the linguistic signal to predict scalar inferences - Li et al (SCiL 2021): Predicting scalar inferences from “or” to “not both” using neural sentence encoders - Jeretic et al (ACL 2020): Are Natural Language Inference Models IMPPRESsive? Learning IMPlicature and PRESupposition - Kim et al (ACL 2021): Which Linguist Invented the Lightbulb? Presupposition Verification for Question-Answering - Parrish et al (CoNLL 2021): NOPE: A Corpus of Naturally-Occurring Presuppositions in English - Kabbara et al (COLING 2022): Investigating the Performance of Transformer-Based NLI Models on Presuppositional Inferences I believe these should be referenced in the main body of the paper as they all present various research efforts on either modeling the pragmatic phenomena of implicature and/or presupposition or understanding the limitations of LLMs in capturing these phenomena. And so, mentioning them properly situates the work. I would leave it to you to best decide to what level of detail to cover them in the Related Work section and referencing them in some way (e.g. grouping several by some topics) VS expanding on which in the appendix. 2) Are the models learning pragmatics? Thanks for presenting these results. The answer is convincing to me. I would think highlighting this in your paper (possibly in the body of the paper by just giving the insight and then further expanding on this in the appendix) will benefit the paper because this specific point (is the model actually learning pragmatics?) is often a point that is raised in this type of papers that attempt to highlight a competence of LLMs especially when it comes to whether they exhibit some form of “understanding”. Specifically, many in the community these days would reject that LLMs can exhibit any kind of understanding, let alone a pragmatic understanding. So such a discussion will strengthen the position of the paper. I replied above to points that either asked for a clarification or that I wanted to further comment on. The rest of the answers are satisfactory. I will raise my score as the rebuttal addressed most of my concerns (raised both overall score and the soundness score).
Thanks for the response!
Thanks a lot for the additional response and for listing these related works in more detail. We agree they do set the stage for our work, and merit discussion. We also agree with the reviewer that the discussion highlighting why we believe models are likely actually learning something interesting here would be an interesting addition to the paper, further motivating the contribution. We thank you for your engagement and believe we will have improved the manuscript as a result of our discussion.
Thank you for your thoughtful response. On the IFT claim, I am convinced, and have updated my confidence. I would encourage the authors to add this to the discussion section in the camera ready if it isn't already there! :)
Decision
Accept (spotlight)