Thank you for the thoughtful response! We address your remarks below:
> **Do you make any assumption on the outcome distributions related to actions with different costs? Or do you just refer the action with the highest quality to the action with the highest cost? If this is the case, why do your results only rely on incentivizing actions with the highest costs without any assumptions made on the distributions? Could you provide more intuitions and explanations for it?**
This is a great question. Our main theoretical results only assume that the target action is implementable - i.e., that there exists some contract incentivizing it (see, e.g., Appendix B.2). Intuitively, implementability is equivalent to the assumption that the observed quality of the target LLM is different (in distribution) from the observed quality of smaller, cheaper models [1]. One intuition for the fact that no further distributional assumptions are required is the equivalence to optimal composite hypothesis testing (by Theorem 1), which doesn't require further assumptions as well.
We also note that the implementability assumption can be verified in polynomial time given outcome distributions, by checking the feasibility of the corresponding linear programs (i.e. equations (1,3)). Additionally, the assumption was verified to hold in the empirical datasets we analyze. In scenarios where the highest-cost action cannot be implemented by any contract (i.e., when the observed quality of the costly LLM is identical to the quality of cheaper ones), this action can be ignored, and the next highest-cost action effectively becomes the highest-cost one and can be targeted instead. We will further emphasize these points in the paper.
Further extending on this remark, we also note that another common theme in the literature is providing stronger guarantees on the resulting contracts by making stronger assumptions about the structure of outcome distributions (see, e.g., [1,2]). Connecting to this theme, in Proposition 2 we show that the MLR structural assumption (Monotone Likelihood Ratio, Def. 5) implies a threshold functional form for the optimal cost-robust contracts. However, from the empirical perspective, it is also worth noting that the outcome distributions we observe in our empirical study don’t seem to satisfy the theoretical assumptions currently available in the literature (for example, see Figure 2 middle right, which shows the non-trivial outcome distributions of the MT-Bench dataset). In this context, we hope that our empirical observations will motivate future theoretical research with refined structural assumptions. We will emphasize this point in the paper as well.
> **By saying basic contract problem, I mean that the problem can be studied in the basic contract setting without the modeling of LLM and Text Generation. The problem is actually a robust contract design setting where the action costs are uncertain, and LLM is just an application. Using the language of contract design with moral hazard would significantly simplify the notations.**
Thanks for the clarification, this is also a very good point. The cost-robustness model and our theoretical results indeed apply more broadly, and extend beyond the context of LLMs and text generation - We view it as one of the paper’s main strengths. Through the use of "application specific" language (i.e. focusing on LLMs), our hope is to promote discussion between different scientific communities, which will eventually increase the applicability of contracts in this setting. The LLM market is nascent and evolving - for example, more players are joining, and different pricing (contract) schemes are emerging. The current naïve pricing schemes, which do not yet tie payments to performance, are likely to be reshaped and improved. Experience from the sponsored search market shows that economic theory informs better pricing [3], and we expect that through contractual payments this will be the case for LLM pricing too. In any case, we will also update the paper to further clarify the significance of our results to general contract design theory.
> **One minor: It seems that the references are not updated. the reference of Saig et al'23 is still the Arxiv one.**
Thank you for this remark! We will gladly fix this.
–
Please let us know if our response has addressed your questions regarding distributional assumptions, and the relation to general contract design theory. Also, if the discussion so far increases the favorability of your assessment, we would greatly appreciate it if you would consider increasing your score. In any case, your remarks are very helpful, and we are more than happy to discuss any additional questions or thoughts that come up.
References:
* [1] Dutting et al., Simple vs Optimal Contracts, EC 2019.
* [2] Saig et al., Delegated Classification, NeurIPS 2023.
* [3] Ostrovsky and Schwarz, Reserve Prices in Internet Advertising Auctions: A Field Experiment, JPE 2023.