Decision-Making Behavior Evaluation Framework for LLMs under Uncertain Context

Summary

This paper proposes a comprehensive framework to evaluate LLM's decision-making behavior under uncertain contexts. By leveraging computational models from behavioral economics, as well as the experiment paradigms, the authors investigated the risk preferences, probability weighting, and loss aversion of LLMs from the parameters in the models. Finally, by injecting demographic information into LLMs, the authors found significant differences in those parameters measured from behaviors, suggesting potential biases from demographic information (e.g., age, education, or gender) on decision-making tendencies.

Strengths

- This paper advances the evaluation framework of LLM's decision-making behavior under uncertainty, borrowing the models from behavioral economic theories to quantitatively model the behavior of LLMs. By leveraging risk preference, probability weighting and loss aversion, the authors have a deeper investigation on risky decision-making behaviors. - This paper also investigates demographic influences on the LLMs' behaviors, suggesting potential biases in generating behaviors under different demographic settings.

Weaknesses

- __Lack of human data__. As the authors have mentioned in the limitation section, it would be fun to investigate the LLM vs. Human behavior at the parameter level of computational models. These parameter differences might explain why humans and LLMs are different or similar on a deeper level. - __Lack of model selections__. The authors ONLY use one model (TCN model) to evaluate the LLM's behaviors. In computational cognitive sciences and psychology (Wilson, Collins, 2019), it is common to propose multiple cognitive models to test which model might best explain the behaviors of humans (and of machines). The authors could strengthen the model evidence by proposing multiple competing models, comparing models, showing goodness-of-fitting, and even possibly simulation and parameter recovery to show the robustness of the currently selected model. Perhaps there might be better models that can explain the behaviors better and thus, the attribution of behaviors may change. - __More investigations on LLMs are recommended__. The authors list one of the contributions as investigating multiple LLMs in this framework. The LLM selections are the most known commercial LLMs (like GPT-4, Claude, and Gemini). These model comparisons somehow make sense but two of them are not open-sourced and the training data are not transparent. If the authors could compare some LLMs with a more transparent basis to provide insights into LLMs, this would strengthen the contribution of the paper. For example, the authors could investigate the relationship between model size and the parameters measured from the behaviors. Choosing models within the same family, and trained on similar datasets would make comparisons more informative. Looking at prompt engineering, the way of instruction-tuning or model structure (e.g., encoder-decoder model or decoder-only model) can also make sense while making this framework provide more information.

Questions

- One interesting finding from the paper is that human studies may not find significant differences in risk preferences in sexually minority groups but LLMs do respond differently under these settings. There might be two but completely different reasons: first, the authors did not get the full knowledge of the literature in that field and the paper listed there may be only one study that reports non-significant research. The other reason could be intriguing: the studies about one specific topic are limited but humans are generally exhibiting such effects which are overlooked in empirical study. However, LLMs are more likely to learn such effects from massive datasets and thus exhibit effects. So there might be a reverse approach that understanding human behaviors (especially in fields that are overlooked) from LLM's behavior. The authors are not required to formally give an answer to this question, but I find it interesting in public discussion about such potentials.

Rating

6

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

As indicated above, the major limitations of the paper are not providing a deeper insight into some human cognitive or LLM mechanistic findings, though this framework is valuable and more comprehensive than previous work.

Summary

This paper presents a novel framework for evaluating the decision-making behavior of large language models (LLMs) under uncertain contexts, grounded in behavioral economic theories. The authors conducted experiments to estimate risk preference, probability weighting, and loss aversion for three commercial LLMs: ChatGPT-4.0-Turbo, Claude-3-Opus, and Gemini-1.0-pro. The study further explores the impact of embedding socio-demographic features on the decision-making process of these models, revealing significant variations and potential biases. The paper concludes with a call for the development of standards and guidelines to ensure ethical and fair decision-making by LLMs.

Strengths

1. Introduces a comprehensive framework for evaluating LLMs' decision-making behavior, which is the first application of behavioral economics to LLMs without preset behavioral tendencies. 2. Provides empirical evidence of LLMs' tendencies towards risk aversion and loss aversion, with a nuanced approach to probability weighting, offering insights into their alignment with or divergence from human behavior. 3. Conducts further experiments with socio-demographic feature embeddings, uncovering disparities in decision-making across various demographic characteristics, which is crucial for understanding potential biases in LLMs.

Weaknesses

1. The paper does not provide a detailed analysis of the potential causes behind the observed variations in LLM behavior when demographic features are introduced, which could be crucial for understanding and mitigating biases. 2. The study's focus on three commercial LLMs may limit the applicability of the findings to some open source LLMs like LLaMA. 3. The authors should discuss the references like chain-of-thought (CoT) prompting and reasoning in LLMs, such as "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models" (https://arxiv.org/abs/2201.11903), "Automatic Chain of Thought Prompting in Large Language Models" (https://arxiv.org/abs/2210.03493), and "Keypoint-based Progressive Chain-of-Thought Distillation for LLMs" (https://arxiv.org/abs/2405.16064). These works could provide additional context and depth to the understanding of LLM decision-making processes.’

Questions

Please see the Weakness.

Rating

4

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes

Summary

In this paper, the paper the authors model the decision-making behavior of certain open-source LLMs on lottery selection tasks. They further extend this work, by repeating similar analysis after priming the models with demographic information. Results indicate that the different LLMs differ in their risk profiles as inferred by the parameters of the evaluative model. Some prominent differences also emerge on demographic-primed LLMs. The overarching theme that comes through is the need for caution when using LLMs in practice, especially as analogue to human decision-making. **Conclusion:** Overall, while the paper considers an interesting problem, the results don’t appear to yield any general insights, and the methodology seems to make assumptions that may be difficult to justify. The paper can still be an alright addition to the conference since it’s timely and touches on an important problem.

Strengths

1. It’s clearly written, provides ample context and the ideas are developed in a streamlined fashion. 2. The problem considered is pertinent and the approach seems to be well motivated by a long history of modeling human behavior. 3. The analysis makes sense — studying the distribution of pertinent parameters conditioned on different agents, and demographics to derive insights about the underlying behavior.

Weaknesses

1. It’s unclear whether the TCN model is actually apt in this setting. For example, it seems to assume deterministic values of parameters like sigma, alpha and lambda, and then tries to infer them based on the LLMs decisions. However, what if these parameters are themselves random variables? Does it make sense to assume the LLM has a fixed risk profile across trials? As an analogy, assuming the coin has a fixed bias p, and estimating it via repeated tossing experiments when p is sampled from a uniform distribution each trial. 2. The utility function was a little unclear to me (line 135). Why does it not cover the entire (x,y) domain? w(p) simplifies to 1/p for alpha = 1. Should it be p? As a tangentially related point, consider numbering the equations. 3. In Step 4 (section 4), can we be certain the estimation algorithm actually converges? This relates to 1, so that the convergence might not be guaranteed if the latent parameters are not fixed. 4. While the qualitative results are sensible — due diligence is needed before LLMs can be entirely trustworthy, the quantitative results don’t impart any deep or actionable insights. Observations like a given version of a pre-trained model tends to show risk aversion doesn’t quite generalize.

Questions

NA

Rating

6

Confidence

4

Soundness

3

Presentation

4

Contribution

2

Limitations

NA

Summary

This paper proposes a framework to evaluation the decision making behavior of LLMs. It focuses on ChatGPT4Turbo, Claude3Opus and Gemini1pro. The results show that LLMs exhibit patterns similar to humans. Impacts of socio-demographic features are also analyzed and the results show that different LLMs can vary from each other.

Strengths

- The problem in consideration is important. The findings appear interesting.

Weaknesses

- While the problem is interesting, the technical contribution of the paper seems limited. The measures chosen are also not fully justified. It would be important to highlight the novelty of the results. - Following the above comment, the examination does not appear comprehensive. The reviewer worries that the questions may only cover a small set of problems. As a result, the conclusions drawn might contain significant randomness.

Questions

- The problem in consideration is important. The findings appear interesting. - While the problem is interesting, the technical contribution of the paper seems limited. The measures chosen are also not fully justified. It would be important to highlight the novelty of the results. - Following the above comment, the examination does not appear comprehensive. The reviewer worries that the questions may only cover a small set of problems. As a result, the conclusions drawn might contain significant randomness.

Rating

4

Confidence

3

Soundness

2

Presentation

2

Contribution

2

Limitations

Please see the above.

Reviewer - Author Discussion

Thanks everyone for their hard work on the papers, reviews, and rebuttals. We now have a comprehensive rebuttal from the authors which responds both overall and to each review. I'd please ask the reviewers to please post a comment acknowledging that they have read the response and ask any followup questions (if any). This period is to be a discussion between authors and reviewers (Aug 7 - Aug 13) so please do engage now, early in the window, so there is time for a back and forth. Thanks!

Update after authors' rebuttal

I appreciate the authors taking the time to offer clarification and answer a couple of of the questions I had. I agree that even though the authors only consider a few popular models, their methodology generalizes irrespective of whether the specific findings around a given LLM may or may not. I continue to believe that this is a nice interdisciplinary paper, and I am maintaining my acceptance decision along with the score.