Summary
This paper prompts LLMs with 25 multiple choice questions (on basic demographic information, education attainment, healthcare coverage, disability status, family status, veteran status, employment status, and income) from the 2019 ACS. The authors use eight kinds of prompts which vary in additional context, instructions, and asking in the second person. However, each time, the next-token probabilities are used to determine the immediate reply by the LLM to the multiple choice question. To evaluate the responses and the differences between LLM and human generated responses, the authors compute the normalized entropy and use KL divergence. They find that "smaller" LLMs are vulnerable to ordering and labeling biases, and that after correcting for these through randomized answer ordering, the LLMs trend towards uniform distributions in their responses (~high entropy). Instruction-tuning seems to increase the variance in the entropy measure for LLMs, but nonetheless the entropy remains higher overall compared to the human generated responses. The authors state that the main takeaway from their paper questions the popular methodology of using survey responses from LLMs using multiple choice questions. They challenge prior work and give the explanation that models consistently appear to better represent subgroups whose aggregate statistics are closest to uniform.
Weaknesses
While I find the paper enjoyable to read and it challenges important earlier findings on LLMs, I am not convinced that the current experiment fully supports the claims:
- The 2019 ACS uses stratified sampling. Therefore, I believe that the variance is already being increases to obtain a representative sample. Since the additional context given to the LLM is very limited + the draws are independent, it seems like a very hard task to generate a matching distribution by the LLM. This is less important when assessing, for example, the political view of an LLM.
- First-token probabilities may be a biased measure to obtain the replies (see Wang et al., 2024).
- Additionally to the first-token probabilities, more advanced prompting techniques, such as Chain-of-Thought (Wei et al., 2022a), could improve the coherence and dependencies in the responses, especially for the sequential generation.
- I believe that the sensitivity to ordering and labeling biases is known (Wei et al. (2022b) and Wei et al. (2023)).
Overall, the authors show that independent draws from an LLM with limited context generates a uniform distribution, which questions earlier findings made used by such as methodology. While I agree with the authors on that statement, I believe that their experiment adds little value to further support their claim on "better represent subgroups whose aggregate statistics are closest to uniform." I believe additional, more fine-grained analysis is required for this.
References
-------------
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., ... & Zhou, D. (2022a). Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35, 24824-24837.
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., ... & Fedus, W. (2022b). Emergent abilities of large language models. arXiv preprint arXiv:2206.07682.
Wei, J., Wei, J., Tay, Y., Tran, D., Webson, A., Lu, Y., ... & Ma, T. (2023). Larger language models do in-context learning differently. arXiv preprint arXiv:2303.03846.
Wang, X., Ma, B., Hu, C., Weber-Genzel, L., Röttger, P., Kreuter, F., ... & Plank, B. (2024). "My Answer is C": First-Token Probabilities Do Not Match Text Answers in Instruction-Tuned Language Models. arXiv preprint arXiv:2402.14499.