Risks from Language Models for Automated Mental Healthcare: Ethics and Structure for Implementation

Summary

The paper presents some experiments to see whether LLM are ready to be used in TAIMH (Task-Autonomous AI in Mental Healthcare) systems.

Rating

7

Confidence

2

Ethics flag

1

Reasons to accept

Most of the paper is devoted to define what is a TAIMH system and this part is interesting to read and offers several references on this topic. The experiments are based in a series of questionnaires fed into several LLMs to analyze their output. The evaluation has been performed manually labeling it by at least two psychiatrists independently. The evaluation has been performed over quite a lot of systems.

Reasons to reject

The improvements of the systems are based on improving the systems prompts but no fine-tuning of the systems have been performed.

Questions to authors

Would it be possible to test some fine tuning of the systems, and if so, how can it be performed?

Summary

The authors have examined the readiness of language models' for providing mental healthcare - prompting 10 LLMs using prompts/ vignettes developed and evalauted with psychiatrists.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

I appreciate actionable references (such as NICE) that can be useful to CS x Mental Health community for designing relevant capabilities. The findings emphasize the need to examine language models for behavioral health outcomes and safety of vulnerable users.

Reasons to reject

Please see Questions to Authors

Questions to authors

1. Pg 5 - Please define "legibility" and "corrigibility" 2. Sec 5.2 - It is interesting that Approach-1 and 2 did not improve response for mania. Unsafe responses also increased on adding TAIMH values for psychosis. Could you elaborate on how responses changed (or, remain unchanged) compared to the responses obtained from original prompts? What ethical principles are violated? It might be useful to discuss what a safe/ethical response looks like. 3. Pg 2 - There is no previous work on how AI may augment the mental healthcare system... In the Discussion, the authors mention "first of its kind framework" (Fig 1) - similar suggestions have been made in the past. Please see below works: Stade, Elizabeth C., et al. "Large language models could change the future of behavioral healthcare: a proposal for responsible development and evaluation." npj Mental Health Research 3.1 (2024): 12. Stade, E. C., Jane P. Kim, and Shannon Wiltsey Stirman. "Readiness Evaluation for AI Deployment and Implementation for Mental Health: A Review and Framework." (2024).

Summary

In this paper, the authors propose a framework to evaluate whether a given TAIMH (task-autonomous AI in mental health care) is suited to be deployed in psychiatric emergencies and follows the ethical requirements needed in this health care area. Thus, the authors proposed a list of requirements for any TAIMH, including imperatives like ‘discourage and prevent harm to users and others’. Ten large language models were tested in five psychiatric emergencies covering depression, self-harm, psychosis, mania, suicidal and homicidal thoughts. The tests were guided by psychiatrists. Results show that no tested model is ready for TAIMH. Then the authors applied modifications in the prompts given to the LLM in order to test if the LLM would give safer answers; this last test was done only on psychosis and mania and only one family of LLM. The results of this last test show that other methodologies are needed to prepare LLM for TAIMH. The work is relevant to the current state of use of LLM for health care. The paper is well-written and easy to understand.

Rating

7

Confidence

3

Ethics flag

1

Reasons to accept

* The paper is relevant given the impact of large language models being used in mental health, with real consequences of undesired use of this technology. * An ethical framework and a way to evaluate how safe the LLM's output is for direct use by people in psychiatric emergencies is needed. This paper provides the first steps towards both aspects. * The design of the test is done by health professionals, and so it is the evaluation of answers.

Reasons to reject

* It is not clear to me how others can reproduce the results. The LLM used is not described in detail.

Summary

The paper proposes a definition and framework for Task-Autonomous AI in Mental Health (TAIMH), focusing on ethical considerations with the potential in augmenting mental healthcare systems with varying levels of autonomy and intervention The study evaluates ten SOTA language models using questionnaires designed to assess their ability to detect and manage basic psychiatric symptoms. The study focuses on the Llama-2 family of models while its focus within the questionnaires lies on those instances where nearly all models generated unsafe responses, particularly in the areas of Psychosis and Mania. Psychiatrists conducted the evaluations and found that none of the models matched human professionals in nuanced understanding and contextual appreciation. Most models displayed issues such as overly cautious or sycophantic responses and lacked necessary safeguards, raising concerns about potential harm if accessed by users in mental health crises. The paper concludes by suggesting improvements for existing models to enhance user protection and facilitate future TAIMH applications.

Rating

6

Confidence

5

Ethics flag

1

Reasons to accept

A very useful evaluation of the use of AI for mental healthcare systems The use of human experts

Reasons to reject

The study should consider models which have been tailored to mental health such as https://arxiv.org/pdf/2309.13567. Thus, the conclusions need to be re-assessed.

Questions to authors

The models you have used are not domain adapted for mental health; useful to reconsider your evaluation results.

Response

The authors have addressed all my questions and recommendations. I am happy to see this paper accepted to the conference