If we do not know where it keeps its brain, do we trust it?

The cross-sectional survey conducted and published by Parikh et al.,[1] analyzed the perceptions of healthcare workers and other professionals toward the chatbot, ChatGPT. The authors aimed to compare the perceptions of healthcare professionals with non-healthcare professionals regarding the pre-trained transformer. As a very early foray into understanding the implications of this still-evolving technology, the study represents a commendable effort. The study was conducted as an anonymous online survey, which comprised nine multiple-choice questions. These questions aimed at understanding the level of awareness among the study participants. The results of this study showed that the percentage of non-healthcare professionals who had already used ChatGPT was significantly higher than the healthcare professionals. However, among healthcare professionals, the percentage of respondents who expected ChatGPT to have a positive impact on their profession was higher (52.2% vs 37.7%), and more healthcare professionals planned to make changes in their professional life due to the influence of the chatbot (20.4% vs 11.3%). The study by Parikh et al.[1] has been conducted at a very nascent stage of the chatbot coming into the public domain, despite which, there was significant awareness among the respondents. The influence of deep learning models is only expected to grow in the foreseeable future among both the general population and the healthcare fraternity. The study conducted by Parikh et al.[1] did not have predefined inclusion and exclusion criteria. This, along with the seemingly arbitrary nature of some of the questions used in the survey, was alluded to in the accompanying editorial by Hammond and Pearce.[2] The ability of the study participants to understand the nuances of the technology and, thus, accurately predict its possible impact on their profession was not assessed, thus the significance of the findings pertaining to the level of change expected by the study participants is questionable. Artificial intelligence has had a growing influence on medicine over the years.[3] William B Schwartz, commented in the 1970s, that computers would, by the year 2000, act as a powerful extension of the physician’s intellect. There were several early attempts at incorporating computation in medicine, through “rule-based approaches” and pattern recognition systems, which fared poorly.[4] Lee et al.,[5] in an analysis of the benefits and limitations of the Generative Pre-Trained Transformer 4 (GPT-4) as a chatbot in medicine, looked at the various possible roles of the chatbot, including medical note-taking and a possible role in “curbside” medical consultation. While commenting that the GPT-4 chatbot is undoubtedly a powerful tool, there exist several important limitations. These limitations include the sensitivity of the chatbot to the choice of wording of the prompt, along with the occurrence of false responses, referred to as “hallucinations.”[6] Bender et al. have argued that there is a need for a deeper understanding of the limitations of language models (LMs). LMs are trained in prediction tasks, do not perform natural language understanding (NLU), and hence require almost an unfathomable amount of training data to make the current conversational model of chatbots possible. The concept of “documentation debt,” which would be the result of large, uncurated data sets that go into the training of the chatbots, would have far-reaching implications in their application in medicine, where the veracity of responses is of utmost importance. Another drawback pointed out by the authors was the propensity of the LM to exhibit various kinds of biases, which would be detrimental in medical applications.[7] In conclusion, while the study by Parikh et al.[1] is a commendable early foray into the understanding of awareness regarding the chatbot among the healthcare fraternity, exploring the full range of capabilities of the said technology is currently an ongoing process, thus limiting the possibility of predicting the degree of its impact on healthcare. However, the conversation started by the authors has possibly far-reaching implications, and increasing awareness of this constantly evolving technology is vital; there needs to be constant discourse with the aim of best adapting machine learning and artificial intelligence (AI) in healthcare. Financial support and sponsorship Nil. Conflicts of interest There are no conflicts of interest.