ChatGPT’s written advice exhibits higher quality and empathy, suggesting potential for a medical assistance role

A study indicates that when rated for quality and empathy, ChatGPT’s written advice outperforms some doctors in exhibiting a better ‘bedside manner’. The authors of the study suggest that this highlights the potential for AI assistants to assist in medicine by helping draft doctors’ communications with patients.

Nonetheless, some cautioned against delegating clinical responsibility, as the findings do not imply that ChatGPT is a superior doctor, and highlighted the chatbot’s tendency to generate inaccurate “facts”.

JAMA Internal Medicine published a study that utilized data from AskDocs, a Reddit forum where verified healthcare professionals answer medical queries posted by members. The study randomly selected 195 AskDocs exchanges where a verified doctor responded to a public question. The questions were then presented to ChatGPT, an AI language model, to answer. The responses were evaluated for quality and empathy by a panel of three licensed healthcare professionals who were unaware of whether the response came from a human physician or ChatGPT.

In general, the panel expressed a preference for ChatGPT’s responses over those provided by a human physician 79% of the time. The quality of ChatGPT’s responses was also rated as good or very good 79% of the time, compared to only 22% of doctors’ responses. Additionally, 45% of ChatGPT’s responses were rated empathetic or very empathetic, while only 5% of doctors’ replies received the same rating.

Dr Christopher Longhurst from UC San Diego Health believes that these results demonstrate that tools like ChatGPT can effectively generate personalized and high-quality medical advice for clinicians to review. Prof James Davenport from the University of Bath, who was not involved in the study, stated that the research does not claim that ChatGPT can replace doctors, but does suggest the need for further investigation into how ChatGPT can assist physicians in generating responses.

Some individuals pointed out that ChatGPT’s text was designed to be likable, which may have contributed to its ability to produce empathetic responses. Furthermore, ChatGPT often provided longer and more conversational answers than human doctors, which could have influenced its higher rating in the study.

However, others advised against relying on language models for factual information, as they have a tendency to generate false