A RECENT study has suggested that generative AI tools, like ChatGPT, could potentially reduce the time burden of electronic health record (EHR) management for urologists. The project, led by Michael Scott, Stanford University School of Medicine, California, USA found that ChatGPT was able to generate responses deemed acceptable for nearly half of the patient messages reviewed, indicating the potential utility of AI in handling routine patient enquiries.
The study involved the collection of 100 patient messages from a men’s health clinic, covering topics including clinical decision-making, treatment plans, postoperative concerns, symptoms, and test results. These messages were put into ChatGPT 3.5, which generated responses that were subsequently evaluated by five urologists. The AI-generated responses were assessed for accuracy, completeness, potential harm, helpfulness, and intelligibility using a 5-point Likert scale.
The reviewers also indicated whether each response was suitable for patient communication. Their findings revealed that 47% of the responses, generated by ChatGPT, were deemed acceptable for patient communication. The performance of the AI was significantly better with simpler questions, with 56% of responses to these questions considered acceptable, compared to 34% for more complex enquiries.
The results also included specific metrics showing that the AI-generated responses scored well for accuracy (average 4.0) and intelligibility (average 4.7), however, were slightly lower in completeness (average 3.9) and helpfulness (average 3.5). A noteworthy result from the study was that minimal risk of harmful content was found across all types of questions.
These results suggest that generative AI tools like ChatGPT could be effectively integrated into EHR systems to handle routine patient communications, potentially improving clinical efficiency and freeing up healthcare providers to focus on more complex tasks. However, the study also highlighted several limitations and risks associated with using AI in healthcare including the fact that ChatGPT’s responses are based on data it has been trained on, which may include unintentional biases.
Katie Wright, EMJ
Reference
Scott M et al. Assessing artificial intelligence-generated responses to urology patient in-basket messages. Urol Pract. 2024;11(5):793-8.
