Artificial intelligence (AI) like ChatGPT can compose essays or emails, write computer code, even generate art, but how well can it manage complex patient care? A team that included Norton Healthcare researchers decided to find out.

They compared ChatGPT’s recommendations for managing an 81-year-old patient with nonresolving pneumonia with those of a pulmonary specialist.

“What we found was the recommendations made by ChatGPT, in a fairly complex case, were both appropriate and consistent with the pulmonary consultation,” said Norton Healthcare infectious diseases specialist Julio A. Ramirez, M.D., FACP, a co-author of the study.

The case study was published in a recent edition of Norton Healthcare Medical Journal. Other co-authors included Norton Infectious Diseases Institute’s Stephen Furmanek, MPH; William Mattingly, MPH; and Thomas Chandler, MPH. Dr. Ramirez is chief scientific officer at Norton Infectious Diseases Institute and editor-in-chief of Norton Healthcare Medical Journal.

AI in health care is growing rapidly and has the potential to assist in diagnosis and treatment, especially in complex cases.

An AI tool such as ChatGPT has the ability to synthesize large amounts of data faster than humans. It potentially could help with day-to-day decision-making, prevent variability in patient care and provide support during shifts on weekends, nights, and holidays.

“Because this technology is new, we still need to determine best practices for implementing AI in health care decision-making. Our study was a step in that direction,” Dr. Ramirez said.

The American Medical Association (AMA) issued principles for AI development in November 2023 supporting governance policies that prioritize development of artificial intelligence in a way that is ethical, equitable, responsible and transparent.  Specifically for generative AI, which creates new content, the AMA advises health care organizations adopt policies that anticipate and minimize risks prior to the adoption and use of a health AI system.

The study focused on a patient hospitalized with a diagnosis of pneumonia and atrial fibrillation. The patient received intravenous antibiotics and medication for atrial fibrillation, but his fever and shortness of breath persisted, and the pneumonia progressed.

On his ninth day in the hospital, he received noninvasive ventilation in the intensive care unit. On day 15, a CT scan showed further progression of the pneumonia.

At this point, the health care team obtained a pulmonary consultation. The advice given by the pulmonologist was compared with the recommendations made by ChatGPT in January 2023.

The AI algorithm’s recommendations were scored as highly appropriate when the response was accurate and specific to the case, moderately appropriate when it was medically accurate but not specific to the case, and inappropriate when the answer was incorrect or potentially harmful. The clinical patient data was presented to the AI application on the same day as the pulmonary consult.

Of the 31 treatment plan recommendations made by ChatGPT, 27 were considered highly appropriate. Four were deemed moderately appropriate. None were considered inappropriate. The findings were consistent with a similar study looking at AI in health care focused on recommendations for the prevention of cardiovascular disease.

For example, among the AI technology’s initial recommendations were to reevaluate the diagnosis and consider additional testing to rule out infections, such as fungal or mycobacterial. This was considered highly appropriate.

Based on the high consistency between ChatGPT and the consulting pulmonologist in patient management, the study concluded that “AI could become a useful tool to support clinical decision-making.”

Because the interface between health care AI such as ChatGPT allows for dialogue and feedback between medical professionals and the algorithm, it allows a clinician to hone the discussion by adding relevant details. In this way, it mirrors the kind of interaction a treating physician might have with peers, according to the study.