April 21, 2010 - A research group led by Dr. Hirotaka Takita and Associate Professor Daiju Ueda at the Osaka Metropolitan University Graduate School of Medicine recently published a systematic review and meta-analysis that provides an in-depth assessment of generative artificial intelligence (AI) in diagnosing medical conditions and comparing their performance to that of physicians.
The team screened a total of 18,371 studies and identified 83 for detailed analysis. These studies involved a variety of generative AI models, including GPT-4, Llama3 70B, Gemini 1.5 Pro, and Claude 3 Sonnet, covering a wide range of medical domains. Of these, GPT-4 was the most studied model. The results show thatThe average diagnostic accuracy of these AI models was 52.1%(951 TP3T CONFIDENCE INTERVAL: 47.01 TP3T – 57.11 TP3T)。The diagnostic accuracy of some models was comparable to that of non-expert physicians, with no statistically significant difference between the two(Accuracy difference: 0.6% [95% confidence interval: -14.5% to 15.7%], p=0.93). However.Expert doctors still outperform AI15,8% (95% confidence interval: 4.4% – 27.1%, p=0.007). Nevertheless, as technology progresses, this gap may be gradually reduced。
The study also found that AI performance was more consistent across most medical specialties, with two exceptions: dermatology and urology. In dermatology, AI performed better, likely because the field involves pattern recognition, which is AI's strength. However, dermatology also requires complex reasoning and patient-specific decision-making, so AI's strengths don't fully reflect its real-world value in this area. For urology, the findings are based on only one large study, so the generalizability of the conclusions is somewhat limited.
"This study shows that generative AI has diagnostic capabilities comparable to those of non-expert physicians. It can be used for medical education, to support non-expert physicians, and to assist diagnosis in areas with limited medical resources." Dr. Hirotaka Takita said, "Future research needs to evaluate in more complex clinical scenarios, use actual medical records for performance evaluation, improve the transparency of AI decision-making, and validate in different patient populations to further substantiate the capabilities of AI."
1AI notes that in addition to the diagnostic field, the study also highlights the potential of generative AI in medical education. The researchers noted, "Current generative AI models perform comparably to physicians in non-expert environments, which provides an opportunity to integrate AI into medical training." AI can be used to simulate real cases to help medical students and trainees learn and assess their skills.
However, the study also raises concerns about the transparency and bias of these models. Many AI systems do not disclose details of their training data, which raises questions about the applicability of their results to all populations. The researchers emphasized that "transparency ensures understanding of model knowledge, context, and limitations" and stressed the need to develop clear, ethical, and well-validated AI applications.
Currently, despite its great potential, generative AI still faces challenges in complex cases involving detailed patient information. Do doctors need to worry about losing their jobs? It's hard to say for sure, but in the field of diagnostics, it could happen.