A study from Mass General Brigham, published in JAMA Network Open on 13 April 2026, evaluated 21 large language models, including GPT-5, Claude, Gemini, Grok and DeepSeek, on a new benchmark for clinical reasoning called PrIME-LLM. The models reached the correct final diagnosis more than 90% of the time when handed all the relevant information in a case, but failed to produce an appropriate differential diagnosis more than 80% of the time.

PrIME-LLM scores ranged from 64% for Gemini 1.5 Flash to 78% for Grok 4 and GPT-5, with reasoning-optimised models outperforming non-reasoning ones. The distinction the authors draw is between arriving at a diagnosis when given a complete case and the harder, stepwise work that defines a clinical workup: ordering the right tests, integrating results, and narrowing a list of candidate diagnoses. The corresponding author, Marc Succi, concluded that "off-the-shelf large language models are not ready for unsupervised clinical-grade deployment," and that their value lies in augmenting rather than replacing physician reasoning.

The result qualifies the headline accuracy figures that often accompany medical-AI announcements. A model that scores well on final-diagnosis questions, where the salient findings are already laid out, may still be weak at the upstream reasoning that clinicians actually perform. The benchmark uses curated cases rather than live patients, so the gap between benchmark and bedside remains the open question; but the study makes the case for evaluating diagnostic reasoning as a process, not only as a final answer.