In a test using hospital records, reasoning AI identified the primary diagnosis more often on average and included more extra diagnoses than standard model versions

In a test using hospital records, reasoning AI identified the primary diagnosis more often on average and included more extra diagnoses than standard model versions

On 29 October 2026, Nature Communications published a comparison of six pairs of language models. Each pair comprised different modes of the same model or related model versions. Some worked through the task step by step; others used standard or minimal reasoning modes. They were given historical hospital records covering admission through discharge, and their answers were checked against the discharge diagnoses.

In a study published in 2025, Google's medical model AMIE helped physicians broaden their lists of possible diagnoses. The current study assessed the identification of possible diseases and the selection of final diagnoses separately. In the main task using Boston hospital records, each model used symptoms, laboratory results and imaging reports to first name ten possible diseases from a list of forty, then select the one to three diagnoses it was most confident in. The assessment measured whether the model identified the primary disease and how many extra diagnoses it retained in its final answer.

The reference diagnoses were diseases from the list of forty that appeared among the first three discharge diagnoses. Across 1157 hospital admissions, reasoning versions included the primary diagnosis in their final answer in an average of 52,8% of cases, compared with 38,7% for standard or minimally reasoning versions. Any final diagnosis outside the reference diagnoses counted as an extra diagnosis. Reasoning versions produced an average of 1,47 extra diagnoses per case, compared with 1,09.

To test the effect of providing a fixed list, the researchers asked the models to name diseases themselves using the same Boston hospital records, then compared their answers against the same reference diagnoses. On average, reasoning versions again identified the primary diagnosis more often and added more extra diagnoses. Increasing the reasoning effort of a single model, GPT-5 Mini, increased both the proportion of cases in which it identified the primary diagnosis and the number of extra diagnoses. The same combination of greater accuracy and more extra diagnoses appeared in an independent sample from intensive care units at other hospitals. In that sample, the disease list was broader and final answers were not limited to three diagnoses. The researchers tested consistency by running the models repeatedly on the same records: on average, reasoning versions more often changed the diagnosis they ranked first.

In an analysis of one error, the language model DeepSeek-R1 described pancreatitis as “less likely” and then included it in the final answer. The same answer contained both the correct primary diagnosis and an extra diagnosis.

Human selection was tested in a separate experiment: 15 physicians reviewed GPT-5 Mini's answers for 150 historical medical records and produced their own final answers, choosing from the same list of forty diseases. Physician review reduced the number of extra diagnoses by approximately 44% compared with the model's unreviewed answers. According to the participating physicians, reviewing an answer with the full reasoning took 15–20 minutes per case, while reviewing one with a brief conclusion took 5–8 minutes.