A medical AI simulation used diagnostic consistency to select cases for autonomous handling
A medical AI simulation used diagnostic consistency to select cases for autonomous handling
On 15 September, Nature Medicine published a study of a locally deployed diagnostic AI agent. The researchers ran each simulated clinical case five times and compared the meaning of the resulting diagnoses. With an agreement threshold of 0,90, the main configuration allowed 272 of 551 cases to proceed autonomously.
The study used MIRA-v2, a collection of simulated clinical cases based on de-identified patient records from MIMIC-IV. In each case, a physician agent questions a patient agent, requests permitted tests and investigations, and then provides a diagnosis with an explanation. The entire process ran on the institution’s local infrastructure.
In June, MIRA turned diagnosis from a written case description into a multistep consultation within a virtual medical record: the agent asked questions, ordered investigations, and chose a management approach. The new paper uses the same environment to examine which cases can be handled autonomously and which require physician review.
Language model responses involve some randomness: a small difference early in a conversation can change subsequent questions, the investigations selected, and the final diagnosis. The researchers therefore ran each of the 551 cases independently five times. They compared the meaning of the five final diagnoses and called this metric ConsistencyDx.
ConsistencyDx was better at distinguishing answers that matched the benchmark’s reference diagnosis from those that differed than the model’s internal probability estimates for its chosen words: the area under the ROC curve was 0,860 versus 0,747. In a stress test, the researchers removed the medical history from the patient agent’s prompt. Accuracy fell from 90,6% to 70,2%; ConsistencyDx also declined, while internal probability estimates and linguistic expressions of confidence remained high in some configurations.
With a ConsistencyDx threshold of at least 0,90, the simulated workflow accepted 272 cases for autonomous handling. In 269 of these cases, the diagnosis matched the benchmark’s reference label, giving an accuracy of 98,9%; the remaining 279 cases were sent for physician review. Five runs required approximately five times as many tokens, the units of text the model processes when generating a response. Diagnostic consistency thus served as the rule for routing each case to either automated handling or a physician.
On VivaBench, an external dataset of clinical case reports published in PubMed, ConsistencyDx also performed best among the metrics tested. At a threshold of 0,85, 32,0% of cases qualified for autonomous handling, with an accuracy of 89,9% in that group. The operational threshold depends on the model, generation settings, number of runs, and method used to compare the meaning of diagnoses, so it must be recalibrated for each deployment.