Medical AI systems are often evaluated after receiving every fact in a case. Real consultations unfold differently. A clinician begins with a few symptoms, asks questions, examines the patient, orders a test and changes course as evidence arrives. The study published in Science evaluates this sequential process.

A structured diagnostic path

Microsoft’s MAI-DxO coordinates simulated clinical roles that propose diagnoses, challenge one another and select the next useful question or test. SDBench includes 304 complex cases from New England Journal of Medicine conferences and 325 emergency-department cases from MIMIC-IV-ED.

MAI-DxO reached about 83% accuracy on the complex cases, compared with 36% for generalists using references and 22% without them. In emergency scenarios it reached about 80%, versus 47% for physicians. A cost-constrained setting retained close to 81% accuracy with a lower estimated testing bill.

MAI-DxO reached about 83% accuracy on the complex cases, compared with 36% for generalists using references and 22% without them. In emergency scenarios it reached about 80%, versus 47% for physicians. A cost-constrained setting retained close to 81% accuracy with a lower estimated testing bill.

Strong numbers in a narrow setting

Those numbers do not prove superiority over a hospital team. The cases were text-based, the comparison involved individual generalists rather than specialists or multidisciplinary groups, and costs were modelled from a price schedule. The Microsoft-employed authors explicitly state that the results do not establish superiority in real clinical practice.

Those numbers do not prove superiority over a hospital team. The cases were text-based, the comparison involved individual generalists rather than specialists or multidisciplinary groups, and costs were modelled from a price schedule. The Microsoft-employed authors explicitly state that the results do not establish superiority in real clinical practice.

The next test involves patients

A prospective study with Beth Israel Deaconess Medical Center will compare the system with independent physician assessments. Safety, bias, workflow, errors and the clinician’s ability to challenge a recommendation will matter more than a headline score.

MAI-DxO therefore represents a useful evaluation advance: it tests the route to a diagnosis, not only the final answer. Clinical value will depend on real-world trials and accountable human oversight.

Conclusion

MAI-DxO therefore represents a useful evaluation advance: it tests the route to a diagnosis, not only the final answer. Clinical value will depend on real-world trials and accountable human oversight.