Microsoft's AI Medical System: High Accuracy Claims, But Real-World Tests Are Pending

Microsoft has announced a medical AI diagnostic system that, in a controlled simulation, correctly identified complex medical cases more than twice as often as human generalist physicians. According to a study published on Thursday in the journal Science, the system, named MAI-DxO, achieved an 83% accuracy rate on difficult cases, while primary care doctors and general internists working individually were correct 36% of the time. The company's AI chief, Mustafa Suleyman, described the result as a significant step toward what he terms medical superintelligence.

The core claim is that Microsoft's system, which uses a coordinated panel of multiple AI agents to question, test, and debate a diagnosis, significantly outperforms individual human doctors on challenging medical problems. The simulation drew from complex cases published in the New England Journal of Medicine, a source of diagnostic puzzles typically reserved for specialists or teams of physicians. However, this comparison did not include those specialists or teams, who were not part of the study's testing group.

The results also come with significant caveats, particularly regarding cost and real-world application. Microsoft's own data indicates that much of the performance gain comes from the strength of the underlying AI models it uses, such as OpenAI's GPT-5 and GPT-5.5, rather than solely from its own orchestration system. Furthermore, the study's data showed that Microsoft's system did not improve performance or reduce costs for cases from emergency departments, a setting where it did not show an advantage over using a standard AI model on its own.

Questions about the system's practical deployment and cost-effectiveness remain. For example, when using GPT-5, Microsoft's system increased accuracy on complex cases by only about four points, but reduced the estimated cost of processing each case by approximately $3,000. On emergency department cases, however, there was little improvement in either accuracy or cost when comparing GPT-5 with and without the assistance of Microsoft's system. For weaker AI models, the system provided a larger boost in accuracy, but the gap between weaker and stronger models remained significant.

The publication of this study places Microsoft alongside Google and OpenAI, which have also claimed that their AI systems can match or exceed human doctors on certain medical tasks. In a related development, Google published a clinical study with Beth Israel in The Lancet this month to evaluate its conversational diagnostic AI in real-world urgent care visits. Microsoft is also moving toward real-world testing through a new partnership with Beth Israel Deaconess Medical Center in Boston to evaluate the clinical reasoning capabilities of its system in hospital settings.

As part of this evaluation, patients will receive standard treatment while also engaging with Microsoft's AI system for those who consent. Microsoft and its partners will then compare the AI's recommendations against those of the human doctors. Microsoft's own study in Science was released alongside a document outlining its vision for medical superintelligence, which it defines as the capacity to surpass the strongest human-led health systems in medical reasoning and coordination.

Mustafa Suleyman, who previously co-founded DeepMind and now leads Microsoft's AI unit, has a long history of working on AI for medical applications. He stated that the company is working with the Mayo Clinic to train a healthcare foundation model for release next year and expects healthcare models to improve an order of magnitude in capability within the next 24 to 36 months. He also said Microsoft is working with partners to develop medical AI capabilities and that the company is committed to accelerating its introduction to the world.