Can Advanced Artificial Intelligence Outsmart Human Medical Licensing Exams
A head-to-head test of 11,400 licensing questions reveals surprising similarities between reasoning and standard models.
Artificial intelligence systems can now ace professional medical licensing exams, but a head-to-head comparison reveals that advanced reasoning models might not hold a practical advantage over standard variants when it comes to text-based medical knowledge. Researchers evaluated two flagship versions of an open-source artificial intelligence platform, examining how they handled nearly two decades of licensing questions.
The study, published in PLOS One, analyzed the performance of the DeepSeek-R1-0528 and DeepSeek-V3-0324 models on 11,400 multiple-choice questions from the Chinese National Medical Licensing Examination spanning the years 2004 to 2022. The examination serves as an industry admittance standard to evaluate whether candidates possess the essential professional knowledge and skills required to practice clinical medicine.
How Two Different Artificial Intelligence Architectures Stacked Up Against 19 Years of Licensing Exams
The research team put the artificial intelligence models through rigorous testing by uploading PDF files containing 150 multiple-choice questions for each examination unit. To prevent context leakage, independent researchers used separate computer systems to conduct fresh chat sessions for each model. The DeepSeek-R1 model relies on a multistage learning framework optimized for deep thinking and complex reasoning, whereas the DeepSeek-V3 model serves as a standard instruction-following system.
Despite their architectural differences, both models achieved remarkably similar scores across the 19-year dataset. The annual average accuracy rate for DeepSeek-R1-0528 reached 91.2 percent, while DeepSeek-V3-0324 achieved 91.1 percent. Statistical analysis confirmed no significant difference in overall performance between the two systems, and both comfortably surpassed the 60 percent passing threshold required for human examinees.
Why Did Advanced Reasoning Fail to Produce Higher Medical Scores?
While the reasoning-optimized model has previously demonstrated superior performance on mathematical benchmarks like the American Invitational Mathematics Examination, that advantage vanished in the medical testing. The authors suggest that this near-identical performance stems from a shared foundation of medical guidelines, textbooks, and research papers present in the training data of both models. Because the test relied entirely on text-based multiple-choice questions, rote knowledge retention played a major role in achieving high scores.
The investigation also uncovered a high degree of overlap in the errors made by both systems. Out of roughly one thousand incorrect answers generated by each model, over 70 percent were shared mistakes. Detailed error analysis revealed that both systems occasionally struggled with complex clinical decision-making and the interpretation of textual imaging findings, even when utilizing accurate underlying medical knowledge.
What Temporal Trends Reveal About Changing Medical Standards
When tracking performance from 2004 to 2022, the researchers identified a general downward trend in accuracy rates over time, particularly after 2014. Statistical regression analysis marked 2014 as a critical cut-off point, coinciding with major national examination reforms in China. These reforms shifted the focus of licensing questions away from pure memorization toward the integration of knowledge with complex, authentic clinical practice.
The authors noted several important limitations in their work. The evaluation was restricted exclusively to text-based multiple-choice questions in clinical medicine, omitting multimodal data such as radiographs, electrocardiograms, and practical skills. Furthermore, the web-based testing interface did not allow researchers to control background computational parameters like output temperature.
While the findings point to strong potential for artificial intelligence as a supplementary tool in medical education and knowledge retrieval, the authors emphasize that high licensing exam scores do not fully guarantee safe clinical performance. Future research will explore how these models handle practical skills and multimodal clinical reasoning.
The research was published in PLOS One on November 19, 2026.
This article has been fact checked for accuracy, with information verified against reputable sources. Learn more about us and our editorial process.
Last reviewed on .
Article history
- Latest version
Reference(s)
- Liao, Qian-Qian., et al. “Comparative performance of DeepSeek-R1 and DeepSeek-V3 on China’s National Medical Licensing Examination questions (2004–2022).” PLOS One, vol. 21, no. 11, November 19, 2026, pp. e0361855 Public Library of Science (PLoS), doi: 10.1371/journal.pone.0361855. <https://doi.org/10.1371/journal.pone.0361855>.
Cite this page:
- Posted by Joseph Jackson