Lee Yong-jin, a professor in the Department of Medicine at Dankook University, has developed a systematic evaluation framework for verifying AI virtual patient responses used in medical students' clinical training, in collaboration with Seoul National University Professor Kim Se-jung and researcher Park Jin-yong.
The team's paper was accepted to NeurIPS 2026 (Neural Information Processing Systems), one of the world's three leading machine learning conferences alongside ICML and ICLR. The findings are set to be presented at the conference's Datasets and Benchmarks Track, held in Sydney, Australia, from Dec. 6 to Dec. 13.
The research team compared virtual patient responses generated by four large language models — GPT-4o, Claude 3.5 Sonnet, Llama-3.1 70B and HyperCLOVA X — against model answers provided by medical education experts. Seventeen senior medical students evaluated the responses without knowing their source, producing a total of 7,480 assessment scores analyzed across 11 medical history-taking question categories and three language formats.
The analysis found that AI virtual patients demonstrated high factual accuracy across most questions, with responses approaching the quality of expert model answers.
However, linguistic fluency was comparatively lower in areas such as identifying a patient's chief complaint, patient education on diseases and treatment options, and clinical reasoning based on symptoms.
Fluency declined further in Korean-language negative-form questions, underscoring the need to verify the natural conversational ability of AI virtual patients separately. The team applied the evaluation framework to a web-based clinical training platform already in operation, establishing a foundation for continuously verifying AI virtual patient response quality and automated scoring criteria.
"It is meaningful that an AI virtual patient evaluation system led by medical educators has been recognized at a world-class AI conference," Lee said.
fob140@heraldcorp.com
