Presentation Information
[2J1-GS-10a-01]CDSS as an AI agent v2: Implementing Metacognition in LLM via Uncertainty Quantification of Predictive AI for Differential diagnosis support
〇Yasuhiko Miyachi2, Osamu Ishii2, Torigoe Keijiro1,2 (1. Torigoe Clinic, Ibara, Okayama, Japan, 2. The Society for Computer-aided Clinical Decision Support System)
Keywords:
Clinical decision support system,Large language model,Metacognition in AI,Uncertainty quantification,Conformal prediction
Predictive AI-based Clinical Decision Support Systems (CDSS) often lack robustness against out-of-distribution cases, leading to potential diagnostic errors. Conversely, LLMs possess vast medical knowledge but tend to follow the inappropriate context provided by external systems uncritically. To address these challenges, we propose "CDSS as an AI Agent," a novel architecture that implements metacognition in LLMs.
In this system, CDSS predictions are used as context for the LLM. Crucially, we employ Conformal Prediction to quantify the CDSS's uncertainty as a Nonconformity Score (NS). This score is provided to the LLM as meta-information. When the NS indicates high uncertainty, it serves as a trigger, prompting the LLM to engage in critical appraisal and self-reflection rather than accepting the CDSS output unconditionally.
Evaluations on difficult-to-diagnose cases demonstrated that the proposed agent successfully identified low-reliability contexts. Consequently, the proposed method significantly improved both diagnostic accuracy and clinical validity compared to conventional approaches.
By implementing pseudo-metacognition, this research mitigates confirmation bias and enables a reliable system that collaborates effectively with medical professionals.
In this system, CDSS predictions are used as context for the LLM. Crucially, we employ Conformal Prediction to quantify the CDSS's uncertainty as a Nonconformity Score (NS). This score is provided to the LLM as meta-information. When the NS indicates high uncertainty, it serves as a trigger, prompting the LLM to engage in critical appraisal and self-reflection rather than accepting the CDSS output unconditionally.
Evaluations on difficult-to-diagnose cases demonstrated that the proposed agent successfully identified low-reliability contexts. Consequently, the proposed method significantly improved both diagnostic accuracy and clinical validity compared to conventional approaches.
By implementing pseudo-metacognition, this research mitigates confirmation bias and enables a reliable system that collaborates effectively with medical professionals.