[5K1-OS-48] OS-48

10:15 AM - 10:30 AM JST(1:15 AM - 1:30 AM UTC)

[5K1-OS-48-06] Evaluating Clinical Expertise of Large Language ModelsA Comparative Analysis of Clinical Role-Play Sessions Using ChatGPT

〇Chihiro Hatanaka¹, Hisae Konakawa¹, Masataka Nakayama¹, Yuka Suzuki¹, Mikiko Yamaguchi¹, Ryusei Adachi¹ (1. Kyoto University)

Keywords:

LLM-based Mental Health Support,Replication of Expertise using Large Language Models,Safety and Ethics in AI Counseling,Clinical Discourse,Role-play

This study evaluates the clinical expertise of Large Language Models (LLMs) in counseling through a comparative analysis of role-play sessions using a common scenario of a university student’s concerns. Dialogue data from clinical psychology graduate students and ChatGPT (under various prompts) were quantified using categories adapted from Suzuki (2017) and compared with human-to-human sessions.Results revealed distinct LLM characteristics: providing comprehensive information and over-structuralizing sessions—such as prompting clients to select emotions from options to reach efficient conclusions. Furthermore, LLMs exhibited excessive affirmation, leaving little space for negative content. While offering verbal assurances for silence, they failed to maintain sufficient "pauses" for free exploration, tending instead to rush the dialogue.These findings suggest that while LLMs serve as objective information providers, they struggle to construct a "containing" space where experiences can be symbolized and integrated into a narrative. This presentation discusses how LLM output characteristics impact clinical discourse quality and explores prospects for replicating professional expertise.