[5Yin-A] Poster 8

[5Yin-A-47] Abstain-aware Reinforcement Learning for Reasoning Large Language Models

〇Masaya Tsunokake¹, Hikaru Tomonari¹, Ryoichi Takase¹, Koki Takeshita¹, Masaharu Ukeda¹, Yasuhiro Sogawa¹ (1. Hitachi, Ltd.)

Keywords:

LLM,reinforcement learning,Abstain-aware model,faithfullness,unanswerable questions

In recent years, Large Language Models designed for reasoning (reasoning models), which generate long chains of thought, have demonstrated effectiveness in solving complex problems requiring step-by-step reasoning. These models are typically constructed by applying post-training methods, such as reinforcement learning, to pre-trained LLMs. Through this process, they acquire behaviors such as hypothetical thinking and self-reflection.However, this behavior poses a challenge: the models tend to over-thinking even on unanswerable questions due to the missing premises, leading to hallucinations. One solution to address this is to employ reinforcement learning that trains the model to refuse to answer unanswerable questions (Abstain-aware RL), thereby encouraging the self-learning of a more accurate reasoning process.In this paper, we investigate the effectiveness of Refusal RL as an additional post-training stage for reasoning models. Experimental results demonstrate that this approach significantly improves the abstain rate for unanswerable questions while maintaining the model's original reasoning capabilities to a certain extent.