[5H2-OS-18a] OS-18

12:00 PM - 12:15 PM JST(3:00 AM - 3:15 AM UTC)

[5H2-OS-18a-01] Functional Analysis of Multi-Head Attention in Neural Combinatorial OptimizationElucidating Roles and Cooperative Mechanisms of Individual Heads

〇Akifumi Endo¹, Noboru Murata¹ (1. Waseda University)

Keywords:

Neural Combinatorial Optimization,Explainable AI,Transformer,Multi-Head Attention

While Neural Combinatorial Optimization (NCO) demonstrates high performance, its black-box nature poses a barrier to real-world application. Particularly in Transformer-based models, detailed functional analysis of the specific roles played by each head in the core Multi-Head Attention (MHA) mechanism remains underexplored.To address this, we propose an analytical framework to elucidate the NCO decision-making process at the head level. We quantified the contribution of each head to the selection probability at each time step, conducted ablation studies, and verified consistency with heuristics.Our analysis reveals that although each head in the Attention Model extracts distinct features, their individual decision-making capabilities are limited. This suggests that for NCO to achieve high performance, a mechanism is essential wherein multiple heads with individual perspectives cooperate to select the next node in an ensemble-like manner.