[4L4-GS-1a] Fundamental AI, theory algorithm

2:45 PM - 3:00 PM JST(5:45 AM - 6:00 AM UTC)

[4L4-GS-1a-06] A Study on Gradient Manipulation for Mitigating Gradient Conflict Caused by Auxiliary Losses

〇Kaito Ohashi¹, Tatsuhito Hasegawa¹ (1. Univ. of Fukui)

Keywords:

Gradient Conflict,Deep Learning,Auxiliary Classifier,Image Recognition

In deep learning, attaching auxiliary classifiers to intermediate layers helps alleviate the vanishing gradient problem and provides regularization effects, thereby contributing to training stabilization and improved accuracy. However, the level of feature abstraction required by auxiliary classifiers differs from that of the final classifier, which means their gradient directions do not always align. In this study, we propose a method that adaptively selects channels for utilizing auxiliary gradients based on the norm of the main gradient and removes interference components within those selected channels. Experiments using CIFAR-100 confirmed a 1.2% accuracy improvement compared to the baseline due to channel selection. On the other hand, the effect of interference removal was limited, and it was shown that increasing the proportion of channels where it is applied led to a degradation in accuracy.