Presentation Information
[A-11-12]Validity Analysis of Freezing Existing Ranks in LoRA Rank Growing
◎△Kazuki Kanaya1, Daisuke Kondo2, Koita Takahiro2, Kohiga Akihito1 (1. Doshisha Univ., 2. Graduate School of Doshisha Univ.)
Keywords:
Deep Learning,Fine-Tuning,LoRA,Rank Growing
LoRA is a parameter-efficient fine-tuning method that adds low-rank parameters to a model and trains only them. Since the LoRA rank affects expressiveness, rank-growing methods gradually add ranks during training. However, if added-rank updates change existing-rank gradients, freezing them may not always be appropriate. This study analyzes the validity of freezing existing ranks in LoRA rank growing. We define a stage as a training interval in which one rank is added. HVP interaction is the first-order approximate effect of added-rank updates on existing-rank gradients, and gradient drift is the actual gradient change before and after a stage. In SST-2 experiments with RoBERTa, we increased the LoRA rank from 1 to 5 and compared frozen and readjusted designs. The frozen design showed a positive correlation between HVP interaction and gradient drift, suggesting that added-rank training affects frozen existing ranks. The readjusted design showed no clear correlation, suggesting that these changes may be absorbed by readjustment. The final Test Acc was 93.6±0.4% for frozen and 94.0±0.4% for readjusted. These results suggest that rank-growing designs should consider rank interactions.