学习率至关重要：Vanilla LoRA 可能已足够
本文通过大规模超参数搜索，系统地重新评估了 Vanilla LoRA 以及九个代表性 LoRA 变体，发现不同 LoRA 方法偏好的学习率区间各异，并将最优学习率区间的差异归因于最大 Hessian 特征值的变化，与经典学习理论相吻合。This work systematically re-evaluate nine representative LoRA variants alongside vanilla LoRA through extensive hyperparameter searches, finding that different LoRA methods favor distinct learning rate ranges and attributes the differing optimal learning rate ranges to variations in the largest Hessian eigenvalue, aligning with classical learning theories.