[Unofficial] GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection

WIP Unofficial implementation of GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection

Roadmap

• layer-wise training tricks
• sample training loop
• add training logs on toy data
• train on real* data

Reference

@article{zhao2024galore,
  title   = {GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection},
  author  = {Jiawei Zhao and Zhenyu Zhang and Beidi Chen and Zhangyang Wang and Anima Anandkumar and Yuandong Tian},
  year    = {2024},
  journal = {arXiv preprint arXiv: 2403.03507}
}