Training scripts and other utilities for NousResearch's finetuning subnet/competition
See the comments in the training.py file for more information on some of the custom options I was using. Feel free to @ me for questions in the discord server.
I may come back to this later to rewrite a few things that I think I could have done cleaner (especially remerging/reverting), or adding an alternate Trainer for LORA training. I have also considered doing a write up on all the things I tried and papers I read, as well as ideas that I hand't implemented yet, but other than that this code is provided as is, and the requirements.txt is more of a guideline as pytorch needs to be tailored to cuda versions and some requirements may change as the competition progresses.
If you want to see some awful (or great depending on perspective) spaghetti code, the intial commit of this is exactly what my folder looked like when I was still working on the competition, before this refactoring.