Ways to reproduce this approach

[Yuyz0112]

Hi @loubnabnl, thanks for this great repo.

I've seen a blog from the VMware OCTO, which described their works on fine-tuning star-coder, but modified the code provided by the [SantaCoder](https://github.com/loubnabnl/santacoder-finetuning) git repository for fine-tuning as it is focused on the code generation task..

There are some more details like:

1. Accelerate and DeepSpeed are used to improve fine-tuning performance.
2. Fine-tuning generates a small PEFT model .

I think this is not the best place to discuss their approach, but since you are the expert on fine-tuning santacoder/star-coder, are there any hints we can reproduce the approach in the blog on top of the current open-source code? I also checked the star-coder fine-tuning repo, but it looks like it suggests using instruction-based fine-tuning.

[loubnabnl]

Hi, you can use the StarCoder repo for using Peft or DeepSpeed you just need to change how you build the dataset samples e.g the prepare_sample_text can just be your code file instead of a question and answer like it's done for instruction tuning, same for the deespeed code.

[Yuyz0112]

@loubnabnl Thank you for the help! I've started following your suggestion. BTW, could you give some hints on the hardware requirement for fine-tuning the starcoder? Issues in the starcoder repo seem not to have a clear answer. I have 4 A16 GPUs for fine-tuning.