How to distill Llama3 to Qwen2-1.5B

[cxchen100]

Now, I want to distill Llama3 to Qwen2-1.5B,but don't know how to do
Please give me some ideas, thank you~

[cxchen100]

Now, I want to distill Llama3 to Qwen2-1.5B,but don't know how to do Please give me some ideas, thank you~

teacher model : https://huggingface.co/meta-llama/Meta-Llama-3-8B
student model: https://huggingface.co/Qwen/Qwen2-1.5B

[cxchen100]

more
Hi @yoshitomo-matsubara
thank you for your reply. It's helpful.

distillation methods? do you mean "Offline Distillation" ,yes "Offline Distillation" is my method：keep the teacher model unchanged.

may be I have no more infomation about distill. but my target is simple:
first ,I want to distill LLama3 to small llm( for example Qwen2-1.5B, other small llm is ok)
hope for run the code first, and then iterate step by step

can you give me more concrete suggestions?
can I use the torchdistll to complete this task?

[yoshitomo-matsubara]

I don't quite understand what you mean by "distill" in this context.

Offline vs. online is just one of the key components, and I need more context to understand your task.

Do you want to feed your dataset into a teacher model, generate texts, make pairs of input and teacher's output, and train student model using their outputs as hard labels given the same inputs?

If so, I'd suggest you modify some existing LM training script and save teacher's outputs, and then use the new dataset to train a student model.

[cxchen100]

I don't quite understand what you mean by "distill" in this context.

Offline vs. online is just one of the key components, and I need more context to understand your task.

Do you want to feed your dataset into a teacher model, generate texts, make pairs of input and teacher's output, and train student model using their outputs as hard labels given the same inputs?

If so, I'd suggest you modify some existing LM training script and save teacher's outputs, and then use the new dataset to train a student model.

Hi @yoshitomo-matsubara
thank you for your suggestion.

I have already been using the method you suggested: I input the output from OpenAI's larger model into LLaMA 3 8B and fine-tuned it.

However, the fine-tuned model is still too large, and I hope to distill it into a smaller model, like QWen 1.5B.

But after discussing with you this time, I started to wonder if I’m going down the wrong path. Would it be just as effective to fine-tune QWen 1.5B directly using the output from OpenAI?

Additionally, could you explain the differences between distillation data and fine-tuning data, and which is more challenging? This will help me evaluate the overall process.

[yoshitomo-matsubara]

I am asking you about low-level implementation ideas in your mind rather than high-level ideas like distill knowledge of model A to model B.

Honestly, I don't know which method is better. I do not quite understand what you meant by "distillation data and fine-tuning data" either.

[cxchen100]

I am asking you about low-level implementation ideas in your mind rather than high-level ideas like distill knowledge of model A to model B.

Honestly, I don't know if which method is better. I do not quite understand what you meant by "distillation data and fine-tuning data" either.

Sorry for confusing you. I will study first and then communicate with you~