Vicuna Models checkpoints transfer script

[sineeli]

Successfully converted checkpoints from vicuna(torch) to Keras3 compatible, please let me know if any refactoring needed.

Thanks

[sineeli]

cc: @mattdangerw

[mattdangerw]

Looks good! A couple comments...

Next up, you could try uploading these to your individual user on Kaggle, and making a PR that updates our presets file here -> https://github.com/keras-team/keras-nlp/blob/master/keras_nlp/src/models/llama/llama_presets.py

That would give us all a way to test the vicuna models end to end, then we can copy them to the Keras org on Kaggle when they look good.

Thanks!

[mattdangerw]

let's do this like the phi3 script

keras-nlp/tools/checkpoint_conversion/convert_phi3_checkpoints.py

Line 29 in 38c6608

from keras_nlp import upload_preset # noqa: E402

That will still allow people to run this who do not have access to the keras kaggle org.

[mattdangerw]

Is the weight conversion all the same as llama 2? If so could we consider consolidating the conversion scripts?

[sineeli]

Yes the weights are same llam2 architecture, we can merge with existing script. I will try that. Thanks!

[sineeli]

@mattdangerw

When we run on cpu and use .numpy() at the end of the weights it causes an error: Bfloat16 scalar not surpported but when we see phi3 script there is no such usage and at global level the backend set to torch and he is loading the weights directly.

LLam2 all weights are in torch so and we can use the same approach as phi3 weights convertion script.

phi3 seems more fault tolerant

When tested on CPU from bfloat16 to bfloat16:

But when used float16(default of hugging face weights) to float32(keras model) we will not face this mismatch.

Thanks

[mattdangerw]

When we run on cpu and use .numpy() at the end of the weights it causes an error: Bfloat16 scalar not surpported but when we see phi3 script there is no such usage and at global level the backend set to torch and he is loading the weights directly.

@sineeli I think keras.ops.convert_to_numpy(x) would gracefully handle bfloat16, maybe try that?

With what precision are the original pytorch checkpoints stored on disk? If they are at float16, we could just do the same (and store at float16 on disk). The disk format does not mean we need to load at that format.

Anyway, as soon as you push with the comments addressed above we can merge this PR. And keep working on the actual checkpoints we ship.

[mattdangerw]

@sineeli can you make your kaggle model public? I'll pull in the script but leave the new preset off for now, we can do that on a follow up PR.

[sineeli]

@sineeli can you make your kaggle model public? I'll pull in the script but leave the new preset off for now, we can do that on a follow up PR.

Sure, waiting for page update. Thank!

https://www.kaggle.com/models/sineeli/vicuna/keras/vicuna_1.5_7b_en