[Not a bug] Support for fine-tuning codestral-mamba?

[ciscodoojung]

Python Version

n/a

Pip Freeze

n/a

Reproduction Steps

n/a

Expected Behavior

n/a

Additional Context

Hi, first of all, thank you all for your help.

I have been trying to use this code (mistral-finetune) to fine-tune the codestral-mamba model (https://huggingface.co/mistralai/Mamba-Codestral-7B-v0.1) but it seems like the code is not expecting mamba architecture. Is there a plan to support to fine-tune codestral-mamba in this repo? or are there any other tools we can use to fine-tune codestral-mamba?

Your response is greatly appreciated! Thank you!

Suggested Solutions

No response

[acodercat]

+1

[ciscodoojung]

Just to update the thread (not related to this repo).

Huggingface transformers v4.44.0 was released yesterday to support codestral-mamba training from HF.
Release: https://github.com/huggingface/transformers/releases/tag/v4.44.0
PR: huggingface/transformers#32080

Hope this helps! Thank you.