Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug: convert-hf-to-gguf.py fails for Gemma models #7897

Closed
maab19 opened this issue Jun 12, 2024 · 1 comment · Fixed by #8597
Closed

Bug: convert-hf-to-gguf.py fails for Gemma models #7897

maab19 opened this issue Jun 12, 2024 · 1 comment · Fixed by #8597
Labels
bug Something isn't working low severity Used to report low severity bugs in llama.cpp (e.g. cosmetic issues, non critical UI glitches)

Comments

@maab19
Copy link

maab19 commented Jun 12, 2024

What happened?

When running the convert-hf-to-gguf.py script for the gemma-1.1-2b-it model I get the following error I added to the relevant log output field.

For reproduction of the error, run the script for any Gemma model e.g.:

$ python convert-hf-to-gguf.py /path/to/model/dir/gemma-1.1-2b-it 

I already figured out what the problem is: in set_vocab() of the GemmaModel class special_vocab.add_to_gguf() is called twice, once at the beginning of the method inside of self._set_vocab_sentencepiece() and then again at the end of set_vocab(). Because of this, the chat template is added twice to the GGUFWriter which raises an exception in the add_key_value() method of GGUFWriter (in gguf_writer.py) at the second call as 'tokenizer.chat_template' is already present in kv_data and add_key_value() contains the following check:

if key in self.kv_data:
            raise ValueError(f'Duplicated key name {key!r}')

My own quick fix was to remove this check, but I am not sure if this is the proper fix or if set_vocab() of the GemmaModel class should be adjusted, so that special_vocab.add_to_gguf() is called only once.

Name and Version

$ python convert-hf-to-gguf.py

What operating system are you seeing the problem on?

Linux

Relevant log output

Traceback (most recent call last):
  File "/home/max/git/llama.cpp/convert-hf-to-gguf.py", line 2878, in <module>
    main()
  File "/home/max/git/llama.cpp/convert-hf-to-gguf.py", line 2863, in main
    model_instance.set_vocab()
  File "/home/max/git/llama.cpp/convert-hf-to-gguf.py", line 2247, in set_vocab
    special_vocab.add_to_gguf(self.gguf_writer)
  File "/home/max/git/llama.cpp/gguf-py/gguf/vocab.py", line 73, in add_to_gguf
    gw.add_chat_template(self.chat_template)
  File "/home/max/git/llama.cpp/gguf-py/gguf/gguf_writer.py", line 565, in add_chat_template
    self.add_string(Keys.Tokenizer.CHAT_TEMPLATE, value)
  File "/home/max/git/llama.cpp/gguf-py/gguf/gguf_writer.py", line 206, in add_string
    self.add_key_value(key, val, GGUFValueType.STRING)
  File "/home/max/git/llama.cpp/gguf-py/gguf/gguf_writer.py", line 166, in add_key_value
    raise ValueError(f'Duplicated key name {key!r}')
ValueError: Duplicated key name 'tokenizer.chat_template'
@maab19 maab19 added bug-unconfirmed low severity Used to report low severity bugs in llama.cpp (e.g. cosmetic issues, non critical UI glitches) labels Jun 12, 2024
@github-actions github-actions bot added the stale label Jul 13, 2024
@TobiasKlapper
Copy link

I have the same problem when trying to quantize my own finetuned CodeLlama 7B

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working low severity Used to report low severity bugs in llama.cpp (e.g. cosmetic issues, non critical UI glitches)
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants