The test LlamaIntegrationTest::test_conversion test is failing

[ydshieh]

The following command

RUN_SLOW=1 python3 -m pytest -v tests/models/llama/test_tokenization_llama.py::LlamaIntegrationTest::test_conversion

gives

>           self.assertEqual(old_serialized, new_serialized)
E           AssertionError: '{\n [1465 chars]   "Sequence": {\n          "id": "B",\n      [1794589 chars]}\n}' != '{\n [1465 chars]   "SpecialToken": {\n          "id": "<s>",\n[1794837 chars]}\n}'

tests/models/llama/test_tokenization_llama.py:337: AssertionError

Who can help?

@ArthurZucker

[Narsil]

I looked into it.

The difference is that the newly converted tokenizer has ids 32000-32004 as special ids which correspond if I'm not mistaken to OpenAssistant llama fork.

Those do not seem to be declared here: https://huggingface.co/hf-internal-testing/llama-tokenizer/tree/main

I'm not sure which part of the code adds them to the slow tokenizer, but this seems indeed like a bug.

Looked at the wrong file. Everything works it's only a different type_id in the post processor.

We simply need to update the tokenizer.json on the hub with the correct value (1)

[Narsil]

(There's also a slight issue with the EOS token being added into the processor for no reason.

[Narsil]

https://huggingface.co/hf-internal-testing/llama-tokenizer/discussions/3

Goes along with

#23400

[ydshieh]

Confirmed it works!