Llama 2 tokenizer: apparition of the token id 29871

[piegu]

System Info

transformers==4.31.0
meta-llama/Llama-2-7b-hf

Who can help?

@ArthurZucker

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

# from https://github.com/philschmid/sagemaker-huggingface-llama-2-samples/blob/master/training/sagemaker-notebook.ipynb
!pip install "transformers==4.31.0"

YOUR_TOKEN = "hf_xxxxx"
!huggingface-cli login --token $YOUR_TOKEN

from transformers import AutoTokenizer

# get tokenizer
model_id = "meta-llama/Llama-2-7b-hf" 
tokenizer = AutoTokenizer.from_pretrained(model_id)
tokenizer.pad_token = tokenizer.eos_token

instruction = "### Instruction\nRepeat the context."
context = "\n\n### Context\n```\nI like Paris.\n```"
answer = "\n\n### Answer\nI like Paris."

prompt = instruction + context + answer

tokens_prompt = tokenizer(prompt, return_tensors="pt").input_ids[0]

tokens_instruction = tokenizer(instruction, return_tensors="pt").input_ids[0]
tokens_context = tokenizer(context, return_tensors="pt").input_ids[0]
tokens_answer = tokenizer(answer, return_tensors="pt").input_ids[0]

# from each number of tokens, we take out 1 token that this the <s> token
# then, we sum all tokens and expect get the same number of tokens that ones of the prompt 
num_tokens_of_sum = (len(tokens_instruction) - 1) + (len(tokens_context) - 1) + (len(tokens_answer) - 1)

# we compare the number of tokens of the prompt (without the <s> token) with the sum claculated before
print((len(tokens_prompt) - 1) - num_tokens_of_sum)

# we get -2, which is wrong: it means there are 2 tokens in the sum of tokens that were not in the prompt

tokens_prompt
# tensor([    1,   835,  2799,  4080,    13,  1123, 11666,   278,  3030, 29889,
#            13,    13,  2277, 29937, 15228,    13, 28956,    13, 29902,   763,
#         3681, 29889,    13, 28956,    13,    13,  2277, 29937,   673,    13,
#        29902,   763,  3681, 29889])

tokens_instruction
# tensor([    1,   835,  2799,  4080,    13,  1123, 11666,   278,  3030, 29889])

tokens_context
# tensor([    1, 29871,    13,    13,  2277, 29937, 15228,    13, 28956,    13,
#       29902,   763,  3681, 29889,    13, 28956])

tokens_answer
# tensor([    1, 29871,    13,    13,  2277, 29937,   673,    13, 29902,   763,
#         3681, 29889])

# we can see that the token 29871 appears 2 times (tokens_context and tokens_answer) but it should not!

Expected behavior

We hope that tokenizing each part of the prompt will lead to the same result as tokenizing the prompt. But not.

Why does token ID 29871 appear?

[ArthurZucker]

Hey, could you try this on transformers == 4.33? Pretty sure the fixes to Llama have been merged

[piegu]

Same (wrong) result with transformers == 4.33.

[ArthurZucker]

Okay, this is actually expected, 29871 is the SPIECE_UNDERLINE token. If you encode each prompt individually, you are adding an underline to the prompt, then adding the special token. If you encode everything concatenated, you add the prefix token to the first token only.

[oir]

Hello! Same issue here. @ArthurZucker, can you clarify your comment? What is SPIECE_UNDERLINE?

you are adding an underline to the prompt

Do you mean tokenizer() call does this automatically (if @piegu is explicitly doing this, I've missed it)? If so, I would at least expect add_special_tokens==False to fix this but it does not:

>>> tkzr = AutoTokenizer.from_pretrained("./llama-2")
>>> tkzr.encode("\nhi")
[1, 29871, 13, 2918]
>>> tkzr.encode("\nhi", add_special_tokens=False)
[29871, 13, 2918]
>>> tkzr.decode([29871, 13, 2918])
'\nhi'
>>> tkzr.decode([13, 2918])
'\nhi'
>>> tkzr.decode([29871])
''

[ArthurZucker]

No the argument you might be looking for is add_prefix_space which we did not include for llama. The SPIECE_UNDERLINE is the prefix space added by sentencepiece. We can't de activate this easily but can add more support for this

[oir]

Got it, I'm following this now. Thanks!

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.