Issues with LLaMA model and constraints

[irajmoradi]

Hello,

I have encountered an issue when running this code, where I am running my code on the LLaMA model.

sample(temperature=0.8)
   "A list of things not to forget when going to the sea (not travelling, and choose Bathing Suite as the first one): \n"
   "- Sunglasses \n"
   for i in range(4):
      "- [THING] \n"
from
   '/riemann/hugging'
where
   THING in set(["Volleyball Shoes", "Bathing Suite"])

The output ends up being just Bathing, and then continuing to run until it ends with just "Bathing".

[lbeurerkellner]

As mentioned in the Discord, I had a look. Seems like the HF LlamaTokenizer behaves different from other tokenizers in ‚transformers‘ when it comes to decoding tokens with leading spaces. Will need a fix. I thought this was fixed on their end, but apparently not yet. It will need some special handling from our side then. Sorry about that. Feel free to create a GH issue.

[lbeurerkellner]

This was fixed in the latest release, e.g. the following code works with the huggyllama/llama-7b model and tokenizer:

sample(temperature=1.8, chunksize=1)
   "A list of things not to forget when going to the sea (not travelling, and choose Bathing Suite as the first one): \n"
   "- Sunglasses \n"
   for i in range(40):
      "- [THING] \n"
from
   'huggyllama/llama-7b'
where
   THING in set(["Volleyball Shoes", "Bathing Suite"])

This should enable broader llama compatibility, including our recent addition of llama.cpp as an inference backend.

Note that HF advises to not use outdated llama models from the hub, as some (slow variant) tokenizer implementation still contained buggy behaviour, which can also affect LMQL.