Major llama.cpp Change Notes

[martindevans]

llama.cpp 3328 just got merged. This contains a major change to the API which will require some big changes to LLamaSharp to support.

• Make a release before this change, with current API
• Make basic changes to low level API
• Adapt current executors to work with new API (minimal changes)
  • I think we're maybe not going to do this, since there's general agreement the executors need completely replacing anyway.
• Work out how to expose new capabilities (entirely new executor)

[AshD]

I ran into this issue when I wanted to try the new mistral model.
I updated the llama.cpp dlls and it failed the token generation with current main branch llamasharp branch.

It's tough to keep up with Llama.cpp breaking changes and all the new models coming in daily :-)

[martindevans]

That's odd, I actually tried out Mistral-7B-Instruct earlier today using 0.5.1 and it worked perfectly.

[AshD]

Thanks Martin. I used the current Llamasharp main branch with the Cuda 11 dll in runtime folder and it worked!

I thought Mistral required the new llama dlls and when I had tried it with yesterday's Llama.cpp dlls, it threw the tokenization error.

Thanks again,
Ash

[martindevans]

(Batched executor is still in progress)