Support for other LLMs?

[carlca]

Is this tokenizer library limited to certain models? The reason I ask is that I've been trying to add a new local LLM (codellama:7b-code) to the Zed editor, and while I can get it to go through the motions of working...

...under the surface there are signs that it's not as simple as I'd hoped...

It's been suggested to me that the problem is because the Zed project uses your tiktoken library and that the library would need updating. I'm operating at the very limits of my abilities here, so any thoughts on this would be really appreciated 😉

[noppej]

@carlca There is a PR #64 for adding Llama tokenizer support. Perhaps you can add your vote there ...