How to install tiktoken_core on MacOs ?

[saifulapm]

Hello, I have tried to installed tiktoken in both way like this:

sudo luarocks install --lua-version 5.1 tiktoken_core
luarocks install tiktoken_core

It's install sucessfully. But in neovim, it's showing that tiktoken_core: missing.

Installing https://luarocks.org/tiktoken_core-0.2.1-1.src.rock
tiktoken_core 0.2.1-1 is already installed in /opt/homebrew
Use --force to reinstall.

How we can fix it ? Am I missing anythijng ?

CopilotChat.nvim [dependencies] ~
- OK plenary: installed
- OK copilot: copilot.lua
- WARNING tiktoken_core: missing, optional for token counting. See README for installation instructions.
- OK treesitter[markdown]: installed
- OK treesitter[diff]: installed

[deathbeam]

Check the README section for it, you basically need to tell luarocks to install it to location where neovim recognizes it, you can see from where is neovim loading lua binaries via this:

:lua print(package.cpath)

[gptlang]

btw should we change "WARNING" to "Optional"? It isn't very clear that it's not really necessary to install it and might confuse some users.

[saifulapm]

Check the README section for it, you basically need to tell luarocks to install it to location where neovim recognizes it, you can see from where is neovim loading lua binaries via this:

:lua print(package.cpath)

But how I can set path in install command ?

sudo luarocks install --lua-version 5.1 tiktoken_core —path some path here

like this ?

[saifulapm]

btw should we change "WARNING" to "Optional"? It isn't very clear that it's not really necessary to install it and might confuse some users.

This is not necessary? If I install this, any extra benefits ?

[saifulapm]

Anyway, I really love your work. That was only missing feature on neovim and that’s why I was using vscode. Now I have it so I back to neovim again.

Thank you

[deathbeam]

Yea its optional and you can skip it, it just makes token counting more accurate but we have fallback counting in place that somewhat works.

Check the README section for it, you basically need to tell luarocks to install it to location where neovim recognizes it, you can see from where is neovim loading lua binaries via this:
:lua print(package.cpath)

But how I can set path in install command ?

sudo luarocks install --lua-version 5.1 tiktoken_core —path some path here

like this ?

I think it has something like that yea, check help on luarocks, it was quite annoying to set up for me as well (luarocks is rly bad)

btw should we change "WARNING" to "Optional"? It isn't very clear that it's not really necessary to install it and might confuse some users.

I dont think neovim has OPTIONAL level, just ERROR,WARNING,OK in the healthcheck report. But it already says optional in the description so should be fine

[GitMurf]

FYI in case anyone stumbles upon this GH Discussion for MacOS when searching / troubleshooting for Windows, I have finally got luarocks / tiktoken working for Windows using PowerShell and wrote a detailed how to guide here: #386 (comment)

It may even help with MacOS issues 🤷‍♂️ or at least give you a couple ideas to try.