How are people using this plugin?

[gsuuon]

Are you sticking to the starters or have you built your own prompts?

[tiberiuichim]

Just getting started, trying this: https://github.com/JoseConseco/nvim_config/blob/master/lua/nv_llm-nvim/init.lua

[tiberiuichim]

@gsuuon So, I've been giving this plugin a good try, I think I now have more knowledge for a feedback.

My use case is that I've been fiddling with AI generators (stabledifussion comfyui, text-generation-webui, etc) so I would like to use my own llama.cpp local server. I think many people may be in the same camp, with not a lot of experience but a desire to explore what's possible.

Setting up was pretty rough:

• I'm an old timer, I might know a bit of vimscript, but not lua. I feel that things could be better documented, but I'm used to reading the source code and finding my way around
• after many reboots trying to get the proper combination of cuda-drivers with cuda-toolkit and nvidia drivers, I managed to compile llama.cpp with CUBLAS support
• I've tested with local llama and codellama-13b-instruct.Q5_K_S.gguf. The basic llamacpp prompt works. Also the "modify" prompt from my previous linked repo, but the quality of code generated is not that great, in my example.
• I've tried to make the "commit" prompt work, but, although I see the server getting the prompt instructions, it doesn't insert anything in my nvim buffer
• llama.cpp has recently gained starcoder compatibility, so it would be interesting to see a prompt example that makes use of it

[gsuuon]

Thanks for the feedback! I think the documentation is lacking too - I try to add mentions of new features as I add them but it's organized very sub-optimally. As it is, poking around the source code and type annotations is the best way to find out how to use certain things. The plugin is built assuming the user will have lua LSP setup, so I haven't fully documented many things outside of type annotations. #28 is tracking documentation improvements, I'd appreciate any specific feedback you have to add there.

I've tested with local llama and codellama-13b-instruct.Q5_K_S.gguf. The basic llamacpp prompt works. Also the "modify" prompt from my previous linked repo, but the quality of code generated is not that great, in my example.

I haven't been able to get good completions out of the quantized smaller models either. Part of the goal of this project is to encourage folks to figure out and share how to get some of these local models to be useful. I mostly just use OpenAI at the moment - I think the 34b may be useful but I can't run that locally at a reasonable speed.

I've tried to make the "commit" prompt work, but, although I see the server getting the prompt instructions, it doesn't insert anything in my nvim buffer

Can you show me the prompt you're using? I wonder if this is a bug.

[tiberiuichim]

Can you show me the prompt you're using? I wonder if this is a bug.

Here's my prompt, although right now it's commented. Notice the codellama provider, although I've tested it with the llamacpp provider, too.

https://github.com/tiberiuichim/yadm-dotfiles/blob/ba2f73640279fe3d3fe134aff471d7e2f5e67bb3/.config/nvim/init.vim#L631-L650

As said previously, the request was performed in the llamacpp server, I had it set to verbose logging. I've also tested the input for that request separately (in the browser) and it performed ok, I got a valid response.

[gsuuon]

I think this is another docs problem - the codellama provider is meant for fill-in-the-middle on non-instruction trained models, I think it's in the comments but maybe not highlighted in the docs. I wouldn't expect instruction prompts to work great with that provider. For llama.cpp you'll need to adapt the prompt params (check the link to llamacpp server docs), so instead of messages.[1].content the prompt just goes in prompt:

    commit = {
      provider = llamacpp,
      mode = llm.mode.INSERT,
      builder = function()
        local git_diff = vim.fn.system {'git', 'diff', '--staged'}

        if not git_diff:match('^diff') then
          vim.notify(vim.fn.system('git status'), vim.log.levels.ERROR)
          return
        end

        return {
          prompt = 'Write a terse commit message according to the Conventional Commits specification. Try to stay below 80 characters total. Staged git diff: ```\n' .. git_diff .. '\n```'
        }
      end,
    },

When I do that I see that the server gets the request but ends immediately (eval time is 0.01ms):

print_timings: prompt eval time =    4744.92 ms /   526 tokens (    9.02 ms per token,   110.86 tokens per second)
print_timings:        eval time =       0.01 ms /     1 runs   (    0.01 ms per token, 125000.00 tokens per second)
print_timings:       total time =    4744.93 ms
slot 0 released (528 tokens in cache)
{"timestamp":1699308545,"level":"INFO","function":"log_server_request","line":2217,"message":"request","remote_addr":"127.0.0.1","remote_port":59319,"status":200,"method":"POST","path":"/completion","params":{"{\"stream\":true,\"prompt\":\"Write
a terse commit message according to the Conventional Commits specification. Try to stay below 80 characters total. Staged git diff: ```\\ndiff --git a\\/lua\\/llm\\/prompts\\/starters.lua b\\/lua\\/llm\\/prompts\\/starters.lua\\nindex 72c7d13..3ffaa38 100644\\n--- a\\/lua\\/llm\\/prompts\\/starters.lua\\n    b\\/lua\\/llm\\/prompts\\/starters.lua\\n@@ -25,7  25,7 @@
local function standard_code(input, context)\\n   local fewshot":"llm.mode.INSERT,\\n\\n```\"}"}}

I'm thinking the issue is with the git diff output not working well with llama.cpp server or however the curl command is being formatted. I'll keep taking a look, but I think this brings up another edge case I should handle where the request succeeds but the server responds with nothing or only whitespace. Again, thanks for the feedback!

[tiberiuichim]

@gsuuon np, and many thanks for your work! I'll return to this plugin in a few weeks, when I have time to play around with the tech