[Bug]: Long prompts throw an error with anthropic adapter

[GordianDziwis]

Your minimal.lua config

---@diagnostic disable: missing-fields

--NOTE: Set config path to enable the copilot adapter to work.
--It will search the follwoing paths for the for copilot token:
--  - "$CODECOMPANION_TOKEN_PATH/github-copilot/hosts.json"
--  - "$CODECOMPANION_TOKEN_PATH/github-copilot/apps.json"
vim.env["CODECOMPANION_TOKEN_PATH"] = vim.fn.expand("~/.config")

vim.env.LAZY_STDPATH = ".repro"
load(vim.fn.system("curl -s https://raw.githubusercontent.com/folke/lazy.nvim/main/bootstrap.lua"))()

-- Your CodeCompanion setup
local plugins = {
  {
    "olimorris/codecompanion.nvim",
    dependencies = {
      { "nvim-treesitter/nvim-treesitter", build = ":TSUpdate" },
      { "nvim-lua/plenary.nvim" },
      { "hrsh7th/nvim-cmp" },
      { "stevearc/dressing.nvim",          opts = {} },
      { "nvim-telescope/telescope.nvim" },
    },
    opts = {
      adapters = {
        anthropic = function()
          return require("codecompanion.adapters").extend("anthropic", {
            env = {
              api_key =
""
            },
            schema = {
              max_tokens = {
                default = 8192
              }
            }
          })
        end,
      },
      --Refer to: https://github.com/olimorris/codecompanion.nvim/blob/main/lua/codecompanion/config.lua
      strategies = {
        --NOTE: Change the adapter as required
        chat = { adapter = "anthropic" },
        inline = { adapter = "anthropic" },
      },
      opts = {
        log_level = "DEBUG",
      },
    },
  },
}

require("lazy.minit").repro({ spec = plugins })

-- Setup Tree-sitter
local ts_status, treesitter = pcall(require, "nvim-treesitter.configs")
if ts_status then
  treesitter.setup({
    ensure_installed = { "lua", "markdown", "markdown_inline", "yaml" },
    highlight = { enable = true },
  })
end

-- Setup completion
local cmp_status, cmp = pcall(require, "cmp")
if cmp_status then
  cmp.setup({
    mapping = cmp.mapping.preset.insert({
      ["<C-b>"] = cmp.mapping.scroll_docs(-4),
      ["<C-f>"] = cmp.mapping.scroll_docs(4),
      ["<C-Space>"] = cmp.mapping.complete(),
      ["<C-e>"] = cmp.mapping.abort(),
      ["<CR>"] = cmp.mapping.confirm({ select = true }),
      -- Accept currently selected item. Set `select` to `false` to only confirm explicitly selected items.
    }),
  })
end

Error messages

args = { "-sSL", "-D", "/run/user/1000/plenary_curl_69104fb2.headers", "--compressed", "-X", "POST", "-H", "Anthropic-Version: 2023-06-01", "-H", "X-Api-Key: "", "-H", "Anthropic-Beta: prompt-caching-2024-07-31", "-H", "Content-Type: application/json", "--data-raw", "{"model":"claude-3-5-sonnet-20241022","temperature":0,"system":[{"cache_control":{"type":"ephemeral"},"type":"text","text":"You are an AI programming assistant named \"CodeCompanion\".\nYou are currently plugged in to the Neovim text editor on a user's machine.\n\nYour core tasks include:\n- Answering general programming questions.\n- Explaining how the code in a Neovim buffer works.\n- Reviewing the selected code in a Neovim buffer.\n- Generating unit tests for the selected code.\n- Proposing fixes for problems in the selected code.\n- Scaffolding code for a new workspace.\n- Finding relevant code to the user's query.\n- Proposing fixes for test failures.\n- Answering questions about Neovim.\n- Running tools.\n\nYou must:\n- Follow the user's requirements carefully and to the letter.\n- Keep your answers short and impersonal, especially if the user responds with context outside of your tasks.\n- Minimize other prose.\n- Use Markdown formatting in your answers.\n- Include the programming language name at the start of the Markdown code blocks.\n- Avoid line numbers in code blocks.\n- Avoid wrapping the whole response in triple backticks.\n- Only return code that's relevant to the task at hand. You may not need to return all of the code that the user has shared.\n- Use actual line breaks instead of '\\n' in your response to begin new lines.\n- Use '\\n' only when you want a literal backslash followed by a character 'n'.\n- All non-code responses must use English.\n\nWhen given a task:\n1. Think step-by-step and describe your plan for what to build in pseudocode, written out in great detail, unless asked not to do so.\n2. Output the code in a single code block, being careful to only return relevant code.\n3. You should always generate short suggestions for the next user turns that are relevant to the conversation.\n4. You can only give one reply for each conversation turn."}],"messages":[{"content":"put =repeat(repeat('a', 80) . '\\n', 4000)\naaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\\naaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\\naaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\\naaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\\naaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\\naaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\\naaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\\naaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa

Log output

Messages:
{ {
    content = "You are an AI programming assistant named \"CodeCompanion\".\nYou are currently plugged in to the Neovim text editor on a user's machine.\n\nYour core tasks include:\n- Answering general programming questions.\n- Explaining how the code in a Neovim buffer works.\n- Reviewing the selected code in a Neovim buffer.\n- Generating unit tests for the selected code.\n- Proposing fixes for problems in the selected code.\n- Scaffolding code for a new workspace.\n- Finding relevant code to the user's query.\n- Proposing fixes for test failures.\n- Answering questions about Neovim.\n- Running tools.\n\nYou must:\n- Follow the user's requirements carefully and to the letter.\n- Keep your answers short and impersonal, especially if the user responds with context outside of your tasks.\n- Minimize other prose.\n- Use Markdown formatting in your answers.\n- Include the programming language name at the start of the Markdown code blocks.\n- Avoid line numbers in code blocks.\n- Avoid wrapping the whole response in triple backticks.\n- Only return code that's relevant to the task at hand. You may not need to return all of the code that the user has shared.\n- Use actual line breaks instead of '' in your response to begin new lines.\n- Use '' only when you want a literal backslash followed by a character 'n'.\n- All non-code responses must use English.\n\nWhen given a task:\n1. Think step-by-step and describe your plan for what to build in pseudocode, written out in great detail, unless asked not to do so.\n2. Output the code in a single code block, being careful to only return relevant code.\n3. You should always generate short suggestions for the next user turns that are relevant to the conversation.\n4. You can only give one reply for each conversation turn.",
    id = 1947269257,
    opts = {
      visible = false
    },
    role = "system"
  }, {
    content = "aaaa...",
    id = 2034632946,
    opts = {
      visible = true
    },
    role = "user"
  }, {
    content = "aaa...",
    id = 2034632946,
    opts = {
      visible = true
    },
    role = "user"
  } }
[INFO] 2024-11-13 17:01:52
Chat request started

Health check output

==============================================================================
codecompanion:                         require("codecompanion.health").check()

- Neovim version: 0.11.0
- Log file: /home/beavis/.repro/state/nvim/codecompanion.log

Plugins: ~
- OK plenary.nvim installed
- OK nvim-treesitter installed
- OK nvim-cmp installed
- OK telescope.nvim installed
- OK dressing.nvim installed

Tree-sitter parsers: ~
- OK markdown parser installed
- OK yaml parser installed

Libraries: ~
- OK curl installed

Describe the bug

1. :CodeCompanionChat
2. put =repeat(repeat('a', 80) . '', 4000)\n
3. <cr>

Now it should send the request without an error.

Reproduce the bug

No response

Final checks

• I have made sure this issue exists in the latest version of the plugin
• I have tested with the minimal.lua file from above and have shared this
• I have shared the contents of the log file

[GordianDziwis]

@olimorris you should make a note in the issue template: that the error message could leak api keys.

[olimorris]

@GordianDziwis it's in there already

[olimorris]

Can you debug this and share the exact error message? I've shared up to 30,000 tokens at a time with Anthropic and never had any issues before. If you can share the specific prompt that breaks it, that would be helpful.

[GordianDziwis]

Check the bug description. Maybe the issue is that it is a long visible promot and not a slash command or #?

[GordianDziwis]

Or can't you see the bug description, because the key was leaked?

[GitMurf]

Ok glad to hear it isn't just me! I get the following plenary error on basically any buffer over 1,000 lines when trying to add #buffer to the prompt.

I am getting the plenary failed to spawn process error. cc @olimorris (I know you asked above for more info on the error).

[GitMurf]

Oh my apologies when I saw anthropic I just assumed it was talking about Claude in general. But now I see this is for the anthropic adapter. Mine is for Copilot adapter using claude. Still may be related, but wanted to call out my misunderstanding.

[olimorris]

What operating system are you both on?

I just got back:

That appears to be a very long string of repeated 'a' characters. I can help analyze or manipulate this text if you have a specific task in mind.

Some possible next steps:
1. Would you like to count the number of characters?
2. Would you like to compress or truncate this text?
3. Would you like to perform some other text manipulation?

with put =repeat(repeat('a', 80) . '', 4000)\n.

How often has this occurred in the real world? First I've ever seen of this error. Are you on the latest plenary release?

[GitMurf]

What operating system are you both on?

Windows 11

How often has this occurred in the real world?

Not speaking to the "repeated a's" scenario, but my mentioned error I basically am getting when trying to include #buffer in any file more than 500 lines or so. So it is happening a lot.

Are you on the latest plenary release?

I did the suggestion with using master. See what lazy.nvim shows below for my setup and commit it shows.

        url:    https://github.com/nvim-lua/plenary.nvim
        branch: master
        commit: 2d9b061

[GitMurf]

As a "simple" test to make sure it kind of looks like regular code (not gigantic strings) and also no weird characters / symbols etc., I just repeated this until I started getting the error around 325 lines which is around 6,800 tokens according to an online token counter.

local dummy_foo_var = 'this is some dummy text and just trying to make the prompt longer a bit'
dummy_foo_var = 'this is some dummy text and just trying to make the prompt longer a bit'
dummy_foo_var = 'this is some dummy text and just trying to make the prompt longer a bit'
dummy_foo_var = 'this is some dummy text and just trying to make the prompt longer a bit'
dummy_foo_var = 'this is some dummy text and just trying to make the prompt longer a bit'
dummy_foo_var = 'this is some dummy text and just trying to make the prompt longer a bit'

The weird part is that it is a plenary error that keeps getting returned, not an error or rejection from the AI model api 🤔

[olimorris]

I still can't recreate this. I'll try and do some more investigating over the weekend.

[GitMurf]

I still can't recreate this. I'll try and do some more investigating over the weekend.

I have confirmed it is not just claude. I tried gpt-4o and 4o-mini and same plenary error. I am starting to think it is actually a plenary issue where too long of a request. It also errors out almost instantly so does not feel like it is a rejection from the api server.

@olimorris can you confirm what version / commit of plenary you are on? Thanks!

[olimorris]

I'm on master but also on MacOS...wondering if Windows is potentially the cause. @GordianDziwis if you could confirm it isn't a Windows issue that would be be a nice thing to rule out...

[GitMurf]

Ok @olimorris I bit the bullet and dove into plenary to get more info from the error because it was too long to show added detail other than the failed to spawn process.

• Good news is: It is NOT a Copilot / Claude context or api limit!
• Bad news is: It is what we expected with the data too big / long for plenary. But I think even more than that, it actually may be a curl issue (not super confident though).

The immediate thought I have is, how do we greatly reduce the amount of "crap" being sent in the headers. Half of the total length is likely spent on excessive damn escapes and " quotes everywhere given the JSON serialization! \\, \", \\n etc.

So many escape characters! I wonder if there is a way we can sanitize / "scrub" around that to limit it?

Without further ado... here is the error I mentioned: ENAMETOOLONG: name too long

[olimorris]

It's definitely a curl issue and I suspect it may be a curl on Windows issue. As I mentioned above, yet to ever experience this issue on macOS.

What version of curl do you have installed?

We could look at trimming the escapes but that's not working around the problem.

[GitMurf]

@olimorris your hunch was correct all along! Look here at options.verbatim on uv.spawn which is what plenary uses for curl.

https://github.com/luvit/luv/blob/master/docs.md#uvspawnpath-options-on_exit

Any idea how things could be stripped for Windows to try this verbatim option? 🤔

[olimorris]

Not that I can think of. The client/http file is relatively simple and just utilises plenary.curl.

[olimorris]

@GitMurf ... okay... this is interesting:

yetone/avante.nvim#673

Looks like it's a known Windows issue and they got around it by writing to a temporary JSON file on disk.

Could you PR this? I don't have a Windows machine to test but I think we can pretty much copy yetone/avante.nvim#673 exactly.

[olimorris]

Infact...my http implementation might be a tad confusing so I can defo do this tonight.

[GitMurf]

Looks like it's a known Windows issue and they got around it by writing to a temporary JSON file on disk.

Oh awesome! I'm actually surprised my crazy googling adventure never stumbled upon this!

Also it looks like someone hit this a couple months ago but it went stale. The exact same error / experience I am hitting: yetone/avante.nvim#622

[olimorris]

Yeah I'm surprised too. Would have thought this would have been encountered numerous times with Plenary before. Also strange that passing curl a JSON file of the body avoids the issue all together.

Anyway thanks for sharing and chipping in on this. Once of the nicer problems to have one you get a critical mass of users is juicy bugs like this 😆.

[GitMurf]

@olimorris I am a big fan of CC.nvim and trust their dev choices and just confirmed they also do the temp file concept :) so i feel good about that direction!

https://github.com/CopilotC-Nvim/CopilotChat.nvim/blob/canary/lua/CopilotChat/utils.lua#L48

[GitMurf]

Also strange that passing curl a JSON file of the body avoids the issue all together.

Yeah I thought it was strange too until I got to understanding what the initial janky windows problem is... the "name too long" error comes from the fact that windows cmd under the hood concatenates ALL of the plenary/spawn/curl args into a single gigantic string and append it to the command like a file path! This is why the error actually comes back on the PID because the process being spawned is basically a "file path" of the entire curl body 🤯🤦🏻‍♂️🤷🏻‍♂️

[GitMurf]

Infact...my http implementation might be a tad confusing so I can defo do this tonight.

@olimorris did you still want me to take a crack at a PR for this? Or are you now planning on tackling it? Thanks my friend!

[olimorris]

I'll sort it out tonight and will tag you to test if that's cool?

[GitMurf]

until I started getting the error around 325 lines which is around 6,800 tokens

Its always nice to get validation that you aren't just completely crazy banging your head against the wall 🤣

[GitMurf]

I'll sort it out tonight and will tag you to test if that's cool?

Yep. Sounds great! I'll have my eye out... I'm always working and living in GitHub anyways ;-)

[GitMurf]

cc @olimorris just to speed up your search for tonight if you want/need to look at the api for plenary curl request with a filepath for the body: <string | filepath | table>

https://github.com/nvim-lua/plenary.nvim/blob/2d9b06177a975543726ce5c73fca176cedbffe9d/lua/plenary/curl.lua#L8

[GitMurf]

Just hacked directly into plenary to send the request with a file instead of the string and went from error'ing out at about 7,000 tokens to not having a problem with 32,000+ tokens!! 🚀

[GordianDziwis]

@olimorris I am on linux.

[GitMurf]

@GordianDziwis checkout this thread (link below) I added some comments to. Do you have any emojis / icons / "non standard" utf characters in your applicable file(s) / prompts you are having issues with?

CopilotC-Nvim/CopilotChat.nvim#464 (comment)

[olimorris]

@GitMurf this should be fixed in the latest commit. Let me know your thoughts!

[olimorris]

Also that Windows limit of 8,191 characters is almost as bad as Excel still not allowing two files with the same name to be open at once 😆

[GitMurf]

@GitMurf this should be fixed in the latest commit. Let me know your thoughts!

Will go try it right now!

[GitMurf]

@olimorris the curl update to use the json file works like a charm! Just did a couple very large context tests and all is working wonderfully. Thanks for getting on it so fast! 🥳

[GordianDziwis]

@olimorris Works! Thanks!

@GitMurf I do not have the unicode issue and this soudns like something to report upstream to anthropic.

[GitMurf]

@GordianDziwis for what it's worth, someone else also confirmed the same issue on Linux and now resolved too: #420 (reply in thread)