GLIBC_2.32 not found when running under Ubuntu 20.04.6

[kalinkrustev]

Getting this:

[Error - 1:44:15 PM] The LLM VS Code server crashed 5 times in the last 3 minutes. The server will not be restarted. See the output for more information.
/home/user/.vscode-server/extensions/huggingface.huggingface-vscode-0.1.0-linux-x64/server/llm-ls: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /home/user/.vscode-server/extensions/huggingface.huggingface-vscode-0.1.0-linux-x64/server/llm-ls)
/home/user/.vscode-server/extensions/huggingface.huggingface-vscode-0.1.0-linux-x64/server/llm-ls: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /home/user/.vscode-server/extensions/huggingface.huggingface-vscode-0.1.0-linux-x64/server/llm-ls)
/home/user/.vscode-server/extensions/huggingface.huggingface-vscode-0.1.0-linux-x64/server/llm-ls: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /home/user/.vscode-server/extensions/huggingface.huggingface-vscode-0.1.0-linux-x64/server/llm-ls)

[McPatate]

Does running the following help:

apt update && apt install libc6

?

[kalinkrustev]

Not really. It returned:

libc6 is already the newest version (2.31-0ubuntu9.9).

I am not aware how Rust works, but when searching for this error, I found this golang/go#58550
Is there an equivalent for static linking (like CGO_ENABLED=0) in Rust?

[McPatate]

Well, I guess the solution would be to build on ubuntu 20.04. I've had issues setting up the build CI and I won't be taking a look at this in the short term, I have too many things to do 😅
If someone wants to give it a go, you can play around with the workflows in .github, my inspiration came from https://github.com/rust-lang/rust-analyzer.

For now the best workaround would be to compile llm-ls yourself, which shouldn't be too hard:

1. clone the repo & cd into it
2. install rust
3. cargo build --release

You may encounter some odd compile time errors.

[kalinkrustev]

There is no need to build on Ubuntu 20.04. The build target should be x86_64-unknown-linux-musl for static linking. I did some experiments and the produced binary runs fine with this target. The bad thing is that for this target name, the build happens under Alpine and produces an Alpine binary, which overwrites the same one built on Ubuntu. Maybe some change on the binary names could help, so that musl target is released for both Alpine and Ubuntu. I am not sure if x86_64-unknown-linux-gnu has any advantage over x86_64-unknown-linux-musl.

[cmosguy]

We are dealing with this is issue on our Cent OS 8 system.

ldd --version
ldd (GNU libc) 2.17

What is the solution to this so far? We really have to compile for older version of GLIBC?

[McPatate]

What is the solution to this so far? We really have to compile for older version of GLIBC?

For the time being yes. You can configure the llm-ls' binary path in every extension, let me know if you're having issues.

[cmosguy]

@McPatate thank you for your time and reply. Can you please send me a doc so that I can compile it on our systems? I know how to run make commands, but I am unsure how to drive the compilation with the cargo systems.

Thank you!

[kalinkrustev]

We are dealing with this is issue on our Cent OS 8 system.

ldd --version
ldd (GNU libc) 2.17

What is the solution to this so far? We really have to compile for older version of GLIBC?

Check my fork, where I disabled the Alpine build and instead released the Ubuntu binary with statically linked GLIBC. Check the released asset llm-ls-x86_64-unknown-linux-musl.gz, it should be usable under older Ubuntu and others.

[McPatate]

@cmosguy let me know if @kalinkrustev's binary works on your platform.

Otherwise it should be as simple as running the command provided here: https://www.rust-lang.org/tools/install & running cargo build in llm-ls' directory.

@kalinkrustev want to submit a PR to add the musl build in the release CI? This way we can redirect people on ubuntu that have incompatible GLIBC versions to download it (which at least skips the step of having to build it).

[kalinkrustev]

I was going to submit a PR, until I found the conflict with the Alpine build. This is where I am unsure how to proceed, as I do not know the reasoning behind the naming of the assets. To resolve the conflict, perhaps a new naming should be introduced. As I mentioned - if both Alpine and Ubuntu statically linked releases are needed (the musl one), a new naming is required.

[McPatate]

You can submit your PR with a new naming, I don't see any other way around it. I'll check in the extensions what to update.

[kalinkrustev]

See #33, not so perfect solution at the moment.

[cmosguy]

@kalinkrustev I just did a cargo build --release on your current PR: #33 and it does not fix the GLIBC issue. We are using Centos8 servers, which only goes up to the GLIBC_2.17.

Am I supposed to specify something else that will make this work during the build process? Otherwise, we get these errors when I specify this binary as the exectuable:

/user/llm-ls/target/release/llm-ls: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by /user/llm-ls/target/release/llm-ls)

[kalinkrustev]

You need to define LLM_LS_TARGET=x86_64-unknown-linux-musl in the environment for the build.

[cmosguy]

Thanks @kalinkrustev -
If I run that command it barfs on the treesitter compilation:

  Compiling tree-sitter v0.20.10
The following warnings were emitted during compilation:

warning: In file included from src/./././tree_cursor.h:4:0,
warning:                  from src/././get_changed_ranges.h:8,
warning:                  from src/./get_changed_ranges.c:1,
warning:                  from src/lib.c:9:
warning: src/././././subtree.h:35:4: warning: declaration does not declare anything
warning: src/././././subtree.h:146:6: warning: declaration does not declare anything
warning: src/././././subtree.h:153:4: warning: declaration does not declare anything
warning: src/././././subtree.h: In function ‘ts_subtree_leaf_symbol’:
warning: src/././././subtree.h:251:18: error: ‘SubtreeHeapData’ has no member named ‘first_leaf’
warning: src/././././subtree.h: In function ‘ts_subtree_leaf_parse_state’:

then completely fails compilation.

[kalinkrustev]

I do not think warnings are the reason. Are you sure you use the code that is currently in the fork's main branch?

The automated compilation works fine, as you can see here. Please note that automated compilation (happening in a docker image) is the only one I tested by downloading the published binary and running it on Ubuntu 20. If you are compiling on CentOS, there is no reason to use my fork, as the purpose of it is to cross compile on Ubuntu 22 for other distributions that are with older glibc.

[cmosguy]

Yes, I was using your main branch in the fork you did.

I thought that the older versions of glibc were all that were required. What version of GLIBC are you running within this setup? I thought even though you were compiling on Ubuntu - it could still work since you were focused on older glibc versions. Does anyone understand how to setup a Centos 8 target in the github workflow yaml?

[kalinkrustev]

I am not sure why you need a new target. Did you try using the binary released here: https://github.com/kalinkrustev/llm-ls/releases/download/main/llm-ls-x86_64-unknown-linux-musl.gz ?

[cmosguy]

Thanks @kalinkrustev ,

I did not realize this was a "release" I thought I had to compile your fork.

Anyways, I took your file and unpacked it, then went into my VS code settings and specified:

"llm.lsp.binaryPath": "/user/llm-ls-x86_64-unknown-linux-musl",

Then reloaded VS code and got an error:

thread 'main' panicked at crates/llm-ls/src/main.rs:684:47:
failed to find home dir: ()
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
[Error - 5:32:36 AM] Server initialization failed.
  Message: Pending response rejected since connection got disposed
  Code: -32097 
[Info  - 5:32:36 AM] Connection to server got closed. Server will restart.
true
[Error - 5:32:36 AM] LLM VS Code client: couldn't create connection to server.
  Message: Pending response rejected since connection got disposed
  Code: -32097 
thread 'main' panicked at crates/llm-ls/src/main.rs:684:47:
failed to find home dir: ()
  Code: -32097 
[Error - 5:32:36 AM] Client LLM VS Code: connection to server is erroring. Shutting down server.
[Error - 5:32:36 AM] Stopping server failed
Error: Client is not running and can't be stopped. It's current state is: startFailed

Is there something else I should be doing in my setup? It simply does not work on Centos 8 GLIBC v2.17.

[cmosguy]

Oh, and by the way, when I execute this by itself it does nothing:

/user/llm-ls/llm-ls-x86_64-unknown-linux-musl

Also, the original binary version that was installed with the extension on my system is gnu version, not musl

[kalinkrustev]

When you execute it manually, It is waiting for some input (likely JSON-RPC). You can type some random text and see it respond with a parse error. This will mean that the GLIBC error is resolved.
As for the home dir error - I guess checking the source code will explain it.

[McPatate]

cf https://docs.rs/home/latest/home/fn.home_dir.html#unix for the homedir error.

Try setting $HOME, it should fix.

[cmosguy]

cf https://docs.rs/home/latest/home/fn.home_dir.html#unix for the homedir error.

Try setting $HOME, it should fix.
Thanks @McPatate but I am doing a remote-ssh into the machine to run this extension. It should have be able to see the $HOME environment variable. There is something wrong when I specify hardcoded location of this: "llm.lsp.binaryPath": "/user/llm-ls-x86_64-unknown-linux-musl", it does not work.

[Error - 9:48:37 AM] Client LLM VS Code: connection to server is erroring. Shutting down server.
[Error - 9:48:37 AM] Stopping server failed
Error: Client is not running and can't be stopped. It's current state is: starting
	at LanguageClient.shutdown (/user/.vscode-server-insiders/extensions/huggingface.huggingface-vscode-0.1.5-linux-x64/node_modules/vscode-languageclient/lib/common/client.js:914:19)
	at LanguageClient.stop (/user/.vscode-server-insiders/extensions/huggingface.huggingface-vscode-0.1.5-linux-x64/node_modules/vscode-languageclient/lib/common/client.js:885:21)
	at LanguageClient.stop (/user/.vscode-server-insiders/extensions/huggingface.huggingface-vscode-0.1.5-linux-x64/node_modules/vscode-languageclient/lib/node/main.js:150:22)
	at LanguageClient.handleConnectionError (/user/.vscode-server-insiders/extensions/huggingface.huggingface-vscode-0.1.5-linux-x64/node_modules/vscode-languageclient/lib/common/client.js:1146:18)
	at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
[Error - 9:48:37 AM] Server initialization failed.
  Message: write EPIPE
  Code: -32099 
[Error - 9:48:37 AM] LLM VS Code client: couldn't create connection to server.
  Message: write EPIPE
  Code: -32099 
[Info  - 9:48:37 AM] Connection to server got closed. Server will restart.
true
thread 'main' panicked at crates/llm-ls/src/main.rs:684:47:
failed to find home dir: ()
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

[cmosguy]

When you execute it manually, It is waiting for some input (likely JSON-RPC). You can type some random text and see it respond with a parse error. This will mean that the GLIBC error is resolved. As for the home dir error - I guess checking the source code will explain it.

@kalinkrustev so you are correct, if I type something it does not error out.

./llm-ls-x86_64-unknown-linux-musl 
this is a test
Content-Length: 75

{"jsonrpc":"2.0","error":{"code":-32700,"message":"Parse error"},"id":null}

I guess I am just stuck on understanding why the extension does not see the $HOME dir sitation

[McPatate]

@cmosguy what client are you using? VSCode (and all other IDEs for that matter) launches llm-ls as a subprocess IIUC, so you'll need to make sure it feeds the env values at that moment.
You could always set the HOME env var in /etc/environment to be sure it's available to every process.

[cmosguy]

I am using VS code and the llm-vscode extension. There is something wrong with how that extension sets the home directory when using an binary outside the extension hierarchy.

[McPatate]

llm-ls uses the same library as cargo to determine what the HOME directory is, so I doubt there's something wrong with that. llm-ls needs the HOME of the user to write some data like logs to disk.
From the docs:

Returns the path of the current user's home directory if known.

# Unix

Returns the value of the `HOME` environment variable if it is set
and not equal to the empty string. Otherwise, it tries to determine the
home directory by invoking the `getpwuid_r` function on the UID of the
current user.

# Windows

Returns the value of the `USERPROFILE` environment variable if it
is set and not equal to the empty string. If both do not exist,
[`SHGetFolderPathW`](https://docs.microsoft.com/en-us/windows/win32/api/shlobj_core/nf-shlobj_core-shgetfolderpathw) is used to return the appropriate path.

llm-vscode does not set any kind of env var, it just reads them.

[spew]

I am not able to get the musl version of llm-ls-x86_64-unknown-linux-musl released in 0.4 to run on ubuntu:20.04. It complains about missing libraries like libgcc_s.so.1 and libstdc++.so.6. I am able to get the one put out by @kalinkrustev in #25 (comment) to work though. So something is different (still reviewing his change to have an idea about what it is).

[TanmayeeSamal]

I have also the same problem. How to resolve it ?

[tirelibirefe]

Ubuntu 20.04; not working.