-
Notifications
You must be signed in to change notification settings - Fork 261
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
linux-x86: Most git operations fail with "failed to stat <some file>" #3844
Comments
This happens in all repositories, even when I do a fresh |
It does NOT happen in a repo that isn't colocated. Only noticing it now because I default to colocated repos. |
Confirming I'm seeing this in my collocated repos as well with the upgrade to 0.18.0 EDIT: I can't even fresh clone a repo actually, it's not even a matter of colocation.
|
I tired to reproduce this briefly on a Mac, but haven't been able to yet. I can also try Linux later. So, I'm totally guessing in the dark. It might be worth double-checking that it's not some sort of a temporary problem with the /home filesystem. Do the files it's complaining about exist? Would rebooting the computer make a difference? Do plain It might also be some sort of incompatibility between a libgit2 and/or gix version and/or some git version. Which version of Git do you use? |
It would also be very helpful if one if you can build jj from source and bisect the v0.17.0..v0.18.0 range. |
Seems like underlying libgit2 code paths are totally broken. It's weird that fstat returned EAGAIN. Running with Btw, which Linux distribution and version are you using? |
I'm running Fedora 40. The thing is that my personal computer doesn't have this issue, only my work computer. (Same setup for all intents and purposes. Obv. not the exact same hardware.) I'll find some time later today to go to the office and do the bisect. |
Well, installing from source works perfectly fine 😕 I'll try to reproduce the CI build more closely, starting with using musl instead of glibc. |
This is quite strange. I built the |
Have you been using If so, another remote possibility (reaching at straws here) is that perhaps the binary uses some CPU instruction that your processor doesn't support (got optimized for Intel-only instructions or AMD-only or ??). |
Also, the release binaries are built with Rust 1.76. I'd be surprised if this made a difference... |
I used My home pc has an amd cpu, the office one is intel (11900k I think?). I'm using the latest rust version (1.78). |
If you installed Rust with rustup, you could relatively easily install 1.76 toolchain ( I guess it's possible that the binary doesn't work correctly on Intel CPUs. We can ask on Discord, that's probably the easiest way to find out. |
Shoot, looks like I made a mistake. Apparantly I had a self-compiled version of jj installed on my personal machine all along. Now that I installed the precompiled binary here as well, I can at least reproduce at home. So Intel / AMD cpu doesn't seem to be the issue. I'll try to compile myself with Rust 1.76 in a minute. |
Compiling with 1.76 also doesn't cause any issues, jj works fine. I had another idea that it might be the feature flags. I'm pretty sure I had them set just like in the CI workflow when testing early at the office. But I'll try again. oppenssl always gives me trouble with Rust builds 😢 |
Nope, the features don't make a difference either. Still at square one: Local builds are fine, prebuilt ones are borked. |
What the... if I download the prebuilt binary manually, it seems to be fine as well. Only |
With the following command, the correct binary seems to be installed:
Notice the snippet of the output (github.com). This is different without the explicit target:
So, QuickInstall. It looks like they also provide prebuilt releases for some rust programs. Presumably, the borked binaries are downloaded from this release: https://github.com/cargo-bins/cargo-quickinstall/releases/tag/jj-cli-0.18.0 |
I have no idea how the QuickInstall system works. But it seems they only provide binaries for x86_64-unknown-linux-gnu (the one we don't provide). Would there be any downside to providing gnu and musl binaries for linux? It would probably be even easier than adding aarch64 binaries. It's a little redundant since musl works fine everywhere, but it's an easy solution. I'll investigate the QuickInstall system a bit to maybe figure out what went wrong there. |
I tried "pinning" this issue and putting the workaround in the first comment. |
Great! The flag |
The quick-install version is built using glibc 2.17 to maintain backwards compatibility with centos 7, perhaps that's the reason why it failed? |
I could:
to workaround this issue |
@NobodyXu good questions, thank you for looking into it! Unfortunately, I don't know about glibc. Re enabling gix, this will not get rid of the libgit2 dependency. Pushing and pulling is only implemented with libgit2 in |
Not sure about pushing, but pulling using gix is definitely doable. |
The binary downloaded from https://github.com/cargo-bins/cargo-quickinstall/releases/tag/jj-cli-0.18.0 appears to link some of the libc functions statically. For example,
( I think that's the source of the problem. |
Quickinstall uses cargo-zigbuild to cross compile to glibc 2.17, perhaps that's the souce of the error? |
Like @senekor I've confirmed that if I install via |
Two questions:
I ask because it would be nice of us to mitigate this for our users ASAP, so they don't run into it. We can/should also update our docs I suppose in the meantime, but I think it's important to stop the bleeding, so to speak. In either case, I would prefer if |
I believe so. Of course, then it would be us to make a glibc-linked binary that works on many linux systems, which I don't have much experience with, but you might.
AFAIU, there's no way currently, but this is now tracked in cargo-bins/cargo-binstall#1721. |
A possible workaround is to lie and make a copy of https://github.com/martinvonz/jj/releases/download/v0.18.0/jj-v0.18.0-x86_64-unknown-linux-musl.tar.gz named https://github.com/martinvonz/jj/releases/download/v0.18.0/jj-v0.18.0-x86_64-unknown-linux-gnu.tar.gz. I'd rather not, though. |
Yes.
You can use |
I think the question is whether we can do this from the project's side. Asking every user to do this manually isn't reliable. I'm personally so used to binstall working well that I often don't check a projects installation documentation and just binstall it. I do wonder: Can't you disable building jj and delete the broken release on the quickinstall side? Is it setup in a way that this is impossible or difficult? |
We actually have recommended the correct command in https://martinvonz.github.io/jj/latest/install-and-setup/#cargo-binstall for a long time, but I expect |
It's tracked in cargo-bins/cargo-binstall#1721 , though I'm quite busy recently.
Yes, I can remove the broken release, though it might be built again. |
Now that this PR is merged, the problem should go away with the next release of |
cargo-binstall 1.7.1 has released, cargo-bins/cargo-binstall#1741 which fixed this issue is released in 1.7.0 |
Update/TLDR
It looks like the issue is that
cargo binstall jj-cli
installs a broken binary from QuickInstall for jj 0.18 on linux x86 systems.Fix:
The problem is solved on recent versions of
cargo-binstall
. Make sure you're running the latest one before reinstalling jj to solve the problem:We'd appreciate a confirmation that the fix works for people.
Alternatively, or if you want to be doubly sure, you can run
Description
I just updated to 0.18 and most jj commands fail (some are fine, e.g.
log
) with similar error messages:(I do have a git config, it just lives in
~/.config/git/config
.)If I touch that file, the process repeats with
./.git/info/grafts
and./.git/shallow
.Finally, git operations of jj seemingly work again (at least
jj new
).However, using the git cli itself I get a warning about
.git/info/grafts
being deprecated.So that doesn't seem like a good solution to just make these dummy files.
Steps to Reproduce the Problem
This is the second computer I updated and the first didn't show this problem.
So I don't know how to reproduce it yet.
Expected Behavior
Git operations should succeed as normal.
Actual Behavior
Specifications
6.8.11-300.fc40.x86_64
The text was updated successfully, but these errors were encountered: