Cannot jj git push to Github Error: unexpected http status code: 500; class=Http (34) or unpacking the sent packfile failed on the remote; class=Net (12)

[jbcdnr]

Description

We have been using jj for 6 months on a repo stored in Github.
Today we cannot push to the repo anymore. We did not change anything to our setup recently.

$ jj git push
Branch changes to push to origin:
  Force branch jb/new-branch-jj-debug from eb4925258b6a to 0da996a6d3d4
Error: unexpected http status code: 500; class=Http (34)

Some of us also see the following error:

$ jj git push
Branch changes to push to origin: Add branch henning/policy-nebius2iam to 15df333145f6
Error: unpacking the sent packfile failed on the remote; class=Net (12)

Even more strange, some of us can push but not if it edits certain files. We could not reproduce this half working state on another computer.

Steps to Reproduce the Problem

1. jj git push

Specifications

• Platform: ubuntu 22.04
• Version: 0.17.1, 0.15.1, 0.15.0

[bnjmnt4n]

A search across GitHub gives me r-universe-org/help#152 (comment), where a similar issue seems to arise because of what seems to be a malformed Git commit signature. Is it possible to verify if there's such a malformed commit in the repository, perhaps through git log --show-signature?

[jbcdnr]

Thanks for the reply. We don't use GPG signature. I tried git log --show-signature but I don't know how to find malformed signature, all the commits show red messages like:

gpg: Signature made Fri May 31 15:44:12 2024 CEST
gpg:                using RSA key B<REMOVED>
gpg: Can't check signature: No public key

[jbcdnr]

It seems that the problem occurs whever we edit one of the files of our repo. We cannot push with that error:

Error: unexpected http status code: 500; class=Http (34)

if we do git gc we then get this error:

remote: error: inflate: data stream error (incorrect header check)
remote: fatal: pack has bad object at offset 1838: inflate returned -3
Error: unpacking the sent packfile failed on the remote; class=Net (12)

[jbcdnr]

After more investigation, the error occurs when we edit some large files (100KB) and try to push. But pushing the same files with git works.

[jbcdnr]

For more context, we have a large repo with many files and a long history containing large files. Is jj sending some metadata or header to the git server that would cause the server to get out of RAM and send this internal error? This is not happening with git.

[yuja]

iirc, libgit2 (which jj uses for fetch/push) doesn't support the v2 protocol. The stuff to be exchanged would be different between git push and jj git push. I don't know if that's significant in this context, though.

https://git-scm.com/docs/protocol-v2

Appears that the protocol can be downgraded by git -c protocol.version=1, but there might still be a minor difference.

[jbcdnr]

@yuja thanks for chiming in.

I am not sure if I get your suggestion right. Would that be a way to break the git push to see if github server relies on protocol.version=2 and that would explain why jj git push does not work?

[jbcdnr]

I tried to push a large file to our repo with git protocol v1 and it works: git -c protocol.version=1 push

[martinvonz]

It seems that the problem occurs whever we edit one of the files of our repo. We cannot push with that error:

Error: unexpected http status code: 500; class=Http (34)

if we do git gc we then get this error:

remote: error: inflate: data stream error (incorrect header check)
remote: fatal: pack has bad object at offset 1838: inflate returned -3
Error: unpacking the sent packfile failed on the remote; class=Net (12)

Does that happen only after you modify a large file? Does git gc work immediately after creating a fresh clone? If it doesn't, and you then modify a large file, then run e.g. jj status to trigger snapshotting, does it happen then? I'm trying to understand if you somehow already have some bad data in the remote of it gets created as soon as you modify a large file.

Is the repo publicly available so we can try it ourselves?

[hmeyer]

I'm working on the same repo as @jbcdnr .

Does that happen only after you modify a large file?

It seems this issue is quite flaky. We saw this mostly happen on large files. But here is what I just tried:

git clone https://github.com/$ORG/$REPO

cd $REPO
git checkout -b henning/test34
touch test.txt
git add test.txt
git commit -m test
git push --set-upstream origin henning/test34
# WORKS!

jj git init --colocate
jj new main
touch test.txt
jj branch create henning/test35
jj describe -m test
jj git push
Branch changes to push to origin:
  Add branch henning/test35 to 410d46ea76ad
Error: unpacking the sent packfile failed on the remote; class=Net (12)

git gc
# Works
jj git push
Branch changes to push to origin:
  Add branch henning/test35 to 410d46ea76ad
# Now it works!

jj new main
echo foo >> $LARGE_FILE
jj describe -m test
jj branch create henning/test-large-jj
jj git push
Branch changes to push to origin:
  Add branch henning/test-large-jj to ccd0c41b410b

# WORKS AS WELL

Yesterday we created multiple commits that we couldn't push and also couldn't fix using git gc. To clarify git gc always ran successfully, but it didn't help with pushing the commits afterwards.
Also if we just jj git clone $REPO, we can't use git gc.

Does git gc work immediately after creating a fresh clone?

Yes. See above.

Is the repo publicly available so we can try it ourselves?

Sadly no. But I'm happy to try any command on it for debugging.

[martinvonz]

My guess is that that suggests a bug in either libgit2's code for crashing the pack file to send, or in gitoxide's code for creating the objects. The latter seems more likely since gitoxide is a younger project.

[hmeyer]

Error: unpacking the sent packfile failed on the remote; class=Net (12)
@martinvonz - do you really think the error happens on the client side? What can we do to debug this further?

[martinvonz]

No, I don't think it happens on the client, but I think it's because of a bug on the client. I think the client produces an invalid request for some reason. One way to debug it might be to find another tool that lets you push using libgit2.

You said it works to push with git right side jj git push fails, right? That seems like an indication that it's actually a libgit2 bug because gitoxide is not involved there.

[martinvonz]

@Byron, do you have any idea what might cause the error? FYI, we use Gitoxide to create commits and then use libgit2 to push them. Can you think of anything that Gitoxide does that might result in packfiles that night surprise libgit2?

[yuja]

libgit2 doesn't provide easy tracing options, but if you can build jj from source, maybe you can hook some libgit2 functions by gdb to inspect response from the server.

For example, anything other than "unpack ok" is transformed to the unpack error here.
https://github.com/libgit2/libgit2/blob/36f7e21ad757a3dacc58cf7944329da6bc1d6e96/src/libgit2/transports/smart_pkt.c#L429

[Byron]

@Byron, do you have any idea what might cause the error? FYI, we use Gitoxide to create commits and then use libgit2 to push them. Can you think of anything that Gitoxide does that might result in packfiles that night surprise libgit2?

It's hard for me to imagine how this would be possible.
Something I would try to debug this is to run a deep git gc before pushing to see if there are any invalid objects.

As last ditch effort to rule that idea out could be to replace commit creation with libgit2.

Unpack failures can certainly also happen if the packs integrity check fails, maybe this could be malformed commits, or a malformed graph structure, like missing objects, or other forbidden things.
Edit: It should be possible to even intercept the pack by proxy, and then verify it locally.

[martinvonz]

As last ditch effort to rule that idea out could be to replace commit creation with libgit2.

@hmeyer, could you try the steps you reported above with jj before 0.11? I think that was the last version before we started using Gitoxide.

[hmeyer]

@martinvonz
Using jj 0.10.0-ce933507df6eb51286cb7c7e3f1aaa53e581a3ef I just jj git clone'd our repo, created a commit and branch and jj git push failed again:

$ ../jj git push
Branch changes to push to origin:
  Add branch henning/test117 to 0155c3e8ce40
Error: unpacking the sent packfile failed on the remote; class=Net (12)

[martinvonz]

That's useful information. That means Gitoxide is not involved (sorry about the noise, @Byron!). @hmeyer, can you try cloning your repo and then simply running git fsck and git repack -a and let us know what they say. I wonder if there's some problem with the repo on there remote.

[hmeyer]

@martinvonz did that just now. Neither commands reported any problems :-/

[jbcdnr]

@martinvonz would you have some other ideas on things to try to narrow down the issue?

[martinvonz]

I would try following @yuja's suggestion. I think it's going to be hard for the rest of us to figure out what's going on since we can't reproduce the problem.

Btw, there was some other issue with cargo binstall (#3844). It doesn't seem like that would be related, but try installing from source to be sure.

[martinvonz]

Btw, could it be that there was some upgrade of git or github/gitlab/etc done on your server?

[yuja]

If you try debugger path, maybe you can use rr to capture trace once and inspect.
https://rr-project.org/

Alternatively, you can use wireshark to capture HTTPS (not SSH) packets and decrypt. However, you'll probably need to hook some OpenSSL functions to obtain encryption keys.
https://wiki.wireshark.org/TLS#using-the-pre-master-secret

[jbcdnr]

@martinvonz we are on a Github enterprise repo hosted by Github. They might have updated something on the server but it's not visible to us.

I will try to build from source and see if I can find some more info in the traces.

Thank you for your help.

[jbcdnr]

We could not reproduce the problem anymore on our side, maybe it was a problem with Github servers. I am closing this issue.

Thank you for the help and the great work, jj is awesome!