Porcelain Clone 0.22.3 not Cloning GitHub Branches Correctly in at least one case

[erl-hpe]

Using dulwich.porcelain clone() at the 0.22.3 release level to clone

git@github.com:Cray-HPE/vtds-configs.git

I wind up with refs that contain a single branch named 'master'. Unfortunately, the remote repository no 'master' branch, but it does have a 'main' branch. In other words, 'master' in the clone should be 'main' but is not. When I do the same thing with 0.22.1 I get what I expect.

Here is a transcript of the (correct) 0.22.1 behavior:

$ python3
Python 3.11.6 (v3.11.6:8b6ee5ba3b, Oct  2 2023, 11:18:21) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from dulwich.porcelain import clone
>>> url='git@github.com:Cray-HPE/vtds-configs.git'
>>> target='foo'
>>> clone(url, target)
<Repo at 'foo'>
Enumerating objects: 248, done.
Counting objects: 100% (248/248), done.
Compressing objects: 100% (100/100), done.
Total 248 (delta 109), reused 247 (delta 109), pack-reused 0 (from 0)
copied 247 pack entries47/248
>>> rating index: 247/248

and the list of branches in the resulting repo:

$ cd foo
$ git branch -l
* main

Here is what I see with 0.22.3:

$ python3
Python 3.11.6 (v3.11.6:8b6ee5ba3b, Oct  2 2023, 11:18:21) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from dulwich.porcelain import clone
>>> url='git@github.com:Cray-HPE/vtds-configs.git'
>>> target='foo'
>>> clone(url, target)
<Repo at 'foo'>
Enumerating objects: 207, done.
Counting objects: 100% (207/207), done.
Compressing objects: 100% (78/78), done.
Total 207 (delta 94), reused 204 (delta 93), pack-reused 0 (from 0)
copied 206 pack entries06/207
>>> rating index: 206/207

and the branches:

(venv) HPE-L91WD9JY2R:tmp erl$ cd foo
(venv) HPE-L91WD9JY2R:foo erl$ git branch -l
* master

[jelmer]

It's hard for me to reproduce this, since it's a private repository. It is probably a regression from the new v2 support.

Can you try passing protocol_version=0 to clone and seeing if that improves things?

[erl-hpe]

That's odd. It is listed as a public repository.

[erl-hpe]

[jelmer]

Ah, I've managed now - not sure what went wrong last time, unrelated issue with my SSH key.

[erl-hpe]

Cool. FYI - I was able to replicate it from another GitHub account using Ubuntu as well as from my MacBook.

[jelmer]

That said, I have trouble reproducing the bug:

% PYTHONPATH=. python3
Python 3.12.7 (main, Oct  3 2024, 15:15:22) [GCC 14.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from dulwich import __version__
>>> __version__
(0, 22, 3)
>>> from dulwich.porcelain import clone
>>> url='git@github.com:Cray-HPE/vtds-configs.git'
>>> target='foo'
>>> clone(url, target)
Enumerating objects: 207, done.
Counting objects: 100% (207/207), done.
Compressing objects: 100% (78/78), done.
Total 207 (delta 94), reused 204 (delta 93), pack-reused 0 (from 0)
copied 206 pack entries06/207
<Repo at 'foo'>x: 189/207
>>> rating index: 206/207
% cd foo
../foo% git branch -l
* main

[erl-hpe]

Huh. That's weird. I get the same behavior I described from two different systems using two different GitHub users. Setting protocol_version=0 does "fix" the behavior, though.

[erl-hpe]

What is funny about the output you pasted is that I see:

Total 207 (delta 94), reused 204 (delta 93), pack-reused 0 (from 0)
copied 206 pack entries06/207

when it is going to fail and

Total 248 (delta 109), reused 247 (delta 109), pack-reused 0 (from 0)
copied 247 pack entries47/248

when it is going to work, but you saw it work with the former output.

[erl-hpe]

I have seen it with python 3.12.3 and python 3.11.6 if that helps at all. I haven't tried 3.12.7.

[erl-hpe]

Is there anything else you would like me to look at, since I can get it to happen?

[erl-hpe]

I tried it from a cloned version of dulwich too (checking out the dulwich-0.22.3 tag and doing a pip install . from there) and got the same behavior.

[erl-hpe]

I doubt this is significant, but on both systems I am working in a venv, in case that sheds any light.

[jelmer]

It's probably not relevant, but do you have the rust extensions compiled?

The other thing you could try is see if things still fail at commit 436ce9f, which was just before the v2 improvements landed.

[erl-hpe]

I don't believe I have the rust extensions. If I do, it would be because of something I don't know about. I hate to say "no" absolutely without checking, any idea how I would check that? I will check 436ce9f for you.

[erl-hpe]

Not seeing the problem at 436ce9f.

[jelmer]

If there's files starting with _ in your dulwich package directory, you've got the Rust extensions.

[jelmer]

A big help would be if you could bisect to find the problematic commit, since I don't have a way to reproduce this. There shouldn't be too many in between 0.22.1 and 436ce9f

[erl-hpe]

Except for the normal python files (init.py, pychache and the like) and some C files and their related .so files, I don't see anything interesting starting with _, so I think, no rust extensions.

I will try to track down the specific commit later this morning.

[erl-hpe]

Looks like it broke between 325de7d and 2400933. I am thinking that there is something not working in the lower level implementation of the v2 protocol because the diffs between those commits are not very interesting in themselves.

[erl-hpe]

I am curious about why you can't reproduce it. I have managed to get it reliably with two very different GitHub users on two very different systems. I am going to see if someone else who is not me (physically) can reproduce it. I will report back shortly.

[erl-hpe]

So, just to see whether my GitHub repo is somehow corrupt or weird, I turned on packet tracing and version 2 protocol with my local git clone command. I don't see any references to master in there:

Cloning into 'foo'...
13:25:50.768043 pkt-line.c:86           packet:        clone< version 2
13:25:50.769334 pkt-line.c:86           packet:        clone< agent=git/github-d37f7b990c25
13:25:50.769375 pkt-line.c:86           packet:        clone< ls-refs=unborn
13:25:50.769397 pkt-line.c:86           packet:        clone< fetch=shallow wait-for-done filter
13:25:50.769418 pkt-line.c:86           packet:        clone< server-option
13:25:50.769437 pkt-line.c:86           packet:        clone< object-format=sha1
13:25:50.769454 pkt-line.c:86           packet:        clone< 0000
13:25:50.769472 pkt-line.c:86           packet:        clone> command=ls-refs
13:25:50.769506 pkt-line.c:86           packet:        clone> agent=git/2.46.1
13:25:50.769529 pkt-line.c:86           packet:        clone> object-format=sha1
13:25:50.769546 pkt-line.c:86           packet:        clone> 0001
13:25:50.769564 pkt-line.c:86           packet:        clone> peel
13:25:50.769581 pkt-line.c:86           packet:        clone> symrefs
13:25:50.769599 pkt-line.c:86           packet:        clone> unborn
13:25:50.769617 pkt-line.c:86           packet:        clone> ref-prefix HEAD
13:25:50.769636 pkt-line.c:86           packet:        clone> ref-prefix refs/heads/
13:25:50.769657 pkt-line.c:86           packet:        clone> ref-prefix refs/tags/
13:25:50.769675 pkt-line.c:86           packet:        clone> 0000
13:25:50.924390 pkt-line.c:86           packet:        clone< 7e26270b75cadd0d38b17847af98c275858c5800 HEAD symref-target:refs/heads/main
13:25:50.924506 pkt-line.c:86           packet:        clone< 9f42e0005c18e0de8de9aa72248252b74c693de4 refs/heads/VSHA-599
13:25:50.924557 pkt-line.c:86           packet:        clone< 9f42e0005c18e0de8de9aa72248252b74c693de4 refs/heads/VSHA-622
13:25:50.924586 pkt-line.c:86           packet:        clone< 848b1c382e1866ff09994156ebbd5ad117a987b0 refs/heads/another-test-branch
13:25:50.924613 pkt-line.c:86           packet:        clone< 7e26270b75cadd0d38b17847af98c275858c5800 refs/heads/main
13:25:50.924639 pkt-line.c:86           packet:        clone< ec8c69499125b56de957619a651383e7982acd71 refs/heads/test-branch
13:25:50.924666 pkt-line.c:86           packet:        clone< 716b5f7702e0846286a3c45bbbb342cc7b6cfa88 refs/tags/0.0.0 peeled:5b168b4d6421f3d8b248753900b516441f708b9c
13:25:50.924701 pkt-line.c:86           packet:        clone< 5b168b4d6421f3d8b248753900b516441f708b9c refs/tags/0.0.1
13:25:50.924721 pkt-line.c:86           packet:        clone< 0000
want 7e26270b75cadd0d38b17847af98c275858c5800 (HEAD)
want 9f42e0005c18e0de8de9aa72248252b74c693de4 (refs/heads/VSHA-599)
want 9f42e0005c18e0de8de9aa72248252b74c693de4 (refs/heads/VSHA-622)
want 848b1c382e1866ff09994156ebbd5ad117a987b0 (refs/heads/another-test-branch)
want 7e26270b75cadd0d38b17847af98c275858c5800 (refs/heads/main)
want ec8c69499125b56de957619a651383e7982acd71 (refs/heads/test-branch)
want 716b5f7702e0846286a3c45bbbb342cc7b6cfa88 (refs/tags/0.0.0)
want 5b168b4d6421f3d8b248753900b516441f708b9c (refs/tags/0.0.1)
13:25:50.933744 pkt-line.c:86           packet:        clone> command=fetch
13:25:50.933774 pkt-line.c:86           packet:        clone> agent=git/2.46.1
13:25:50.933784 pkt-line.c:86           packet:        clone> object-format=sha1
13:25:50.933792 pkt-line.c:86           packet:        clone> 0001
13:25:50.933801 pkt-line.c:86           packet:        clone> thin-pack
13:25:50.933808 pkt-line.c:86           packet:        clone> no-progress
13:25:50.933816 pkt-line.c:86           packet:        clone> ofs-delta
13:25:50.933830 pkt-line.c:86           packet:        clone> want 7e26270b75cadd0d38b17847af98c275858c5800
13:25:50.933851 pkt-line.c:86           packet:        clone> want 9f42e0005c18e0de8de9aa72248252b74c693de4
13:25:50.933859 pkt-line.c:86           packet:        clone> want 9f42e0005c18e0de8de9aa72248252b74c693de4
13:25:50.933870 pkt-line.c:86           packet:        clone> want 848b1c382e1866ff09994156ebbd5ad117a987b0
13:25:50.933879 pkt-line.c:86           packet:        clone> want 7e26270b75cadd0d38b17847af98c275858c5800
13:25:50.933913 pkt-line.c:86           packet:        clone> want ec8c69499125b56de957619a651383e7982acd71
13:25:50.933925 pkt-line.c:86           packet:        clone> want 716b5f7702e0846286a3c45bbbb342cc7b6cfa88
13:25:50.933933 pkt-line.c:86           packet:        clone> want 5b168b4d6421f3d8b248753900b516441f708b9c
13:25:50.933945 pkt-line.c:86           packet:        clone> done
13:25:50.933952 pkt-line.c:86           packet:        clone> 0000
13:25:51.119064 pkt-line.c:86           packet:        clone< packfile
13:25:51.258701 pkt-line.c:86           packet:     sideband< PACK ...
13:25:51.258911 pkt-line.c:86           packet:     sideband< 0000

and the command retrieved the repo the way I was expecting it to (i.e. with a main branch not a master branch). I don't see a way to turn on packet tracing with the porcelain clone operation, or I would do that and see what I get for comparison.

There are two things I am seeing here that seem interesting. You seem to be able to reproduce one of them but not the other:

• the log output shows 207 objects and 206 pack entries when I run it with the version 2 clone using porcelain versus 248 objects and 247 pack objects when I run it with the git command (any version) or the version 1 (i.e. version=0) porcelain clone. You seem to get this same result.
• I get a master branch instead of a main branch, and I don't see anywhere in the repo data where a master branch would be coming from -- it seems to come out of thin air. You don't appear to be getting that result.

It might be useful to focus on the disparity in objects and pack entries, since you are able to reproduce that (check whether you see 248/247 running with version=0, I expect you will). I expect that the issue I am seeing is related to / caused by whatever causes that.

I am getting a Raspberry Pi I have lying around set up to try this too so I can see if the problem exists there. That is taking some time because I have to update Python on it. I will update when I have results there.

[erl-hpe]

I freshly installed Python 3.12 on my Pi, cloned dulwich and checked out the dulwich-0.22.3 branch then ran my test and got the same result there. That was with my non-HPE github user (like what I used on Ubuntu) and nothing special in my environment. So far, I have not seen an environment in which this doesn't happen. I am going to stop looking for that.

[jelmer]

I think I know what's happening; with v2 there are two separate fetches of capabilities, and the "symref" capability is only extracted from the first one, while it appears in the second.

(The symref capability provides information as to where HEAD will point, with dulwich defaulting to "master" if none is provided by the server)

[jelmer]

@erl-hpe Would you be able to double check that this fixes the issue for you as well? I'm a bit hesitant to declare it fixed since I had trouble reproducing it earlier.

[erl-hpe]

I just tried it on my Pi, and now I am seeing no branches at al (i.e. no main and no master).

(venv) pi@softrock:~/dulwich $ git log
commit b1287d36edf7f54d38a0ed93021f2dc84f6db027 (HEAD -> master, origin/master, origin/HEAD)
Merge: 15d6c817 f1075d25
Author: Jelmer Vernooij <jelmer@jelmer.uk>
Date:   Sun Oct 20 13:11:57 2024 +0100

     Fix handling of symrefs with protocol v2. Fixes #1389  (#1392)

commit f1075d25f3a1ab7011f77290dfc3dbd3fb3f29c6
Author: Jelmer Vernooij <jelmer@jelmer.uk>
Date:   Sun Oct 20 02:08:56 2024 +0100

    Fix handling of symrefs with protocol v2. Fixes #1389

commit c6abf72bc691981e6468dd8b8e60151d6cdeb9df
Author: Jelmer Vernooij <jelmer@jelmer.uk>
Date:   Sun Oct 20 00:34:12 2024 +0000

    Factor out extraction of symrefs

commit 15d6c817bc5e2628a9a8eb3d8c4326f1bd86eb24
Merge: cd30df4e 7b881b38
Author: Jelmer Vernooij <jelmer@jelmer.uk>
..... <SNIP> ....

(venv) pi@softrock:~/dulwich $ pip install .
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Processing /home/pi/dulwich
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: urllib3>=1.25 in /home/pi/venv/lib/python3.12/site-packages (from dulwich==0.22.3) (2.2.3)
Building wheels for collected packages: dulwich
  Building wheel for dulwich (pyproject.toml) ... done
  Created wheel for dulwich: filename=dulwich-0.22.3-cp312-cp312-linux_armv7l.whl size=262055 sha256=338069918e97e8c931d5fc8f58274e06dde382da054205350cb46a1d75254fd1
  Stored in directory: /tmp/pip-ephem-wheel-cache-a302jopd/wheels/11/aa/07/b3f3284398e3f6df0f82c9b0189035fbf66bbec7d43ca54c0b
Successfully built dulwich
Installing collected packages: dulwich
  Attempting uninstall: dulwich
    Found existing installation: dulwich 0.22.3
    Uninstalling dulwich-0.22.3:
      Successfully uninstalled dulwich-0.22.3
Successfully installed dulwich-0.22.3

(venv) pi@softrock:~/dulwich $ cat do_clone.py 
from dulwich.porcelain import clone

url='git@github.com:Cray-HPE/vtds-configs.git'
target='foo'
clone(url, target)

(venv) pi@softrock:~/dulwich $ python do_clone.py 
Enumerating objects: 207, done.
Counting objects: 100% (207/207), done.
Compressing objects: 100% (78/78), done.
Total 207 (delta 94), reused 204 (delta 93), pack-reused 0 (from 0)
copied 206 pack entries06/207

(venv) pi@softrock:~/dulwich $ cd foo
(venv) pi@softrock:~/dulwich/foo $ git branch -l
(venv) pi@softrock:~/dulwich/foo $ 

[stspdotname]

Alright, I will be taking a look at this today.

[stspdotname]

There are two different issues to be considered here.

The first is indeed due to a lack of tests of symref handling in our compat tests. I have implemented tests for this, will send a PR shortly. The added tests uncovered a bug in fetch over HTTP (AbstractHttpGitClient) where the symref discovered in self._discover_references() gets clobbered by an empty symrefs dict returned from self._negotiate_upload_pack_capabilities(server_capabilities(). This can be fixed by using a separate variable and now the tests are passing.

The second problem seems to be specific to github's server because this problem does not show when I clone the repository from git-daemon. In the Git protocol trace above we see that Git is requesting specific reference prefixes. The dulwich implementation does not ask for ref-prefixes by default, assuming this would trigger the server to return all refs, as it does with git-daemon. However, github only returns the HEAD ref unless we specifically ask for "refs/".

So this code works with github:

from dulwich.porcelain import clone; clone('git@github.com:Cray-HPE/vtds-configs.git', 'foo', protocol_version=2, ref_prefix=[b'refs/'])

And this code works with git-daemon serving the cloned repository:

git daemon --verbose --log-destination=stderr --export-all --base-path=/tmp/foo/.git --listen=localhost --reuseaddr /tmp/foo/.git

from dulwich.porcelain import clone; clone('git://localhost/', 'foo2')

Not sure what to do here. Should we always add a default ref_prefix to please github? Should we document the issue and require users to pass a ref_prefix to avoid it?

[jelmer]

The behaviour of clone should be the same whether v0 or v2 is used, otherwise existing code breaks.

In that case, i think that means if ref_prefix is not specified then it should probably default to [b'refs/']

[stspdotname]

Alright, I like the idea of using a default argument. I will include HEAD in the ref-prefix list because Git does that, too. Just in case the HEAD ref would otherwise get filtered out. Doesn't seem to be the case with github, but who knows what other implementations might do.

[erl-hpe]

Thanks folks! It sounds like you have been busy. Let me know when you are happy with the state of things and I will try it out with my application.

[jelmer]

@erl-hpe please give it another try

[erl-hpe]

I am seeing another failure from my which could be a coding error in my application, but it is a difference in behavior between 0.22.1 and 0.22.4. You will see (below) that I went back to my simple example and am still seeing a problem (different but still there) with the behavior there too.

My code has in it the following (where self.repo is a Dulwich Repo class object created using the repo directory that was created by the clone() call:

        if self.default_version or self.version == b"HEAD":
            # Either no version is known yet for this repo, or the                                            
            # user requested HEAD (which we interpret at the remote                                           
            # HEAD since the local HEAD floats). Set the version to                                           
            # the remote HEAD symref branch.                                                                  
            symrefs = self.repo.refs.get_symrefs()
            self.version = (
                symrefs[b'refs/remotes/origin/HEAD'].split(b'/')[-1]
            )

That is getting a KeyError exception on b'refs/remotes/origin/HEAD' in 0.22.4 but works in 0.22.1.

Playing with this a bit interactively, I can see a difference in behavior for protocol 2 and protocol 1:

(venv) $ python3
Python 3.11.6 (v3.11.6:8b6ee5ba3b, Oct  2 2023, 11:18:21) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from dulwich.porcelain import clone
>>> from dulwich.repo import Repo
>>> from dulwich import __version__
>>> print(__version__)
(0, 22, 4)
>>> url='git@github.com:Cray-HPE/vtds-configs.git'
>>> target='foo'
# Using Protocol 1 for the repo in 'foo':
>>> clone(url, target, protocol_version=0)
<Repo at 'foo'>
Enumerating objects: 288, done.
Counting objects: 100% (288/288), done.
Compressing objects: 100% (119/119), done.
Total 288 (delta 125), reused 282 (delta 122), pack-reused 0 (from 0)
copied 287 pack entries87/288
>>> rating index: 287/288
>>> repo = Repo(target)
>>> symrefs = repo.refs.get_symrefs()
>>> symrefs
{b'refs/remotes/origin/HEAD': b'refs/remotes/origin/main', b'HEAD': b'refs/heads/main'}
# Now switching to 'bar' to clone a repo with Protocol 2:
>>> target='bar'
>>> clone(url, target)
<Repo at 'bar'>
Enumerating objects: 229, done.
Counting objects: 100% (229/229), done.
Compressing objects: 100% (90/90), done.
Total 229 (delta 103), reused 221 (delta 99), pack-reused 0 (from 0)
copied 228 pack entries28/229
>>> rating index: 228/229
>>> repo = Repo(target)
>>> symrefs = repo.refs.get_symrefs()
>>> symrefs
{b'HEAD': b'refs/heads/master'}

Notice that we only see a local head, and that head is names master not main as it is in the remote ref and as it is in the local ref using Protocol 1.

Let me know if I am making an invalid assumption here about there being a b'refs/remotes/origin/HEAD' key in symrefs, and what the valid assumption would be, or let me know if this should be filed as another issue. I don't want to drive this issue down a rabbit hole.

On the other hand, this might be related to the original problem?

[erl-hpe]

As I noted in the previous comment, when I switched from not specifying a version (i.e. wanting the current head of the default branch) to specifying main, I got the old application failure back, there being no main branch in the resulting repo. I re-ran my code snippet that I filed originally and got different behavior (now there are no branches listed):

(venv) $ python3
Python 3.11.6 (v3.11.6:8b6ee5ba3b, Oct  2 2023, 11:18:21) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from dulwich import __version__
>>> __version__
(0, 22, 4)
>>> from dulwich.porcelain import clone
>>> url='git@github.com:Cray-HPE/vtds-configs.git'
>>> target='foo'
>>> clone(url, target)
<Repo at 'foo'>
Enumerating objects: 229, done.
Counting objects: 100% (229/229), done.
Compressing objects: 100% (90/90), done.
Total 229 (delta 103), reused 221 (delta 99), pack-reused 0 (from 0)
copied 228 pack entries28/229
>>> rating index: 228/229
>>> exit()
$ cd foo/
(venv) $ git branch -l
(venv) $ 

[erl-hpe]

If I drop back to 0.22.1, this is what I see:

(venv) HPE-L91WD9JY2R:foo erl$ python3
Python 3.11.6 (v3.11.6:8b6ee5ba3b, Oct  2 2023, 11:18:21) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from dulwich import __version__
>>> print(__version__)
(0, 22, 1)
>>> from dulwich.porcelain import clone
>>> url='git@github.com:Cray-HPE/vtds-configs.git'
>>> target='foo'
>>> clone(url, target)
<Repo at 'foo'>
Enumerating objects: 288, done.
Counting objects: 100% (288/288), done.
Compressing objects: 100% (119/119), done.
Total 288 (delta 125), reused 282 (delta 122), pack-reused 0 (from 0)
copied 287 pack entries87/288
>>> rating index: 287/288
>>> ^D
(venv) HPE-L91WD9JY2R:foo erl$ cd foo
(venv) HPE-L91WD9JY2R:foo erl$ git branch -l
* main

[erl-hpe]

I tried the same thing in 0.22.4 with protocol version set to 0, and got the 0.22.1 behavior:

(venv) HPE-L91WD9JY2R:foo erl$ python3
Python 3.11.6 (v3.11.6:8b6ee5ba3b, Oct  2 2023, 11:18:21) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from dulwich import __version__
>>> print(__version__)
(0, 22, 4)
>>> from dulwich.porcelain import clone
>>> url='git@github.com:Cray-HPE/vtds-configs.git'
>>> target='foo'
>>> clone(url, target, protocol_version=0)
<Repo at 'foo'>
Enumerating objects: 288, done.
Counting objects: 100% (288/288), done.
Compressing objects: 100% (119/119), done.
Total 288 (delta 125), reused 282 (delta 122), pack-reused 0 (from 0)
copied 287 pack entries87/288
>>> rating index: 287/288
>>> exit()
(venv) HPE-L91WD9JY2R:foo erl$ cd foo
(venv) HPE-L91WD9JY2R:foo erl$ git branch -l
* main

[erl-hpe]

If I were to add a fetch after the clone call, I think I would get closer to the behavior I am looking for. Is this how the porcelain operations are meant to be done? If so, is there a way other than looking for the remote HEAD branch (which may not exist) to determine what the remote repo thinks its default branch is? I am thinking that there must be, because git clone is able to do it, but I have not found the proper mechanism in dulwich.

[jelmer]

This is a genuine bug, no extra calls should be necessary.

[erl-hpe]

Okay. Cool. For the moment, I am working around it in my application by forcing the protocol_version to 0 in my call to clone. Everything else I am doing seems to work fine when I do that. It is easy enough to change that out either to test fixes or to return to the regularly scheduled behavior when it is fixed. For now it lets me follow the release stream without needing to pin an older version of Dulwich. Thanks for your work on this. If there is anything I can reasonably do to help, let me know and I will.

[stspdotname]

The above behaviour is due to a bad default setting in the porcelain layer. The PR I submitted should fix it.

[stspdotname]

The second problem seems to be specific to github's server because this problem does not show when I clone the repository from git-daemon. In the Git protocol trace above we see that Git is requesting specific reference prefixes. The dulwich implementation does not ask for ref-prefixes by default, assuming this would trigger the server to return all refs, as it does with git-daemon. However, github only returns the HEAD ref unless we specifically ask for "refs/".

So this code works with github:

from dulwich.porcelain import clone; clone('git@github.com:Cray-HPE/vtds-configs.git', 'foo', protocol_version=2, ref_prefix=[b'refs/'])

And this code works with git-daemon serving the cloned repository:

git daemon --verbose --log-destination=stderr --export-all --base-path=/tmp/foo/.git --listen=localhost --reuseaddr /tmp/foo/.git

from dulwich.porcelain import clone; clone('git://localhost/', 'foo2')

@carlosmn Hey, any idea if something could be done about the above problem at Github's end? It looks like a server-side bug to me.

[erl-hpe]

@stspdotname will explicitly setting ref_prefix work everywhere (at a cost for git-daemon)? Or will there be other problems? I think I am okay with returning all refs, unless I am misunderstanding the impact of that. I would be willing to make that my permanent solution to this problem for my code (though there is still the question of whether you want to change something here to avoid a similar regression for others between v1 and v2 on GitHub).

On reflection, I could imagine cases where ref_prefix would want to be something other than b'refs/' so I suppose, if I were to go that route, I would want to make it configurable for each given repo that I need to pull from (defaulting to b'/refs' in the config). That would also be manageable in my particular case.

I am also curious why git clone is able to get all the refs appropriately from GitHub if this is a GitHub bug, I have tried git clone -c 'protocol.version=2' (with packet tracing) and, looking at the resulting cloned repo, it seems to do the right thing and I have tried git clone -c 'protocol.version=0' (with packet tracing) and gotten a substantially different trace, so I am confident that the protocol version is actually changing.

Not saying it isn't a GitHub bug, or at least inconsistency, just wondering why it impacts Dulwich differently from the Git commands.

[erl-hpe]

Also, I can confirm that the refs/remotes/origin/HEAD issue appears to have been fixed by your PR. At least it works with the latest in master and it doesn't with straight 0.22.4. I was settingr ref_prefix in both cases. That means I have a workaround I can use and leave in place once the refs/remotes/origin/HEAD fix makes it into a release.

UPDATE: actually, I am finding that what is at the tip of master right now works with or without the ref_prefix setting, so it works as originally written in my code for both the explicitly specified main case and the implicit refs/remote/origin/HEAD case. I think this is victory, unless I am missing something.

[jelmer]

@erl-hpe Several more fixes have gone into main - can you try again? The original issue now reliable results in a "main" branch being created for me.

[erl-hpe]

The latest main (consistently pass around ref_prefix and protocol_version in dulwich.client (#1421) is my latest log entry) appears to be working with no problem with my application. I have tried main, another branch that I have, and NULL for the versions of the repose I am pulling down (NULL results in trying to use refs/remotes/origin/HEAD). All three seem to work well.

Thanks for the quick and diligent response to this issue!

[jelmer]

Thanks for your patience in helping us debug this, and to @stspdotname for debugging and fixes.

[carlosmn]

Did you figure out what was causing the bare ls-refs not to work? Testing against GitHub I can see that a git ls-remote gets all of the refs without specifying a prefix.