HTTP 407 error when using a git source

[jakecoffman]

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: macOS 12.5.1, Debian 11
• Poetry version: 1.2.0, also tested against master and older 1.1 releases
• Link of a Gist with the contents of your pyproject.toml file: https://github.com/dsp-testing/poetry-407/blob/main/pyproject.toml

Issue

Hi Poetry friends 👋

In Dependabot we're seeing an issue where when pyproject.toml contains a git source for a dependency we get a 407 Proxy Authentication Required. Dependabot uses a proxy that requires authentication to do updates. To reproduce the issue, I created https://github.com/dsp-testing/poetry-407 since it takes a bit of setup.

It seems that when the git source is present, Poetry loses the username and password to the proxy.

Let me know if you have other questions or I've left something out. Thanks!

[dimbleby]

if this is specifically a problem with git dependencies then probably it's a dulwich issue and wants reporting there.

Even better if you can create a standalone repro using dulwich code.

Looks like dulwich does have code to try to handle this - https://github.com/jelmer/dulwich/blob/3d4322c06fa79ddfff95e09dbc365b83d9b3f15b/dulwich/client.py#L1834. I could believe either that that's not working properly, or that poetry is somehow doing it wrong.

meanwhile you could try sidestepping dulwich and falling back to system git per https://python-poetry.org/docs/configuration/#experimentalsystem-git-client

[jakecoffman]

meanwhile you could try sidestepping dulwich and falling back to system git

That's great to know! Dependabot will have to run this way since we have to force https. We do this with a shim to the system git.

I tried adding that setting to my repro and it still fails with a 407. I think that should eliminate dulwich as the problem? But also like I said we'll have to use the system git in Dependabot anyway.

Also I mentioned I tested this with previous versions (1.1.15) and it fails in the same way. It looks like dulwich was introduced in 1.2.0? We had reports of this happening back in January here and here.

[dimbleby]

well now - and in earlier releases - poetry should just be shelling out to git. So now you seem to have a question about how to get git to use http proxy.

I've no idea whether it respects those environment variables, I see that git has config settings for http.proxy, at this point I think you want to experiment with calling git directly and take poetry out of the equation altogether

[dimbleby]

Just to confirm: I updated your repro so that script.sh ends with git clone https://github.com/uiri/toml and doesn't do anything poetry at all.

That sees the same output:

Cloning into 'toml'...
127.0.0.1:59226: client connect
127.0.0.1:59226: CONNECT github.com:443
              << 407 Proxy Authentication Required 129b

which seems to make clear that poetry is not introducing a problem here.

[jakecoffman]

Yes I did the same thing! I think the repro isn't quite right and your analysis that it may be dulwich is correct.

I'm updating Dependabot to use Poetry's experimental.system-git-client feature which seems to have taken care of it locally for me.

Thanks for the help!

[dimbleby]

No worries.

dulwich is reasonably active, if you do find the time to reproduce accurately they'd likely welcome the report.

[neersighted]

I would encourage getting support into dulwich -- long-term we don't plan to keep the system-git-client option, but only after dulwich proves stable/feature complete for our uses.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.