Update keyring docs further

[matteius]

          > From your comment @matteius, it sounds like the _intended_ behavior and usage pattern is that the keyring packages be installed inside of the project venv, and pipenv leverages those within-venv packages to perform installs of other packages?

That is the only way Pip, and thus Pipenv, was able to use keyring. This is no longer the only way.

• You can install keyring anywhere
• Make sure it is found on the PATH
• Make sure virtualenv seeds new venvs with Pip 23.1 or higher. (run virtualenv --upgrade-embed-wheels if that is not the case, or look at the output of virtualenv --help if that doesn't cause the version to change.)
• Run pip config set --global keyring-provider subprocess (or use --user if you prefer)
• Make sure the index has a username that is correct for the keyring backend
  • oauth2accesstoken for keyrings.google-artifactregistry-auth
  • VssSessionToken for artifacts-keyring

I wonder if it makes sense to have something like an install virtualenv, akin to how python packaging has the "build" environment, where packages that pipenv relies on to execute the installation process itself can be installed and walled off from the actual project virtualenv.

With the above you configure Pip.
Pipenv then requires the right index urls.

So improving the documentation seems like the better way to go about this.
If you disagree, let me introduce you to virtualenv's seeder concept. My keyring-subprocess package provides a seeder which seeds itself into new virtual environments.

PS. I contributed the --keyring-provider flag to Pip, eventually new Python versions ship with a wheel of Pip greater than or equal to 23.1 in their ensurepip module. Then Python's venv module will also create virtual environments with a Pip version >=32.1...

Originally posted by @Darsstar in #4706 (comment)

[kalebmckale]

I've recently become interested in using keyring but found it's not working as expected with pipenv, in particular, the subprocess option. Even though I formatted the URL with the username, when using pipenv, it still asks for input and causes error. The reason I'm posting this here as part of the discussion is my venture into keyring led me to Seeding in virtualenv although I'm even more confused how one actually does this. If using import keyring option, it's going to be necessary to have it installed in the venv but I also don't want to make it a dependency of every package Pipfile nor do I expect this to work well since it needs to be installed before the locking occurs. So, my first question is: is there a way to extend the current Seeding within pipenv without changing the source itself?

[Darsstar]

Sorry, I see I messed up the command. It should be global.keyring-provider.
So pip config set --global global.keyring-provider subprocess

[mungojam]

I'm pretty confused by how to get keyring to work, without needing users to either change their path or manually install keyring and my keyring extension in every single repository venv that they work in.

It would be good if there were some option to use the global keyring and extensions.

[Darsstar]

If the global python is on the path, or more specifically the path where pip install keyring puts the keyring executable, then you could tell people to pip install keyring <keyring backend> with the global pip can work as long as you use subprocess as the keyring-provider.

[mungojam]

If the global python is on the path, or more specifically the path where pip install keyring puts the keyring executable, then you could tell people to pip install keyring <keyring backend> with the global pip can work as long as you use subprocess as the keyring-provider.

The issue is that we don't want to add that folder to the path since it causes issues when we have multiple versions of python on the machine (with pyenv). We find it's better to push people to use python -m module.

I think I will try just calling pipenv run pip install keyring our-keyring-extension as part of our developer setup script that gets run for each repo. It may annoy people who try to run pipenv sync manually instead in new repos, but it's workable.

[mungojam]

Still not really working after manually installing keyring and our extension.

This works fine and shows it pulling requests from my authenticated store (AWS code artifact):

python -m pipenv run pip install requests

But this didn't work right before it, complaining that it couldn't find any available version of requests:

python -m pipenv lock --verbose
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
INFO:pipenv.patched.pip._internal.resolution.resolvelib.reporter:Reporter.starting()
INFO:pipenv.patched.pip._internal.resolution.resolvelib.reporter:Reporter.adding_requirement(SpecifierRequirement('requests'), None)
CRITICAL:pipenv.patched.pip._internal.resolution.resolvelib.factory:Could not find a version that satisfies the requirement requests (from versions: none)

[mungojam]

Looks like I got it working (with a warning).

set PIP_KEYRING_PROVIDER=import

Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
Success!
[====] Locking...Warning: WARNING:pipenv.patched.pip._internal.network.auth:Keyring is skipped due to an exception: expected str, bytes or os.PathLike object, not NoneType
Locking [dev-packages] dependencies...
Building requirements...
Resolving dependencies...
Success!
[ =] Locking...Warning: WARNING:pipenv.patched.pip._internal.network.auth:Keyring is skipped due to an exception: expected str, bytes or os.PathLike object, not NoneType

[Darsstar]

That warning means (vendored & patched) pip was not able to get credentials and it disabled keyring lookup.
So I'm not sure what you mean by 'working'.

Also, since you are using pyenv, does that mean you have people install pipenv for every python version?
If so I can recommend pipx, it comes with an easy pipx ensurepath command that puts itself on the path. You could than install pipenv and keyring with pipx. This is what I have my colleagues do.

[mungojam]

I'll check further, it didn't manage to lock before I told it to use keyring, it did succeed afterwards, but with those warnings.

pipx sounds helpful thanks. We do use pyenv.

[mungojam]

I've got a bit further after debugging into pipenv. It seems that there are two types of call made to keyring, one succeeds the other fails, in pipenv.patched.pip._internal.network.auth

        try:
            return self.keyring_provider.get_auth_info(url, username)
        except Exception as exc:
            logger.warning(
                "Keyring is skipped due to an exception: %s",
                str(exc),
            )

The first ones succeed as they pass in the full https://{url-from-pipfile} for url and {username-from-pipfile-url} for the username.

The ones that fail pass in None for the username, and just {domain-from-pipfile} for the URL, missing the https:// and the url path.

Any idea if that is by design somehow? I couldn't work out the call stack to really understand where the two different calls come from.

For the working call I see:

For the failing call I see:

[mungojam]

The lack of username is easier to workaround than the lack of URL path, as we return different credentials for some url paths.

[mungojam]

ok, I know what it is. It probably isn't a bug, though there might be something better that would help this rather niche case. It is a bug I believe, more details in later comments.

I've got two authenticated pip URLs, both supported by my keyring extension, but I had set one for my PIPENV_PYPI_MIRROR and PIP_INDEX_URL and the other in my Pipfile. The one in my Pipfile gets called first and gets passed the username and the full path and works fine for locking. The one in my environment variables can't be found in the pipfile and hasn't got the username by that point in the patched pip code and so goes via the other route in this condition, receiving a null username and netloc which is just the domain:

        # If we don't have a password and keyring is available, use it.
        if allow_keyring:
            # The index url is more specific than the netloc, so try it first
            # fmt: off
            kr_auth = (
                self._get_keyring_auth(index_url, username) or
                self._get_keyring_auth(netloc, username)
            )

It seems like a fix for this would be to maintain the username that I have in PIPENV_PYPI_MIRROR and PIP_INDEX_URL. Raw pip seems to pick it up just fine. For now I'll workaround this and put my second index into my Pipfile but in reality I don't want it in there as other users won't have access to it.

[mungojam]

Sorry for the spam, the above is mostly a red herring. I have specified the same repository in both environment variables and Pipfile. I think the issue is just that the username is lost in the second call so it can't then locate the index in the Pipfile. I'll do more digging into why.

[mungojam]

self.index_urls is ending up None under the failing path, which is then leading to the issue

[mungojam]

Sorry for the screenshots rather than references in the repo, but below is where the source URL is dropped which then ultimately leads to the warnings. I think the fix would be to pass down the source url as the only index_url which would then populate self.index_urls in my screenshot in my previous comment.

@matteius, I realise this is probably a bug report in itself. I'm off to bed shortly and might not get back to this for a while. But hopefully the fix is fairly simple and just needs another parameter passing through the call stack.

[mungojam]

I've put in a PR #5994 that fixes the warnings for me. Apologies it's a bit of a mess in terms of PR write-up etc. Feel free to amend the PR as needed