Implement new command 'pip index versions'

[NoahGorny]

Instead of doing dirty tricks in order to get all available versions of a package(https://stackoverflow.com/questions/4888027/python-and-pip-list-all-versions-of-a-package-thats-available), we can now call
pip search <package> --list-all-versions in order to see all of the available versions!

Lemme know what you think @uranusjr @pradyunsg!

[NoahGorny]

Not sure what issue number to use with the NEWS entry... I remember that using the PR number is not a good idea :(

[McSinyx]

Not sure what issue number to use with the NEWS entry...

It could be GH-7975 I think. I'm glad that this i picked up after I totally forgot about it 😅

[NoahGorny]

Not sure what issue number to use with the NEWS entry...

It could be GH-7975 I think. I'm glad that this i picked up after I totally forgot about it sweat_smile

I see, added a news entry now 😄

[NoahGorny]

@pradyunsg @uranusjr @McSinyx Im humbly bumping this PR, because I want to know what do you think about this feature offer. should this be discussed in the python forums?

[pfmoore]

Looks OK to me. I'm a little bit ambivalent about adding this to search, but simply because the existing search is so bad that I'm not entirely sure we want to have people thinking it's useful for anything 😉 But I've no objection to this going in.

[pradyunsg]

I'd prefer to hold off, since we have the UX study doing a full pip functionality overview right now, and I don't want us to make changes to the commands in the middle of that.

/cc @ei8fdb @nlhkabu in case they want to weigh in or something.

[ei8fdb]

Hi @NoahGorny! As @pradyunsg mentions, we are doing some UX research on what users need from pip search. We're still figuring out the results, but this PR will allow us to test something with users. Thanks for this.

I'm going to try out your PR to see the output. Ideally we'd be able to usability test this with users and get feedback. Would you be OK with discussing usability improvements we might recommend?

I've tried it out and so far so good. 😄

[NoahGorny]

Hi @NoahGorny! As @pradyunsg mentions, we are doing some UX research on what users need from pip search. We're still figuring out the results, but this PR will allow us to test something with users. Thanks for this.

I'm going to try out your PR to see the output. Ideally we'd be able to usability test this with users and get feedback. Would you be OK with discussing any usability changes?

I welcome any feedback and suggestions, and happy to help! I think you guys are doing an amazing job improving pip which is one of the most important and used tools today!

I will try to be as available as I can, and I am open to new ideas and changes 😄

[NoahGorny]

@ei8fdb @pradyunsg are there any updates on things I should do? I do not want this PR to be forgotten 😢

[uranusjr]

I like overloading pip search since the command is a mess already, and we likely want to come up with something entirely different (e.g. pip find) to replace it. We can also build on this and add more experiements before committing them into the new command.

[uranusjr]

I wonder if it would be better to use --use-feature for this though. The flag would somehow communicate the feature is more provisional and subject to change in the future.

[NoahGorny]

I wonder if it would be better to use --use-feature for this though. The flag would somehow communicate the feature is more provisional and subject to change in the future.

We can just add a warning that this API is experimental, and if we want to move away from it, we can deprecate it in the future

[pradyunsg]

We can just add a warning that this API is experimental, and if we want to move away from it, we can deprecate it in the future

I'm on board if we're letting ourselves remove this in a future release without a deprecation cycle.

WARNING: pip search --list-all-versions is currently an experimental flag. It may be removed in a future release without prior warning.

[NoahGorny]

We can just add a warning that this API is experimental, and if we want to move away from it, we can deprecate it in the future

I'm on board if we're letting ourselves remove this in a future release without a deprecation cycle.

WARNING: pip search --list-all-versions is currently an experimental flag. It may be removed in a future release without prior warning.

Why not use a deprecation cycle? it only requires us to remember deprecating and removing after a while

[NoahGorny]

I switched to json API as the xml-rpc is deprecated and causing me trouble. Not sure this is the right way to do it as I handled the http errors myself

[uranusjr]

I’m adding this to the 21.0 milestone because

1. People need an alternative to the legacy resolver’s pip install foo== hack after the legacy resolver is removed entirely.
2. We should provide an alternative, and this is a choice. Even if we decide to not go with this, we must implement something else, and having this PR in the milestone makes sure we do that.

[pfmoore]

A quick note. Pip 21.0 is due for release in about 2 weeks (see #9282). This item is on the 21.0 milestone, so it either needs to be implemented and merged to master before then, or it will miss the release (and I'll move it to the 21.1 milestone).

[NoahGorny]

I mostly wait for people to have feedback on this, as I use the JSON API and want to hear ppl opinion about it

[pradyunsg]

People need an alternative to the legacy resolver’s pip install foo== hack after the legacy resolver is removed entirely.

We've essentially restored this with #9228 being closed. I don't think this blocks 21.0, but, we should figure out what we really want to do here.

[uranusjr]

It'd be useful of we could get an update to the UX research about this.

[BrownTruck]

Hello!

I am an automated bot and I have noticed that this pull request is not currently able to be merged. If you are able to either merge the master branch into this pull request or rebase this pull request against master then it will be eligible for code review and hopefully merging!

[earonesty]

is the only thing holding this back a need to rebase?

[jgspratt]

Who can merge this?

[uranusjr]

Looks reasonable to me. So, as the first shot at this, @NoahGorny could you modify this PR to implement pip index versions <package> instead? The rest can come later after we get this in.

[NoahGorny]

Looks reasonable to me. So, as the first shot at this, @NoahGorny could you modify this PR to implement pip index versions <package> instead? The rest can come later after we get this in.

I see, will do in the following days!

I think you can try and review my changes in the meantime.. I am not super happy about the way I retrieve the information. I should probably split the JSON API logic to a separate class, or use the simple API instead.

[uranusjr]

Let's use the simple API instead. You should be able to reuse may logic in pip._internal.index

[NoahGorny]

Hi @uranusjr @pradyunsg, I have updated the PR as needed, added tests, and I feel like its ready for a review.

Let me know what you think 😄

[pradyunsg]

Happy to see this merged, and we can iterate on this.

A few non-blocking suggested rephrasings.

[uranusjr]

A general note: The current implementation lists only applicable versions of a project, but that behaviour (same as pip install etc.) sometimes leads to a common user confusion: Why does pip not see a newly published version?

I guess the current impleemntation is probably good enough as an initial effort, but eventually we probably need to also list “found but not applicable” versions as well (and potentially also why they are not applicable). I’m not sure how it can be done best, however.

[pradyunsg]

I clicked the wrong button. :)

There are other concerns. :)

[NoahGorny]

Thanks for the review @uranusjr and @pradyunsg, I have applied your suggestions

[pradyunsg]

I think this is good to merge, and we can iterate on this as we go! :)

I'll defer to @uranusjr for the merge.

[uranusjr]

I created #10052 to track the follow-up messaging improvement I mentioned earlier.