Carry out research to understand pip functionality

[ei8fdb]

Carry out research on pip functionality

Overall objective of the study

The objective of this research is to understand:

1. What pip functionality is most/least useful?
2. Do pip's commands and settings make sense?
3. How do users understand pip's output including error messages?
4. How can user research help the pip team move forward on specific issues?

Research questions

Each of these research questions will be a separate Github issue, linked back to this epic.

1. What pip functionality is most/least useful?

Note: This research question will be informed by the user types identified in #8518

In this research we'll ask questions like:

• How much do you value each pip feature?
• Do you use pip to install from PyPI, a private index, or both?
  • (for both) What problems or limitations do you encounter? Where do you go for help?
• Is security and package integrity important to you?
  • How do you ensure the packages you install are secure?
  • How do you ensure the packages you install are what you expected?
  • Do you only rely on pip's security features, or do you something else (e.g. audit the code you install)?
• Is there any functionality that you feel is missing from pip?

Related GitHub issue: #6221

2. Do pip's commands and settings make sense?

In this research we'll ask questions like:

• Can you remember pip's different commands and their syntax? Which ones do you regularly forget?
• (Show user a list of pip commands without descriptions) Can you tell me what each command does?
  • Follow up question: Would you like pip to have fewer commands and settings? Why?
• How regularly do you use pip help? Why?
• How do the pip commands compare to other tools you use? Do you prefer the way another tool names their commands? Why?
• What is the difference between pip check, pip list, pip show and pip freeze? If we grouped these under one command, what would you call it?
• If a pip discover or pip find command existed, what would you expect it to do?
• How do you use the pip wheel command?
• What do you know about PEP 517 and 518? Are the changes described there going to disrupt your current workflow?

Related GitHub issues: #8454, #8130, #8077, #7975, #6817

3. How do users understand pip's output including error messages?

Note: this question will require us to identify two example commands for users to run:

• example pip error should throw an error caused by pip
• example package error should throw an error caused by a package pip is trying to install

In this research we'll ask questions like:

• When you run pip install flask what do you understand from pip's output? What is useful? What is not useful?
• Have you ever used pip's verbosity settings? Do they make sense?
• When you experience example pip error, what would you do? What information should pip provide? How you you feel about being redirected to the documentation in this situation?
• When you experience example package error, what would you do? Do you know how to find the root of your problem? What information is missing?

Related GitHub issues: #8346, #6959

4. How can user research help the pip team move forward on specific issues?

In this research we'll ask questions related to pip search (#5216),pip force install/override (#8452) and pip --force-reinstall (#8238).

We'll ask questions like:

• How do you use pip search? Where do you expect it to search? How would you like it to work?
• In what situations (if any) would you want to force pip to install packages that have broken dependencies?
  • What are your expectations if you later use pip in an environment that has broken dependencies?
  • Does your answer change if you'd forgotten that you previously forced the install?
• What does pip --force-reinstall do? What should it do?

Related GitHub issues: #5216, #8452, #8238

Preliminary research plan

Note: We should expect some foundational insights from @ei8fdb's work on #8518

Some initial ideas:

• Collect feedback from the community via targeted surveys
• Conduct user interviews with a variety of pip users.
  It is important that we interview users with a range of experience levels, as we exepect their comfort and familiarity with pip to vary substantially.

Possible outputs

• UX recommendations shared on relevant GitHub issues (e.g. force install recommendations posted to Research: pip force install #8452)
• Error message guidelines - published in the pip documentation and shared on relevant issues
• Recommendations on how to clean up pip's output, including if/how we should reduce the default verbosity and/or amend the verbosity settings
• Recommendation on how to improve pip's commands and settings
• Recommendation on how to improve pip help
• Prioritisation tool/matrix that identifies core vs niche functionality. Purpose is to help developers prioritise what's most important/valueable to work on, and what can be marked as not important/won't fix.

[nlhkabu]

UX update:
We have set up surveys and started interviews to cover most of this content.

Research gaps to address:

• Error message research

Survey results and interview notes to be published here.

[pradyunsg]

Closing this, since I think the notes are going to published in #10133.