Improve the state of Python linting

[cdrini]

Python linting is currently a problem. Everyone is constantly being hit with linting issues, creating new commits to fix linting issues, either wasting manual time to fix issues or using automatic commands that make changes to loads of unrelated places. And it makes our git history a mess, and diffs/PRs hard to read.

I think the end goal we want is: Easy way to automatically apply linting before a commit happens.

A subtask of #4776

Steps to get there:

• Make our current lint-diff less aggressive while we improve the flow
  • @cclauss : change lint-diff to flake8 --select=E9,F63,F7,F82 ; only catch syntax errors and undefined
  • Slim down scripts/flake8-diff.sh #5842
  • fix pre-commit hook for flake8 #5844
• Run auto-formatter on codebase -- ignoring imports #5836 Black the entire codebase
  • /tests pyupgrade + black on /tests #5338
  • /scripts pyupgrade + black on /scripts #5841
  • Integration tests pyupgrade + black integrations tests #5850
  • black --skip-string-normalization ./scripts ./tests *.py #5859
  • Use from __future__ import annotations type hints #5864
  • /openlibrary pyupgrade + black+ codespell code for the entire repo #5855
• @RayBB Update gitpod to use pre-commit hooks Precommit hooks for gitpod #5851
• Add codespell to pre-commit and setup.cfg #5847
• Add black, mypy, and pyupgrade to pre-commit #5849
• pre-commit: Run flake8-diff.sh outside of Docker #5870
• Cause any/all lint issues to fail GitHub CI GitHub Action to run pre-commit #5848
• Investigate: Do we want to auto-fix in GitHub CI? Or pre-commit-ci.com?
• Update setup docs to make enabling pre-commit hooks a part of the flow
• Update docs with instructions on how to auto-lint-on-save for most popular editors
  • vscode
  • pycharm
• Profit?

Stakeholders

@cclauss @RayBB @jimman2003

[cclauss]

Closing because these tools are now all in place but need to be documented... #6227