chore: Update Python development tools and unify Black into Ruff

[achimnol]

• setup: Update python tool dependencies
  • black: 23.9.1 -> completely replaced with ruff
    • The tool resolve still remains to prevent crashing of black initialization in our CI that runs pants tailor, which implicitly installs the black to scan BUILD files.
  • mypy: 1.6.1 -> 1.7.1
  • ruff: 0.1.1 -> 0.1.7
  • pytest: 7.4.2 -> 7.4.3

Note

You need to re-run:
pants export --resolve=black --resolve=mypy --resolve=ruff --resolve=black
to use the latest tool versions with your IDE or editor with LSPs.

Note

Note that black is deprecated but still remains as a dependency resolve set (tools/black-requirements.txt and tools/black.lock) to prevent Pantsbuild from failing on setting up pants tailor command when linting BUILD files.

Important

Ruff now provides an intrinsic formatter that works almost in the same way Black does.
Now Ruff replaces Black completely, so please DISABLE the black formatter but ENABLE the ruff formatter in your IDE and editors.

• style: Apply the updated black and ruff rules
  • Now we can use more concise nested parenthesis/braces/brackets inside another one
    when there are no sibling items.
  • Actually this was my preferred style before adopting Black!
• fix: newly found typecheck errors
  • mypy found some potential bugs with its improved type checking!

Tip

This PR changes a wide range of source files as we replace the formatter implementation.

When you encounter conflicts during merging the updated main branch to your topic branch:

• Run git checkout --ours to keep your work.
• Reformat everything by pants fix :: && pants fmt ::.

If you still see previously unobserved type errors, refer the commit 805128c to fix them.

Checklist: (if applicable)

• Milestone metadata specifying the target backport version

[achimnol]

NOTE: require:db-migration is a false-positive as black reformats some alembic migration scripts.

[achimnol]

Currently I'd like to hold merging of this PR due to:

• 🏖️ Black 2024 Preview Style astral-sh/ruff#8678

[achimnol]

Ah, we can use https://docs.astral.sh/ruff/settings/#format-preview !
Ruff now provides an intrinsic formatter that works almost in the same way that Black does, even including some subset of preview styles.

But still there are some deviations, and the outcome is different for some complex (in terms of style) files like src/ai/backend/common/config.py.

[achimnol]

For now, let's stay on the 2023 stable style whereas both ruff and black almost identically works.

[achimnol]

Decided to commence a complete transition from black to ruff in favor of speed. (We are mono-repo with 670+ files to lint.)

Backported and vendored the Ruff plugin with proper "formatter" support from Pantsbuild 2.20.0dev0. Now all our pants lint, pants fix, pants fmt works with Ruff.