Standardize airflow build process and switch to Hatchling build backend

[potiuk]

This PR changes Airflow installation and build backend to use new
standard Python ways of building Python applications.

We've been trying to do it for quite a while. Airflow tranditionally
has been using complex and convoluted build process based on
setuptools and (extremely) custom setup.py file. It survived
migration to Airflow 2.0 and splitting Airlfow monorepo into
Airflow and Providers, adding pre-installed providers and switching
providers to use flit (and follow build standards).

So far tooling in Python ecosystme had not been able to fuflill our
needs and we refrained to develop our own tooling, but finally with
appearance of Hatch (managed by Python Packaging Authority) and
few recent advancements there we are finally able to swtich to
Python standard ways of managing project dependnecy configuration
and project build setup (with a few customizations).

This PR makes airflow build process follow those standard PEPs:

• Airflow has all build configuration stored in pyproject.toml
  following PEP 518 which allows any fronted (pip, poetry,
  hatch, flit, or whatever other frontend is used to
  install required build dependendencies to install Airflow
  locally and to build distribution pacakges (sdist/wheel)

• Hatchling backend follows PEP 517 for standard source tree and build
  backend implementation that allows to execute the build in a
  frontend-independent way

• We store all project metadata in pyprooject.toml - following
  PEP 621 where all necessary project metadata components were
  defined.

• We plug-in into Hatchling "editable build" hooks following
  PEP 660. Hatchling internally builds editable wheel that
  is used as ephemeral step and communication between backend
  and frontend (and this ephemeral wheel is used to make
  editable installation of the projeect - suitable for fast
  iteration of code without reinstalling the package.

With Airflow having many provider packages in single source tree
where we want to be able to install and develop airflow and
providers together, this is not a small feat to implement the
case wher editable installation has to behave quite a bit
differently when it comes to packaging and dependencies for
editable install (when you want to edit sources directly) and
installable package (where you want to have separate Airflow
package and provider packages). Fortunately the standardisation
efforts in the Python Packaging community and tooling implementing
it had finally made it possible.

Some of the important ways bow this has been achieved:

• We continue using provider.yaml in providers as the single source
  of trutgh for per-provider dependencies. We added a possibility
  to specify "devel-dependencies" in provider.yaml so that all
  per-provider dependencies in generated/provider_dependencies.json
  and pyproject.toml are generated from those dependencies via
  update-providers-dependencies pre-commit.

• Pyproject.toml is generally managed manually, but the part where
  provider dependencies and bundle dependencies are used is
  automatically updated by a pre-commit whenever provider
  dependencies change. Those generated provider dependencies contain
  just dependencies of providers - not the provider packages, but
  in the final "standard" wheel file they are replaced with
  "apache-airflow-providers-PROVIDER" dependencies - so that the
  wheel package will only install the provider and use the
  dependencies of that version of provider it installs.

• We are utilising custom hatchiling build hooks (PEP 660 standard)
  that allow to modify 'standard' wheel package on-the-fly when
  the wheel is being prepared by adding preinstalled package
  dependencies (which are not needed in editable build) and by
  removing all devel extras (that are not needed in the PyPI
  distributed wheel package). This allows to solve the conundrum
  of having different "editable" and "standard" behaviour while
  keeping the same project specification in pyproject.toml.

• We added description of how Hatch can be employed as build
  frontend in order to manage local virtualenv and install Airflow
  in editable way easily - while keeping all properties of the
  installed application (including working airflow cli and
  package metadata discovery) as well as how to use PEP-standard
  ways of bulding wheel and sdist packages.

• We have a custom step (following PEP-standards) to inject
  airflow-specific build steps - compiling www assets and
  generating git commit hash version to display it in the UI

• We also show how all this makes it possible to make it easy to
  manage local virtualenvs and editable installations for Airflow
  contributors - without vendor lock-in of the build tools as
  by following standard PEPs Airflow can be locally and editably
  installed by anyone using any build front-end tools following
  the standards - whether you use pip, poetry, Hatch, flit
  or any other frontent build tools, Airflow local installation
  and package building will work the same way for all of them,
  where both "editable" and "standard" package prepration is
  managed by hatchling backend in the same way.

• Previously our extras contained a "." which is not normalized
  name for extras - pip and other tools replaced it automatically
  with `_'. This change updates the extra names to contain
  '-' rather than '.' in the name, following PEP-685. This should be
  fully backwards compatible, users will still be able to use "." but it
  will be normalized to "-" in Airflow packages. This is also future
  proof as it is expected that all package managers and tools
  will eventually use PEP-685 applied to extras, even if currently
  some of the tools (pip + setuptools) might generate warnings.

• Additionally, this change organizes the documentation around
  the extras and dependencies, explaining the reasoning behind
  all the different extras we have.

• As a bonus (and this is what we used to test it all) we are
  documenting how to use Hatch frontend to:

  • manage multiple Python installations
  • manage multiple Pythob virtualenv environments
  • build Airflow packages for release management

Fixes: #30764

[potiuk]

The failures are expected. Im still solving some problems but all the big ones are solved an I successfully run it locally both on Mac and Linux in various combinations and I am really confident it works :)

[potiuk]

BTW. After a good night sleep I got an idea and I've figured a way to simplify it even more - I think I can make it so that I do not need editable version of dependencies at all ( following the pattern tha I did with preinstalled dependencies - I think our pyproject.toml should only contain the provider dependences in [provider] extra and I will be able to dynamically replace them with apache-airflow-providers-provider in the .whl (and standard installation which also uses the same standard wheel) ,

This way there will be literally 0 changes for the contributor workflow:

This one will install amazon dependencies:

pip install -e `.[amzaon]`

But these will install amazon provider:

pip install `.[amazon]`

pip install `apaache-airflow[amazon]`

I will try to get it working today actually :) @uranusjr

[jscheffl]

Mainly I was reading the docs w/o testing locally if all works - anyway as being DRAFT state and CI is read at the moment. But wanted to leave early feedback. Otherwise while reading I am now joining the "convinced" crowd :-D

[potiuk]

Thanks @jscheffl for VERY THOROUGH review. I addressed most comments (and the new iteration is way simpler and automatically addressed some of your comments regardles). I left a few unresolved conversations when things should be fixed before "undrafting" it.

Pushed new version, and next step of this PR is to make the build green.