Make Dask configurable in our configuration

[schlunma]

Description

This PR makes Dask configurable in our configuration. These are the default settings:

dask:
  client: {}  # keyword arguments for distributed.Client, can include "address" to use external cluster
  clusters:
    default:
      type: default
    debug:
      type: default
      scheduler: synchronous
    local:
      type: distributed.LocalCluster
      n_workers: 2
      threads_per_worker: 2
      memory_limit: 4GiB
  config: {}  # keyword arguments for dask.config.set
  run: default  # Start the `default` cluster defined above

The entries given under clusters can be selected via the run option; this also works in the command line, e.g.,

esmvaltool run --dask='{"run": "local"}' recipe_example.yml

I chose not to add a predefined SLURMCluster setting since this needs to be customized by the user anyway (account and scratch directory).

This PR also deprecates the usage of the file ~/.esmvaltool/dask.yml (see below).

Closes #2040
Closes #2369

Link to documentation: https://esmvaltool--2616.org.readthedocs.build/projects/ESMValCore/en/2616/quickstart/configure.html#dask-configuration

Deprecation (since v2.12.0, will be removed in v2.14.0)

Usage of Dask configuration file ~/.esmvaltool/dask.yml: please use the new configuration option dask instead, see details above. If the deprecated file is present, it will be used but a warning will be printed. In this case, all dask configuration options are ignored. To force using the new Dask configuration option and ignore ~/.esmvaltool/dask.yml already now, you can set the environment variable ESMVALTOOL_USE_NEW_DASK_CONFIG to any truthy value.

---

Before you get started

• ☝ Create an issue to discuss what you are going to do

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

• 🧪 The new functionality is relevant and scientifically sound
• 🛠 This pull request has a descriptive title and labels
• 🛠 Code is written according to the code quality guidelines
• 🧪 and 🛠 Documentation is available
• 🛠 Unit tests have been added
• 🛠 Changes are backward compatible
• 🛠 Any changed dependencies have been added or removed correctly
• 🛠 The list of authors is up to date
• 🛠 All checks below this pull request were successful

---

To help with the number pull requests:

• 🙏 We kindly ask you to review two other open pull requests in this repository

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 94.95%. Comparing base (6532a0a) to head (e1acf51).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2616      +/-   ##
==========================================
+ Coverage   94.93%   94.95%   +0.02%     
==========================================
  Files         252      252              
  Lines       14620    14681      +61     
==========================================
+ Hits        13880    13941      +61     
  Misses        740      740              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.