[Model WatchTower] Delete running versions on failed pipelines with delete_new_version_on_failure option

[avishniakov]

Describe changes

I implemented a clean-up step delete_running_versions_without_recovery in pipleine.run_ - it runs on the failure of the pipeline and cleans up all hanging ModelVersions in running created during the pipeline run, but not having a delete_new_version_on_failure option.

Here we have one conceptual renaming: recovery of ModelConfig becomes delete_new_version_on_failure to avoid confusion on model version retrieval logic going forward.

Also inside Pipeline new method get_new_version_requests is added to fetch all model configurations across the pipeline requesting new versions to be created. This aims for several goals:

• Output of it is passed to register_running_versions on success to make running versions normal numbered versions or to delete_running_versions_without_recovery on failure to clean-up versions not requesting to keep them on failure.
• It warns users if all steps inside the pipeline are defining their own model configuration and overrides one defined in the pipeline if defined.
• It warns users that several model configurations (in pipeline and step(s) or in several steps) request new model version creation for the same model - this is not impacting logic, still, one version will be created, but this is a poor design pattern and the new version should be requested in one place only.

I also touched a bit ModelVersionResponseModel to better support versioned artifacts:

• artifact_object_ids, deployment_ids and model_object_ids becomes Dict[str, Dict[str, UUID]]: {"artifact_name":{"1":[...],"2":[...]}}, where 1 and 2 are artifact link versions for versioned artifact
• get_model_object and others also accept version now

The issue with this implementation to be fixed in #1839:

• Using artifact_name as a key in artifact_object_ids and others of ModelVersionResponseModel causes collisions with not uniquely named artifacts

Pre-requisites

Please ensure you have done the following:

• I have read the CONTRIBUTING.md document.
• If my change requires a change to docs, I have updated the documentation accordingly.
• If I have added an integration, I have updated the integrations table and the corresponding website section.
• I have added tests to cover my changes.
• I have based my new branch on develop and the open PR is targeting develop. If your branch wasn't based on develop read Contribution guide on rebasing branch to develop.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Other (add details above)

[fa9r]

See unresolved comments from last review

[fa9r]

Nice, thanks for adjusting, looks good to me now 👍

[avishniakov]

Test failure caused by not stable behavior of MySQL due to the way or storing datetime. Fixed that, but will not wait for full CI once again.