PyPI v1.14.2 - ClearML

New Features and Improvements

• Report upload/download with tqdm if installed
• Add support for a PipelineController.version property
• Add support for specifying an AWS profile when providing bucket storage credentials
• Add an ignore_remote_overrides argument to Task.connect() and Task.connect_configuration() allowing users to disable overriding values when running remotely
• Add support for setting a custom certificate path using the CLEARML_API_HOST_VERIFY_CERT environment variable (instead of just allowing the user to disable verification)
• Add dataset reporting example (#1180)
• Update README GIFs (#1191)

Bug Fixes

• Fix missing component callbacks on multiple step calls (#1195, thanks @materight!)
• Fix unsafe usage of tempfile.mktemp
• Fix potential path traversal on file download (CVE-2024-24591)
• Fix Python 3.12 support by not using distutils
• Fix TaskScheduler does not save local state before running remotely
• Fix pipeline should continue instead of aborting when PipelineController.abort_on_failure is set to False and a step has been aborted
• Fix HPO crashes when its corresponding task is initialized with continue_last_task=True

PyPI v1.14.1 - ClearML

Bug Fixes

• Fix created pipelines do not appear in the UI pipelines section

PyPI v1.14.0 - ClearML

New Features and Improvements

• Add support for decorated pipeline steps (#1154)
• Improve torch.distributed behavior and auto-detect Huggingface accelerate distributed execution
• Improve jsonargparse integration: allow_config_file_override_from_ui argument added for enabling/disabling overriding configuration file entries
• Add raise_on_error parameter to Model.update_weights() and Model.update_weights_package()
• Add HPO multi-objective optimization support by specifying multiple optimization objectives to the HyperParameterOptimizer class
• Improve pipelines from functions: libraries used by pipeline steps no longer have to be imported in the step itself and can be imported globally instead
• Support clearing repository details in task.set_repo()
• Auto detect tensorflow-intel as tensorflow
• Add support for auto detecting torch and transformers accelerate distributed execution
• Add tabular data multi-stage pipeline process example

Bug Fixes

• Fix langchain integration (#1124, #1126)
• Fix using target_project with TaskScheduler.add_task() corrupts project if used with pipelines (#1137)
• Fix PipelineController.add_function_step() fails when adding custom decorated step (#1154)
• Fix jsonargparse binding does not capture parameters before Task.init() is called (#1164)
• Fix invalid jsonargparse arguments should be deleted
• Fix handling of appended parameters to Hydra defaults list
• Fix Task.set_repo() and Task.set_packages() do not wait for automatic asynchronous repository detection
• Fix active GPU filtering in resource monitor to support GPU ID and not just GPU idx
• Fix misleading error raised when using the SDK with no configuration file or an invalid configuration file
• Fix process sometimes hangs issue by improving exit and exception handlers, unregistering signal handling in child processes except for the BackgroundMonitor

PyPI v1.13.2 - ClearML

New Features

• Support parallel dataset verification (#1131, thanks @charlienewey-odin!)
• Support lightgbm model loading from a string (#1136, thanks @bedapisl!)
• Add support for Hydra command-line syntax for modifying omegaconf
• Add support for setting output_uri in pipeline decorators and functions
• Add more visibility when overriding jsonargparse arguments
• Use StorageManager cache when downloading models
• Support parsing GS credentials as a JSON string if they do not point to a JSON file
• Support ignoring default_output_uri on remote runs when Task.init(output_uri=False) was specified and the backend task data has no output_uri specified
• Add configuration files logging example
• Support raising LoginError when initializing
• Add HuggingFace Transformers Example (#1146)
• Improve docstrings for tag filters
• Add a security policy file (#1148)

Bug Fixes

• Fix Google Storage subdir path in bucket URL breaks validation (#1117, thanks @materight!)
• Fix jsonschema DeprecationWarning in newer versions (#1128, thanks @xquyvu!)
• Fix task.set_script(diff="") does not clear the uncommitted changes (#1106)
• Fix parallel dataset verification fails on older Python versions (#1144)
• Fix attrs.__version__ deprecation warning (#1145)
• Fix Task.get_parameters_as_dict(cast=True) raises a YAML parser error
• Fix potential threading issues when setting self.lock to None
• Fix pipeline version is not displayed in the table view and version list
• Fix displayed pipeline run time in pipeline components is not the same as that shown in the info tab in full detail
• Fix Model.connect() in remote execution might result in the wrong model being connected
• Fix exception raised on ConnectionError (which actually indicates that a read has timed out, and should be retried)
• Fix missing configuration error not raised in certain cases
• Fix loading queues containing metadata fails due to an incorrect queue metadata scheme
• Fix pyjwt requirements for Python < 2.9
• Fix pipeline examples

PyPI v1.13.1 - ClearML

Bug Fixes

• Fix Hydra overrides integration (#1120)
• Fix GPU memory used reports 0 when memory can not be queried per process (edge case)

PyPI v1.13.0 - ClearML

New Features

• Add support for recursive list, dict, and tuple ref parsing for pipeline controller.add step() parameter overrides (#1099, thanks @natephysics!)
• Update PyNVML to the latest NVIDIA version for better GPU monitoring
• Add force_single_script_file argument to Task.create() to avoid git repository auto detection
• Use os.register_at_fork instead of monkey patching fork for python > 3.6
• Add support for programmatically archive and unarchive models from the model registry using the Model.archive() and Mode.unarchive() methods (#1096)
• Improve documentation

Bug Fixes

• Fix Hydra overrides integration (#552)
• Fix model reporting in TensorFlow 2.13 does not work properly (#1112)
• Fix GPU memory used reports 0 when memory can not be queried per process
• Fix FastAI binding not reporting scalars if TensorBoard is installed
• Fix JSON serialization error does not raise exception
• Fix OutputModel should prefer connected task name over filename stem
• Fix dataset zip extraction might fail when creating folders concurrently (with Exception [Errno 17] File exists warning)
• Fix import sys should not appear in pipeline component auto detected packages
• Fix compatibility issue with requests < 2.28

PyPI v1.12.2 - ClearML

New Features and Improvements

• Add always_create_from_code argument to PipelineController (default True) to allow remote execution to create the Pipeline DAG at runtime (as opposed to adhering to the existing structure stored on the task when running locally)

Bug Fixes

• Fix using OutputModel reporting and other methods fail if OutputModel.update_weights() is not called before that (#1078)
• Fix hash for Pandas objects artifacts
• Fix Hydra/OmegaConf --overrides support
• Fix Pipeline metrics not forwarded to pipeline controller at the end of a pipeline step

PyPI v1.12.1 - ClearML

New Features and Improvements

• Raise error if dataset name is empty (#1083, thanks @mathematicalmichael!)
• Add support for spawn start method for Python multiprocess. This should help circumvent issues like this
• Improve docstrings

Bug Fixes

• Fix S3 uploads fail with LazyEvalWrapper type error (#1081)

PyPI v1.12.0 - ClearML

Breaking Changes

• As a result of fixing #1051 and similar issues with retrieving task data/models/artifacts/samples stored in GCS we introduced some breaking changes. To access resources registered to GCS from ClearML SDK versions 1.11.0 to 1.11.1, see the instructions here.

New Features and Improvements

• Add include_archive parameter to Dataset.list_datasets(), now possible to include archived datasets in the returned list (#1069, thanks @natephysics!)
• Add support to specify the multipart threshold and chunk size using the aws.boto3.multipart_chunksize and aws.boto3.multipart_threshold configuration options (#1059, thanks @cgaudreau-ubisoft!)
• Add PipelineController.get_pipeline() for retrieving previously run pipelines

Bug Fixes

• Fix AWS driver issues (#1000, thanks @achaiah!)
  • Fix credential authentication failure when attempting to use token
  • Fix instantiation within VPC without AvailabilityZones
• Fix continue_last_task=0 is ignored in pipelines run with retry_on_failure (#1054)
• Fix Task.connect_configuration() doesn’t handle dictionaries with special characters
• Fix pipeline steps created with PipelineDecorator aren't cached
• Fix Task.get_by_name() doesn't return the most recent task when multiple tasks have same name
• Fix boto3 certificate verification ignores custom self-signed certificates

PyPI v1.11.1 - ClearML

New Features and Improvements

• Support lightning>=2.0 (#1033, thanks @aweinmann!)
• Add clearml-init support for standard HTTP and HTTPS ports in webapp in conjunction with non-default api/files server ports (#1031, thanks @pktiuk!)
• Improve docstrings

Bug Fixes

• Fix remote execution does not log configuration values from modified jsonargparse configuration files
• Fix ValueError when setting task properties at the start of a pipeline component
• Fix OS exceptions on passing Path where str expected
• Fix infinite recursion when remotely updating a dict added to task with Task.connect()
• Fix Task.connect race condition overwriting task description with connected input model