Datasets: Can't add S3 external files

[freddessert]

Describe the bug

S3 External files can no longer be added to Datasets.

To reproduce

Add an external S3 file (an image from a public dataset) to a dataset like the following:

dataset = Dataset.get(dataset_id="<some_id>")
s3_file = "s3://cellpainting-gallery/cpg0006-miami/broad/images/2021_08_23_Batch12/images/BR00126103__2021-09-01T23_54_39-Measurement1/Images/r02c18f06p01-ch3sk1fk1fl1.tiff"
dataset.add_external_files(source_url=s3_file, dataset_path="data/painting.png")

Notice you get the following error:

Traceback (most recent call last):
  File "test_datasets.py", line 30, in <module>
    dataset.add_external_files(source_url=s3_file, dataset_path="data/painting.png")
  File "/home/frederic/.local/lib/python3.8/site-packages/clearml/datasets/dataset.py", line 475, in add_external_files
    num_added_this_call, num_modified_this_call = future_.result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 437, in result
    return self.__get_result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/usr/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/frederic/.local/lib/python3.8/site-packages/clearml/datasets/dataset.py", line 3035, in _add_external_files
    relative_path = link[len(source_url):]
TypeError: 'NoneType' object is not subscriptable

The bug is when creating the link object it fails to correctly fetch the file name, which is found at the key key, not the name key for S3 objects. This happens in helper.py in the get_object_metadata method.

This works as expected in SDK Version 1.7.2 FYI.

Expected behaviour

You should be able to add an S3 file as an external file like you can do with GCS.

Environment

• Server type self hosted
• ClearML SDK Version: 1.8.3
• ClearML Server Version: WebApp: 1.6.0-213 • Server: 1.6.0-213 • API: 2.20
• Python Version: 3.8
• OS (Linux)

Related Discussion

If this continues a slack thread, please provide a link to the original slack thread.

[erezalg]

Hi @freddessert,

Ouchy... It seems like a regression was introduced here.
I'll make sure to release an rc ASAP with a fix!

Thanks!

[erezalg]

@freddessert
can you please install clearml 1.8.4rc0 (pip install clearml==1.8.4rc0) and see if it solves your issue?

[freddessert]

@erezalg It works as expected on 1.8.4rc0!

[erezalg]

@freddessert Awesome!! Happy to hear :)

[jkhenning]

Closing this issue as v1.9.3 is already out 🙂. Please reopen if it's still relevant.