Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incorrect detection of directories in AzureBlobPath #444

Closed
M0dEx opened this issue Jul 2, 2024 · 1 comment
Closed

Incorrect detection of directories in AzureBlobPath #444

M0dEx opened this issue Jul 2, 2024 · 1 comment
Labels
bug Something isn't working

Comments

@M0dEx
Copy link

M0dEx commented Jul 2, 2024

Currently, the detection of whether a given path points to a blob (file) or a directory is based mostly on the presence of its metadata:

def _is_file_or_dir(self, cloud_path: AzureBlobPath) -> Optional[str]:
# short-circuit the root-level container
if not cloud_path.blob:
return "dir"
try:
self._get_metadata(cloud_path)
return "file"
except ResourceNotFoundError:
prefix = cloud_path.blob
if prefix and not prefix.endswith("/"):
prefix += "/"
# not a file, see if it is a directory
container_client = self.service_client.get_container_client(cloud_path.container)
try:
next(container_client.list_blobs(name_starts_with=prefix))
return "dir"
except StopIteration:
return None

This is not correct, as directories can have non-empty metadata present (and in newer versions of Azure Blob Storage API they almost always do), which leads to incorrect is_file and is_dir results.

@M0dEx M0dEx changed the title Incorrect detection of directories in AzureCloudPath Incorrect detection of directories in AzureBlobPath Jul 2, 2024
@jayqi jayqi added the bug Something isn't working label Jul 3, 2024
@pjbull
Copy link
Member

pjbull commented Aug 29, 2024

This should be part of the ADLS Gen2 support released now in v0.19.0 if you want to upgrade and test your use case

@pjbull pjbull closed this as completed Aug 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
3 participants