Delta Lake connector doesn't remove content of MANAGED_TABLE Glue tables created by Databricks

[findinpath]

Scenario:

Databricks SQL

CREATE SCHEMA my_schema LOCATION "s3://trino-ci-test/my_schema/";


CREATE TABLE my_schema.my_table (a, b) %s AS VALUES (1, 2), (2, 3), (3, 4)

Trino

DROP TABLE delta.my_schema.my_table;

The operation succeeds in Trino. However the S3 content of the table is still present on AWS S3 after completing this operation.
This means that there can't be created new tables having the same name as the previously dropped table, because the directory corresponding to the table already exists.

Context:

aws s3api head-object --bucket trino-ci-test --key my_schema/my_table/_delta_log/

{
    "AcceptRanges": "bytes",
    "LastModified": "2022-06-28T11:44:47+00:00",
    "ContentLength": 0,
    "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
    "ContentType": "application/octet-stream",
    "Metadata": {}
}

The above mentioned code corresponds to what is happening in

trino/plugin/trino-hive/src/main/java/io/trino/plugin/hive/s3/TrinoS3FileSystem.java

Lines 735 to 745 in fe608f2

	ObjectMetadata getS3ObjectMetadata(Path path)
	throws IOException
	{
	String bucketName = getBucketName(uri);
	String key = keyFromPath(path);
	ObjectMetadata s3ObjectMetadata = getS3ObjectMetadata(path, bucketName, key);
	if (s3ObjectMetadata == null && !key.isEmpty()) {
	return getS3ObjectMetadata(path, bucketName, key + PATH_SEPARATOR);
	}
	return s3ObjectMetadata;
	}

As seen in the listing above, the ContentType of this "object" is set to application/octet-stream which is not the same as application/x-directory

trino/plugin/trino-hive/src/main/java/io/trino/plugin/hive/s3/TrinoS3FileSystem.java

Line 425 in fe608f2

MediaType.parse(metadata.getContentType()).is(DIRECTORY_MEDIA_TYPE),

This interesting detail breaks the logic used for deleting Delta Lake table directories in AWS S3.

[ebyhr]

Filed https://community.databricks.com/s/question/0D58Y000093AR43SAG/how-to-identify-s3-object-type-directory-or-file-created-by-databricks. I will update this issue once I got the response from Databricks community.