Ozone S3 + Spark (S3A filesystem) Issues with bucket operations #7050

mcgrawia · 2024-08-08T15:51:15Z

mcgrawia
Aug 8, 2024

Hi,

I'm trying to use Ozone for the first time and running into issues with Spark / S3 compatibility and was hoping someone here may be able to help me. Thanks in advance!

Summary of problem

My Spark job attempts to overwrite a Spark ModelPipeline object:

LOGGER.info("Saving pipeline")
val baseS3Path = getS3ModelBasePath(modelId, orgId)
endToEndPipelineModel.write.overwrite.save(baseS3Path + "isolation_forest_pipeline")

This jobs fails with the following errors:

24/08/08 15:12:56 INFO TrainIsolationForest: Saving pipeline
24/08/08 15:12:56 INFO FileSystemOverwrite: Path s3a://bucket/anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline already exists. It will be overwritten.
24/08/08 15:12:57 INFO BlockManagerInfo: Removed broadcast_59_piece0 on 10.129.1.229:51248 in memory (size: 7.9 KiB, free: 434.3 MiB)
24/08/08 15:12:57 INFO BlockManagerInfo: Removed broadcast_59_piece0 on 10.128.1.127:51248 in memory (size: 7.9 KiB, free: 116.9 MiB)
24/08/08 15:12:57 INFO BlockManagerInfo: Removed broadcast_59_piece0 on 10.128.1.126:51248 in memory (size: 7.9 KiB, free: 116.9 MiB)
24/08/08 15:12:57 INFO MultiObjectDeleteSupport: Bulk delete operation failed to delete all objects; failure count = 3
24/08/08 15:12:57 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline

24/08/08 15:12:57 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata

24/08/08 15:12:57 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/_temporary/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/_temporary

24/08/08 15:12:57 INFO BlockManagerInfo: Removed broadcast_57_piece0 on 10.129.1.229:51248 in memory (size: 54.7 KiB, free: 434.4 MiB)
24/08/08 15:12:57 INFO BlockManagerInfo: Removed broadcast_57_piece0 on 10.128.1.127:51248 in memory (size: 54.7 KiB, free: 117.0 MiB)
24/08/08 15:12:57 INFO BlockManagerInfo: Removed broadcast_57_piece0 on 10.128.1.126:51248 in memory (size: 54.7 KiB, free: 117.0 MiB)
24/08/08 15:12:58 INFO MultiObjectDeleteSupport: Bulk delete operation failed to delete all objects; failure count = 2
24/08/08 15:12:58 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline

24/08/08 15:12:58 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata

24/08/08 15:12:59 INFO MultiObjectDeleteSupport: Bulk delete operation failed to delete all objects; failure count = 1
24/08/08 15:12:59 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline

24/08/08 15:13:03 INFO deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
24/08/08 15:13:03 INFO HadoopMapRedCommitProtocol: Using output committer class org.apache.hadoop.mapred.FileOutputCommitter
24/08/08 15:13:03 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
24/08/08 15:13:03 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
24/08/08 15:13:03 INFO MultiObjectDeleteSupport: Bulk delete operation failed to delete all objects; failure count = 7
24/08/08 15:13:03 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/_temporary/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/_temporary

24/08/08 15:13:03 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata

24/08/08 15:13:03 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline

24/08/08 15:13:03 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359

24/08/08 15:13:03 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0

24/08/08 15:13:03 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/: Directory is not empty. Key:anomaly_detection/isolation_forest

24/08/08 15:13:03 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/: Directory is not empty. Key:anomaly_detection

24/08/08 15:13:04 INFO MultiObjectDeleteSupport: Bulk delete operation failed to delete all objects; failure count = 7
24/08/08 15:13:04 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/_temporary/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/_temporary

24/08/08 15:13:04 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata

24/08/08 15:13:04 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline

24/08/08 15:13:04 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359

24/08/08 15:13:04 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0

24/08/08 15:13:04 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/: Directory is not empty. Key:anomaly_detection/isolation_forest

24/08/08 15:13:04 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/: Directory is not empty. Key:anomaly_detection

24/08/08 15:13:06 INFO MultiObjectDeleteSupport: Bulk delete operation failed to delete all objects; failure count = 7
24/08/08 15:13:06 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/_temporary/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/_temporary

24/08/08 15:13:06 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata

24/08/08 15:13:06 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline

24/08/08 15:13:06 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359

24/08/08 15:13:06 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0

24/08/08 15:13:06 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/: Directory is not empty. Key:anomaly_detection/isolation_forest

24/08/08 15:13:06 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/: Directory is not empty. Key:anomaly_detection

24/08/08 15:13:11 INFO MultiObjectDeleteSupport: Bulk delete operation failed to delete all objects; failure count = 7
24/08/08 15:13:11 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/_temporary/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/_temporary

24/08/08 15:13:11 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata

24/08/08 15:13:11 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline

24/08/08 15:13:11 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359

24/08/08 15:13:11 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0

24/08/08 15:13:11 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/: Directory is not empty. Key:anomaly_detection/isolation_forest

24/08/08 15:13:11 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/: Directory is not empty. Key:anomaly_detection

24/08/08 15:13:22 INFO MultiObjectDeleteSupport: Bulk delete operation failed to delete all objects; failure count = 7
24/08/08 15:13:22 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/_temporary/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/_temporary

24/08/08 15:13:22 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata

24/08/08 15:13:22 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline

24/08/08 15:13:22 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359

24/08/08 15:13:22 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0

24/08/08 15:13:22 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/: Directory is not empty. Key:anomaly_detection/isolation_forest

24/08/08 15:13:22 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/: Directory is not empty. Key:anomaly_detection

24/08/08 15:13:30 INFO MultiObjectDeleteSupport: Bulk delete operation failed to delete all objects; failure count = 7
24/08/08 15:13:30 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/_temporary/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/_temporary

24/08/08 15:13:30 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata

24/08/08 15:13:30 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline

24/08/08 15:13:30 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359

24/08/08 15:13:30 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0

24/08/08 15:13:30 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/: Directory is not empty. Key:anomaly_detection/isolation_forest

24/08/08 15:13:30 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/: Directory is not empty. Key:anomaly_detection

24/08/08 15:14:04 INFO MultiObjectDeleteSupport: Bulk delete operation failed to delete all objects; failure count = 7
24/08/08 15:14:04 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/_temporary/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/_temporary

24/08/08 15:14:04 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata

24/08/08 15:14:04 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline

24/08/08 15:14:04 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359

24/08/08 15:14:04 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0

24/08/08 15:14:04 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/: Directory is not empty. Key:anomaly_detection/isolation_forest

24/08/08 15:14:04 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/: Directory is not empty. Key:anomaly_detection

When I list the contents of the bucket, I see a lot of empty files, which seem to be preventing the MultiObjectDelete command from succeeding:

aws s3 ls --recursive --endpoint http://localhost:9878 s3://bucket/anomaly_detection/
2024-08-08 11:03:25          0 anomaly_detection/
2024-08-08 11:03:25          0 anomaly_detection/isolation_forest/
2024-08-08 11:03:25          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/
2024-08-08 11:03:25          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/
2024-08-08 11:03:25          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/
2024-08-08 11:03:25          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/
2024-08-08 11:03:25          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/_temporary/
2024-08-08 11:03:25          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/_temporary/0/

Troubleshooting steps

I've also tried creating my bucket using the --layout OBJECT_STORE format. This seems to cause a different issue. I no longer see the error above, but my spark job eventually fails with this log:

Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Server Error (Service: Amazon S3; Status Code: 500; Error Code: 500 Server Error; Request ID: null; S3 Extended Request ID: null; Proxy: null), S3 Extended Request ID: null

When I look into the S3G pod, I see the following logs:

2024-08-08 14:37:14,675 [qtp313082880-328] WARN  server.HttpChannel (HttpChannel.java:handleException(776)) - /bucket
javax.servlet.ServletException: javax.servlet.ServletException: java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:162)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:516)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: javax.servlet.ServletException: java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:410)
	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:358)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:311)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
	at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1459)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
	at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656)
	at org.apache.hadoop.ozone.s3.RootPageDisplayFilter.doFilter(RootPageDisplayFilter.java:53)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at org.apache.hadoop.ozone.s3.EmptyContentTypeFilter.doFilter(EmptyContentTypeFilter.java:76)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at org.apache.hadoop.hdds.server.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1665)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at org.apache.hadoop.hdds.server.http.NoCacheFilter.doFilter(NoCacheFilter.java:48)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
	... 17 more
Caused by: java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
	at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)
	at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)
	at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248)
	at java.base/java.util.Objects.checkIndex(Objects.java:372)
	at java.base/java.util.ArrayList.remove(ArrayList.java:536)
	at org.apache.hadoop.ozone.client.OzoneBucket$KeyIterator.getNextShallowListOfKeys(OzoneBucket.java:1244)
	at org.apache.hadoop.ozone.client.OzoneBucket$KeyIterator.getNextListOfKeys(OzoneBucket.java:1150)
	at org.apache.hadoop.ozone.client.OzoneBucket$KeyIterator.hasNext(OzoneBucket.java:1124)
	at org.apache.hadoop.ozone.s3.endpoint.BucketEndpoint.get(BucketEndpoint.java:212)
	at jdk.internal.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189)
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:256)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235)
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684)
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
	... 51 more
2024-08-08 14:37:14,675 [qtp313082880-328] WARN  server.HttpChannelState (HttpChannelState.java:onError(787)) - unhandled due to prior sendError
javax.servlet.ServletException: javax.servlet.ServletException: java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:162)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:516)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: javax.servlet.ServletException: java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:410)
	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:358)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:311)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
	at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1459)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
	at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656)
	at org.apache.hadoop.ozone.s3.RootPageDisplayFilter.doFilter(RootPageDisplayFilter.java:53)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at org.apache.hadoop.ozone.s3.EmptyContentTypeFilter.doFilter(EmptyContentTypeFilter.java:76)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at org.apache.hadoop.hdds.server.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1665)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at org.apache.hadoop.hdds.server.http.NoCacheFilter.doFilter(NoCacheFilter.java:48)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
	... 17 more
Caused by: java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
	at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)
	at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)
	at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248)
	at java.base/java.util.Objects.checkIndex(Objects.java:372)
	at java.base/java.util.ArrayList.remove(ArrayList.java:536)
	at org.apache.hadoop.ozone.client.OzoneBucket$KeyIterator.getNextShallowListOfKeys(OzoneBucket.java:1244)
	at org.apache.hadoop.ozone.client.OzoneBucket$KeyIterator.getNextListOfKeys(OzoneBucket.java:1150)
	at org.apache.hadoop.ozone.client.OzoneBucket$KeyIterator.hasNext(OzoneBucket.java:1124)
	at org.apache.hadoop.ozone.s3.endpoint.BucketEndpoint.get(BucketEndpoint.java:212)
	at jdk.internal.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189)
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:256)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235)
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684)
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
	... 51 more

Looking using the AWS CLI, I can see there seems to be a directory marker file in the bucket:

aws s3 ls --recursive --endpoint http://localhost:9878 s3://bucket/anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=3c1e36b1-6be4-448c-87a2-de0def6edafc/
2024-08-08 10:00:56          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=3c1e36b1-6be4-448c-87a2-de0def6edafc/

But when I attempt to get this file, the AWS CLI fails:

aws s3 ls --endpoint http://localhost:9878 s3://bucket/anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=3c1e36b1-6be4-448c-87a2-de0def6edafc/

An error occurred (500) when calling the ListObjectsV2 operation (reached max retries: 2): Internal Server Error

Some Background / Setup Details

I'm running Ozone via the Ozone helm chart in Kubernetes with default values, using version 1.4.0:

helm upgrade --install -n default ozone ./charts/ozone

I created my bucket with the following commands:

ozone sh volume create /s3v
ozone sh volume create /vol1

ozone sh bucket create /vol1/bucket
ozone sh bucket link /vol1/bucket /s3v/bucket

and set the S3G domain in my ozone-site.xml:

<property><name>ozone.s3g.domain.name</name><value>s3g.internal</value></property>

My current Spark S3A settings are the following:

spark.hadoop.com.amazonaws.services.s3a.enableV4             true
spark.hadoop.fs.s3a.impl                                     org.apache.hadoop.fs.s3a.S3AFileSystem
spark.hadoop.fs.s3a.connection.ssl.enabled                   true
spark.hadoop.fs.s3a.path.style.access                        true
spark.hadoop.fs.s3a.change.detection.version.required        false
spark.hadoop.fs.s3a.change.detection.mode                    none
spark.hadoop.fs.s3a.aws.credentials.provider                 org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider,com.amazonaws.auth.DefaultAWSCredentialsProviderChain

Additionally, I have also tried setting spark.hadoop.fs.s3a.directory.marker.retention=delete, and it did not seem to change anything.

Thanks for the help 🙏

mcgrawia · 2024-08-08T16:14:13Z

mcgrawia
Aug 8, 2024
Author

I also tested by setting spark.hadoop.fs.s3a.multiobjectdelete.enable=false, and it still fails with this following error:

24/08/08 16:10:54 ERROR Instrumentation: org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory s3a://bucket/anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata already exists
	at org.apache.hadoop.mapred.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:131)
	at org.apache.spark.internal.io.HadoopMapRedWriteConfigUtil.assertConf(SparkHadoopWriter.scala:299)
	at org.apache.spark.internal.io.SparkHadoopWriter$.write(SparkHadoopWriter.scala:71)
	at org.apache.spark.rdd.PairRDDFunctions.$anonfun$saveAsHadoopDataset$1(PairRDDFunctions.scala:1090)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
	at org.apache.spark.rdd.RDD.withScope(RDD.scala:414)
	at org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopDataset(PairRDDFunctions.scala:1088)
	at org.apache.spark.rdd.PairRDDFunctions.$anonfun$saveAsHadoopFile$4(PairRDDFunctions.scala:1061)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
	at org.apache.spark.rdd.RDD.withScope(RDD.scala:414)
	at org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:1026)
	at org.apache.spark.rdd.PairRDDFunctions.$anonfun$saveAsHadoopFile$3(PairRDDFunctions.scala:1008)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
	at org.apache.spark.rdd.RDD.withScope(RDD.scala:414)
	at org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:1007)
	at org.apache.spark.rdd.PairRDDFunctions.$anonfun$saveAsHadoopFile$2(PairRDDFunctions.scala:964)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
	at org.apache.spark.rdd.RDD.withScope(RDD.scala:414)
	at org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:962)
	at org.apache.spark.rdd.RDD.$anonfun$saveAsTextFile$2(RDD.scala:1578)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
	at org.apache.spark.rdd.RDD.withScope(RDD.scala:414)
	at org.apache.spark.rdd.RDD.saveAsTextFile(RDD.scala:1578)
	at org.apache.spark.rdd.RDD.$anonfun$saveAsTextFile$1(RDD.scala:1564)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
	at org.apache.spark.rdd.RDD.withScope(RDD.scala:414)
	at org.apache.spark.rdd.RDD.saveAsTextFile(RDD.scala:1564)
	at org.apache.spark.ml.util.DefaultParamsWriter$.saveMetadata(ReadWrite.scala:413)
	at org.apache.spark.ml.Pipeline$SharedReadWrite$.$anonfun$saveImpl$1(Pipeline.scala:250)
	at org.apache.spark.ml.Pipeline$SharedReadWrite$.$anonfun$saveImpl$1$adapted(Pipeline.scala:247)
	at org.apache.spark.ml.util.Instrumentation$.$anonfun$instrumented$1(Instrumentation.scala:191)
	at scala.util.Try$.apply(Try.scala:213)
	at org.apache.spark.ml.util.Instrumentation$.instrumented(Instrumentation.scala:191)
	at org.apache.spark.ml.Pipeline$SharedReadWrite$.saveImpl(Pipeline.scala:247)
	at org.apache.spark.ml.PipelineModel$PipelineModelWriter.saveImpl(Pipeline.scala:346)
	at org.apache.spark.ml.util.MLWriter.save(ReadWrite.scala:168)
	at org.apache.spark.ml.PipelineModel$PipelineModelWriter.super$save(Pipeline.scala:344)
	at org.apache.spark.ml.PipelineModel$PipelineModelWriter.$anonfun$save$4(Pipeline.scala:344)
	at org.apache.spark.ml.MLEvents.withSaveInstanceEvent(events.scala:174)
	at org.apache.spark.ml.MLEvents.withSaveInstanceEvent$(events.scala:169)
	at org.apache.spark.ml.util.Instrumentation.withSaveInstanceEvent(Instrumentation.scala:42)
	at org.apache.spark.ml.PipelineModel$PipelineModelWriter.$anonfun$save$3(Pipeline.scala:344)
	at org.apache.spark.ml.PipelineModel$PipelineModelWriter.$anonfun$save$3$adapted(Pipeline.scala:344)
	at org.apache.spark.ml.util.Instrumentation$.$anonfun$instrumented$1(Instrumentation.scala:191)
	at scala.util.Try$.apply(Try.scala:213)
	at org.apache.spark.ml.util.Instrumentation$.instrumented(Instrumentation.scala:191)
	at org.apache.spark.ml.PipelineModel$PipelineModelWriter.save(Pipeline.scala:344)
	at ai.com.spark.jobs.TrainIsolationForest$.saveEndToEndPipeline(TrainIsolationForest.scala:270)
	at ai.com.spark.jobs.TrainIsolationForest$.trainAndSave(TrainIsolationForest.scala:236)
	at ai.com.spark.jobs.TrainIsolationForest$.main(TrainIsolationForest.scala:80)
	at ai.com.spark.jobs.TrainIsolationForest.main(TrainIsolationForest.scala)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:966)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:191)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:214)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1054)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1063)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

It seems like the empty directory markers are what's causing the issue:

aws s3 ls --recursive --endpoint http://localhost:9878 s3://bucket/anomaly_detection/
2024-08-08 12:07:35          0 anomaly_detection/
2024-08-08 12:07:35          0 anomaly_detection/isolation_forest/
2024-08-08 12:07:35          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/
2024-08-08 12:07:35          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/
2024-08-08 12:07:35          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/
2024-08-08 12:07:35          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/
2024-08-08 12:07:35          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/_temporary/
2024-08-08 12:07:39          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/stages/
2024-08-08 12:07:39          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/stages/0_emptystringfiller_1be2e8dde774/
2024-08-08 12:07:39          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/stages/0_emptystringfiller_1be2e8dde774/metadata/
2024-08-08 12:07:39          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/stages/0_emptystringfiller_1be2e8dde774/metadata/_temporary/
2024-08-08 12:07:42          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/stages/1_strIdx_b35c52fd17a7/
2024-08-08 12:07:45          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/stages/1_strIdx_b35c52fd17a7/data/
2024-08-08 12:07:45          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/stages/1_strIdx_b35c52fd17a7/data/_temporary/
2024-08-08 12:07:42          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/stages/1_strIdx_b35c52fd17a7/metadata/
2024-08-08 12:07:42          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/stages/1_strIdx_b35c52fd17a7/metadata/_temporary/
2024-08-08 12:07:47          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/stages/2_oneHotEncoder_2fe087f49c18/
2024-08-08 12:07:49          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/stages/2_oneHotEncoder_2fe087f49c18/data/
2024-08-08 12:07:49          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/stages/2_oneHotEncoder_2fe087f49c18/data/_temporary/
2024-08-08 12:07:47          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/stages/2_oneHotEncoder_2fe087f49c18/metadata/
2024-08-08 12:07:47          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/stages/2_oneHotEncoder_2fe087f49c18/metadata/_temporary/
2024-08-08 12:07:52          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/stages/3_vecAssembler_04a638b48e31/
2024-08-08 12:07:52          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/stages/3_vecAssembler_04a638b48e31/metadata/
2024-08-08 12:07:52          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/stages/3_vecAssembler_04a638b48e31/metadata/_temporary/
2024-08-08 12:07:54          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/stages/4_isolation-forest_dfdfa72ad592/
2024-08-08 12:07:57          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/stages/4_isolation-forest_dfdfa72ad592/data/
2024-08-08 12:07:57          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/stages/4_isolation-forest_dfdfa72ad592/data/_temporary/
2024-08-08 12:07:54          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/stages/4_isolation-forest_dfdfa72ad592/metadata/
2024-08-08 12:07:54          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/stages/4_isolation-forest_dfdfa72ad592/metadata/_temporary/

0 replies

adoroszlai · 2024-08-08T17:01:52Z

adoroszlai
Aug 8, 2024
Collaborator

@mcgrawia A couple of S3A compatibility issues were fixed since 1.4.0, please see HDDS-8423. Especially HDDS-10570 and HDDS-10572 seem to be related to your woes. Can you please try building Ozone from source (master branch), and see if it works better?

1 reply

mcgrawia Aug 8, 2024
Author

Hi @adoroszlai thank you for the quick response. I can try building latest and see if that resolves the issues. Is there a target date for the 1.5.0 release (or maybe a 1.4.1 that has these fixes)? Thanks

tanvipenumudy · 2024-08-08T17:56:24Z

tanvipenumudy
Aug 8, 2024
Collaborator

Caused by: java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0

The IndexOutOfBoundsException while listing OBS buckets I believe has been fixed by HDDS-10399 in Ozone 1.4.1 and 1.5.0.

A tentative workaround is to disable shallow listing by setting the property ozone.s3g.list-keys.shallow.enabled to false.

cc: @SaketaChalamchala, @kerneltime

1 reply

mcgrawia Aug 8, 2024
Author

hi @tanvipenumudy,

I tried recreating my bucket with setting the ozone.s3g.list-keys.shallow.enabled setting to false, but I'm still seeing the same error in the S3G pod.

Bucket info:

ozone sh bucket info /vol1/bucket
2024-08-08 19:28:33,988 [main] WARN  util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(60)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
{
  "metadata" : { },
  "volumeName" : "vol1",
  "name" : "bucket",
  "storageType" : "DISK",
  "versioning" : false,
  "usedBytes" : 287264445,
  "usedNamespace" : 22,
  "creationTime" : "2024-08-08T19:03:43.332Z",
  "modificationTime" : "2024-08-08T19:03:43.332Z",
  "sourcePathExist" : true,
  "quotaInBytes" : -1,
  "quotaInNamespace" : -1,
  "bucketLayout" : "OBJECT_STORE",
  "owner" : "hadoop",
  "link" : false
}

cat /etc/hadoop/ozone-site.xml
<configuration>
<property><name>ozone.s3g.domain.name</name><value>s3g.internal</value></property>
<property><name>ozone.s3g.list-keys.shallow.enabled</name><value>false</value></property>
<property><name>ozone.scm.block.client.address</name><value>ozone-scm-0.ozone-scm-headless</value></property>
<property><name>hdds.datanode.dir</name><value>/data/storage</value></property>
<property><name>ozone.scm.client.address</name><value>ozone-scm-0.ozone-scm-headless</value></property>
<property><name>hdds.scm.safemode.min.datanode</name><value>3</value></property>
<property><name>ozone.datanode.pipeline.limit</name><value>1</value></property>
<property><name>ozone.scm.datanode.id.dir</name><value>/data</value></property>
<property><name>ozone.metadata.dirs</name><value>/data/metadata</value></property>
<property><name>ozone.scm.names</name><value>ozone-scm-0.ozone-scm-headless</value></property>
<property><name>dfs.datanode.use.datanode.hostname</name><value>true</value></property>
<property><name>ozone.om.address</name><value>ozone-om-0.ozone-om-headless</value></property>
</configuration>

Error logs:

2024-08-08 19:22:28,707 [qtp313082880-160] WARN  server.HttpChannel (HttpChannel.java:handleException(776)) - /bucket/
javax.servlet.ServletException: javax.servlet.ServletException: java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:162)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:516)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: javax.servlet.ServletException: java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:410)
	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:358)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:311)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
	at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1459)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
	at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656)
	at org.apache.hadoop.ozone.s3.RootPageDisplayFilter.doFilter(RootPageDisplayFilter.java:53)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at org.apache.hadoop.ozone.s3.EmptyContentTypeFilter.doFilter(EmptyContentTypeFilter.java:76)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at org.apache.hadoop.hdds.server.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1665)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at org.apache.hadoop.hdds.server.http.NoCacheFilter.doFilter(NoCacheFilter.java:48)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
	... 17 more
Caused by: java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
	at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)
	at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)
	at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248)
	at java.base/java.util.Objects.checkIndex(Objects.java:372)
	at java.base/java.util.ArrayList.remove(ArrayList.java:536)
	at org.apache.hadoop.ozone.client.OzoneBucket$KeyIterator.getNextShallowListOfKeys(OzoneBucket.java:1244)
	at org.apache.hadoop.ozone.client.OzoneBucket$KeyIterator.getNextListOfKeys(OzoneBucket.java:1150)
	at org.apache.hadoop.ozone.client.OzoneBucket$KeyIterator.hasNext(OzoneBucket.java:1124)
	at org.apache.hadoop.ozone.s3.endpoint.BucketEndpoint.get(BucketEndpoint.java:212)
	at jdk.internal.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189)
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:256)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235)
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684)
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
	... 51 more

yazynin · 2024-08-09T17:27:10Z

yazynin
Aug 9, 2024

Hello @mcgrawia, maybe this instruction help you https://docs.cloudera.com/cdp-private-cloud-base/7.1.9/ozone-storing-data/topics/ozone-config-spark-s3a.html

1 reply

mcgrawia Aug 9, 2024
Author

Hi @yazynin, thanks for sending but I had already tried those settings.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ozone S3 + Spark (S3A filesystem) Issues with bucket operations #7050

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 4 comments 3 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

Ozone S3 + Spark (S3A filesystem) Issues with bucket operations #7050

mcgrawia Aug 8, 2024

Summary of problem

Troubleshooting steps

Some Background / Setup Details

Replies: 4 comments · 3 replies

mcgrawia Aug 8, 2024 Author

adoroszlai Aug 8, 2024 Collaborator

mcgrawia Aug 8, 2024 Author

tanvipenumudy Aug 8, 2024 Collaborator

mcgrawia Aug 8, 2024 Author

yazynin Aug 9, 2024

mcgrawia Aug 9, 2024 Author

mcgrawia
Aug 8, 2024

Replies: 4 comments 3 replies

mcgrawia
Aug 8, 2024
Author

adoroszlai
Aug 8, 2024
Collaborator

mcgrawia Aug 8, 2024
Author

tanvipenumudy
Aug 8, 2024
Collaborator

mcgrawia Aug 8, 2024
Author

yazynin
Aug 9, 2024

mcgrawia Aug 9, 2024
Author