S3-compatible storage: support S3ForcePathStyle and expanded configs

[tonychoe]

Is your feature request related to a problem? Please describe.

S3-compatible storages require s3ForcePathStyle. Additionally Oracle Object Storage requires a region identifier to use other than a home region.

Currently, Tempo doesn't allow these configurations, therefore we cannot use some S3-compatible storages as Tempo's storage.

Describe the solution you'd like

Allow passing s3forcepathstyle and region to the S3 SDK used in Tempo, as the following example config:

    s3:
      s3forcepathstyle: true
      bucket: tempo 
      region: us-phoenix-1
      endpoint: endpoint1234.compat.objectstorage.us-phoenix-1.oraclecloud.com      access_key: tempo
      secret_key: supersecret
      insecure: false

As you may know, Loki has been supporting the S3-compatible storage, a region identifier and more. Perhaps Loki's code could be re-used to implement this feature.

Describe alternatives you've considered
N/A

Additional context
N/A

[annanay25]

Thanks for reporting @tonychoe, agree that we need this. Would you be interested in PR'ing the enhancement?

[mdisibio]

Some findings:
Region is straightforward to readd. MinIO automatically detects the region from the endpoint so we thought it was no longer needed.

s3forcepathstyle:

• Loki uses the cortex s3 client here: https://github.com/cortexproject/cortex/blob/master/pkg/chunk/aws/s3_storage_client.go
• Tempo is using the MinIO client. I believe the corresponding functionality is the BucketLookupType which can be Auto,DNS,Path: https://github.com/minio/minio-go/blob/master/api.go#L127
• AWS blog post about dns vs path style lookups: https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-plan-the-rest-of-the-story/

In summary, I believe we can support these 2 features (region and path style) using the MinIO client. The cortex client has additional functionality which may be useful, if we want to go that route.

[joe-elliott]

Yes, please add these options. I'd say anything that doesn't unnecessarily complicate the code or config is welcome.

There's a reason all the Cortex options are there and we'll probably need them sooner or later anyway :).