Feature/s3

[andersleung]

No description provided.

[brainstorm]

Overall looks good :) Other than the inner AWS region hardcoding I'm curious on how you'll handle the AWSCredentialProviderChain, within this Feature/s3 classes or somewhere else?

Thinking about giving access to S3 objects sitting behind an org that restrict public S3 object access, therefore either CloudFront, Presigned URLs or another mechanism must be in place... again, perhaps out of scope and abstraction for this PR, apologies if so ;)

/cc @ohofmann @victorskl @resingerf

[brainstorm]

Hardcoded region, should be parametrizable externally, perhaps CLI so that it can be easily passed to a docker container during deployment (we are in AWS Sydney region :)).

[brainstorm]

This might be a bit clearer/safer?: uri.getPath().replaceAll("^/+$", "")

[jb-adams]

thanks @brainstorm !

@andersleung , let's at least allow for multiple AWS regions (which you may have already covered).

Regarding credentials, signed URLs, etc, there are 2 scenarios where we'd want the server to behave slightly differently in each case.

1. The DRSObject and bytes are not access controlled, but the S3 bucket is private. Any unauthenticated client can access the data in S3, but must go through the DRS server. This case doesn't require user auth, the credentials to access the S3 bucket can be stored on the server itself.

2. The DRSObject and bytes are access controlled, the S3 bucket is private and only the authorized users should be able to access it. This case requires user auth, so credentials need to be passed to the DRS server. In this case, we will use passports somehow to facilitate the authorization evaluation (and subsequent access to the bytes).

Any access control is out of scope for this PR, as we are just getting up and running with completely public data.

[andersleung]

Just to clear one thing up, the default AWS region is just the region that the server will try first when trying to access an S3 bucket. If the access fails, the server will inspect the error and try again with the right region, which is more or less how amazon's own libraries handle "multi" region support. There isn't really an AWS region associated with the DRS server itself, but if you think being able to configure the default region would be useful, it should be easy to support.

[jb-adams]

@brainstorm this AWS multi-region discussion raises an important issue that we need a more sophisticated means of specifying various types of data sources (e.g. local files vs s3 buckets). Connected S3 bucket data sources require unique properties to be specified (region, bucket name, etc.). We've started #21 to be able to handle this in the configuration and startup.