GCS Signed URL Support

[tustvold]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

#4876 added support for generating S3 URLs, via the Signer trait. This ticket tracks implementing Signer for GoogleCloudStorage

Describe the solution you'd like

The process for generating signed URLs is described here.

Once the stringToSign has been constructed there are two mechanisms for generating the signature:

• Directly sign the URL using the RSA key pair of a service account
• Make an authorized API call to the signBlob API

The latter approach will support all GcpCredentialProvider and is therefore probably the approach to start with. The former is a touch more fiddly, and will likely involve some rejigging of GoogleCloudStorageBuilder::build to expose this ServiceAccountKey in such a way that it can be used by the Signer implementation. It would be perfectly acceptable for the first version to only support the signBlob approach.

Describe alternatives you've considered

Additional context

Split out of #3027

[l1nxy]

I'd like to give this a try and write a first draft. Maybe I can write more later.

[l1nxy]

@tustvold Could you please assign this issue to me? It will serve as a reminder for me.

[l1nxy]

@tustvold Hi, I'm confused about the code. Can I ask some questions?

1. The GCSCredential seems too simple. It uses a self-signed JWT to request the temporary token and then retrieves the objects. I think it should use the service account credential to retrieve the objects, like how it is done in the AWS part.
2. Now, I'm stuck on the signBlob request because it requires the service account email in order to create a signature, but this information is not included in the GCS credential struct.
3. My question is what should I do next? If we can solve the credential problem, both methods - signblob and using an RSA key to generate a URL - will be complete.

additional information:

SERVICE_ACCOUNT_EMAIL is the email address of the service account you want to use to create the signature. For example, service-7550275089395@my-pet-project.iam.gserviceaccount.com.

[Xuanwo]

1. The GCSCredential seems too simple. It uses a self-signed JWT to request the temporary token and then retrieves the objects. I think it should use the service account credential to retrieve the objects, like how it is done in the AWS part.

Yep. GCP has different kind of services accounts like service_account, external_account and impersonated_service_account. We should store those services account information to presign users request.

reqsign implemented most of them which worth to take a look: https://github.com/Xuanwo/reqsign/blob/main/src/google/credential.rs

[tustvold]

I think it is important to separate the credential from how it is obtained, as we support a large number of auth options beyond service account credentials. For request authorization all that is needed is a JWT so that is what GcpCredential contains, and there are then various ways of obtaining such a JWT

Now the problem comes that signing requires additional information, this will need to be plumbed through from the builder, sourced either from the underlying credential provider before it is type erased, or as an explicit config option passed to the builder.

#5073 is also likely relevant here.

I'll try to find some time today to stub out some code for you

[tustvold]

Done some more digging, in no particular order

InstanceMetadata

So the instance metadata token endpoint only provides an access token, and no email, although this can be retrieved with a retrieved with

curl -H "Metadata-Flavor: Google" http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/email

AuthorizedUserCredentials

As described here the configuration metadata does not contain information on the calling identity. Additionally as described here the returned access token is opaque and does not provide information on the identity either.

However, the information can be retrieved by calling a specific API endpoint here.

curl "https://oauth2.googleapis.com/tokeninfo?access_token=ACCESS_TOKEN"

This would then enable calling the signBlob API.

HMAC Keys

I am not sure this is something we will want to support, but you can also sign requests using HMAC keys.

Conclusion

I think we will need an approach that allows different signing methodologies based on the underlying credential provider. The simplest way to do this is likely to implement Signer for the various different credentials providers, i.e. SelfSignedJwt, InstanceCredentialProvider, etc... to store an Arc<dyn Signer> on GoogleCloudStorage and implement Signer for by calling through to it.

Let me know if anything isn't clear.

[l1nxy]

Thank you for your assistance, and I will attempt to implement it.

[tdikland]

Any progress on this issue? I would be willing to step in and help out where possible.