Cloud Storage: Signed URL determine generation from BlobInfo

[dmitry-fa]

For V2: URL returned by signURL() method doesn't include parameter part: &generation=<number>, one should add it manually.
For V4: there is no way to obtain URL pointing to not the latest version of an object, adding &generation=<number> doesn't help.

As part of cloud java-storage project only V2 problem could be fixed.

V4 problem also exists in nodejs-storage: Issue googleapis/google-cloud-java#953
And very similar issue is Issue googleapis/google-cloud-java#7044
It should be fixed on the server side.

Steps to reproduce:

Create a blob with two generation:

#> gsutil versioning set on gs://bucket_generation_signed
Enabling versioning for gs://bucket_generation_signed/...
#> echo The very first version > some.txt
#> gsutil cp some.txt gs://bucket_generation_signed
Copying file://some.txt [Content-Type=text/plain]...
/ [1 files][   23.0 B/   23.0 B]                                                
Operation completed over 1 objects/23.0 B.                                       
#> echo Updated version of the file > some.txt
#> gsutil cp some.txt gs://bucket_generation_signed
Copying file://some.txt [Content-Type=text/plain]...
/ [1 files][   28.0 B/   28.0 B]                                                
Operation completed over 1 objects/28.0 B.                                       
#> gsutil ls -a gs://bucket_generation_signed/some.txt
gs://bucket_generation_signed/some.txt#1576656755290328
gs://bucket_generation_signed/some.txt#1576656816500788

Run the following code:

    public static void main(String... args) throws Exception {
        List<String> al = Arrays.asList("https://www.googleapis.com/auth/cloud-platform");
        String googleCredentialsJson = new String(Files.readAllBytes(Paths.get("bucket_generation_signed.json")));
        GoogleCredentials credentials = GoogleCredentials
                .fromStream(new ByteArrayInputStream(googleCredentialsJson.getBytes())).createScoped(al);
        Storage storage = StorageOptions.newBuilder().setCredentials(credentials).build().getService();

        String bucketName = "bucket_generation_signed";
        String blobName = "some.txt";
        long generation = 1576656755290328L;

        BlobId blobId = BlobId.of(bucketName, blobName, generation);
        Blob blob = storage.get(blobId, Storage.BlobGetOption.generationMatch());
        System.out.println("blob: " + blob);
        System.out.println("v2 " + blob.signUrl(20, TimeUnit.MINUTES, Storage.SignUrlOption.withV2Signature()));
        System.out.println("v4 " + blob.signUrl(20, TimeUnit.MINUTES, Storage.SignUrlOption.withV4Signature()));
    }

The output will look like:

blob: Blob{bucket=bucket_generation_signed, name=some.txt, generation=1576656755290328, size=23, content-type=text/plain, metadata=null}

v2 https://storage.googleapis.com/bucket_generation_signed/some.txt?GoogleAccessId=my-service-account...uJWWFsyd9cfa4FrOSmw%3D%3D

v4 https://storage.googleapis.com/bucket_generation_signed/some.txt?X-Goog-Algorithm=GOOG4-RSA-SHA256&...076f845e952a7

Both returned URLs will point to the latest version:

Updated version of the file

Despite the blob generation is explicitly specified: generation=1576656755290328

[codyoss]

If it is a backend change that needs to happen, a feature request should be opened on the backend service issue tracker.

[dmitry-fa]

backend issue: https://issuetracker.google.com/issues/146802530

[frankyn]

Hi @dmitry-fa,

Thanks for filing this issue.

Right now a user of the signUrl() method is required to provide Query Parameters here's the working example. V4 signed URLs require that all Query Parameters be signed compared to V2 signed URLs which doesn't require all Query Parameters to be signed. This allows the flexibility you are experiencing.

package com.example.storage;

import com.google.auth.oauth2.GoogleCredentials;
import com.google.cloud.storage.Blob;
import com.google.cloud.storage.BlobId;
import com.google.cloud.storage.Storage;
import com.google.cloud.storage.StorageOptions;
import com.google.common.collect.ImmutableMap;

import java.io.ByteArrayInputStream;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.Arrays;
import java.util.List;
import java.util.concurrent.TimeUnit;

public class ExampleOther {
    public static void main(String... args) throws Exception {
        Storage storage = StorageOptions.getDefaultInstance().getService();

        String bucketName = "anima-frank";
        String blobName = "test";
        long generation = 1579134868998095L;

        BlobId blobId = BlobId.of(bucketName, blobName, generation);
        Blob blob = storage.get(blobId, Storage.BlobGetOption.generationMatch());
        System.out.println("blob: " + blob);
        System.out.println("v2 " + blob.signUrl(20, TimeUnit.MINUTES, Storage.SignUrlOption.withV2Signature()));
        System.out.println("v4 " + blob.signUrl(20, TimeUnit.MINUTES, Storage.SignUrlOption.withV4Signature(),
                Storage.SignUrlOption.withQueryParams(ImmutableMap.of("generation", "1579134868998095"))));
    }
}

At the moment signUrl doesn't get information from the generation value set in the Blob, but is a good feature request to follow-up on.

HTH for now.

[dmitry-fa]

Hi @frankyn,

Thanks for your response. The example works for me.

At the moment signUrl doesn't get information from the generation value set in the Blob, but is a good feature request to follow-up on.

Do you believe it is a backend duty to take care of generations (issue#146802530) or this should be implemented on the client side?

[frankyn]

No this is primarily client side only. I closed the bug with some context.

For right now, this is WAI. V2 doesn't require all query parameters to be signed, therefore a user can add additional query parameters without failing the signature check.

I'd leave this open as a feature request rather than a bug because we have an intended path for folks to use.

[dmitry-fa]

It seems that Storage.signUrl(BlobInfo) can distinguish cases when generation is explicitly specified, from cases when generation is not needed:

storage.signUrl(BlobInfo.newBuilder(bucketName, blobName, generation).build(), ...)
storage.signUrl(BlobInfo.newBuilder(bucketName, blobName).build(), ...)

But this is not always easy:

BlobInfo blobInfo = BlobInfo.newBuilder(bucketName, blobName).build(); 
      // blobInfo.getGeneration() !== null
Blob blob = storage.createBlob(blobInfo);
     // blob.getGeneration() !== null
storage.signUrl(blob);  

Including generation in signed URL by default is not a good idea. People will be confused if their URL starts point to an outdated resource. I hope, giving examples on how to sign URL for a blob generation should be enough.