deps: update dependency com.google.cloud:google-cloud-storage to v2.1…

…7.1 (#1086) [![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [com.google.cloud:google-cloud-storage](https://github.com/googleapis/java-storage) | `2.16.0` -> `2.17.1` | [![age](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-storage/2.17.1/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-storage/2.17.1/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-storage/2.17.1/compatibility-slim/2.16.0)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/maven/com.google.cloud:google-cloud-storage/2.17.1/confidence-slim/2.16.0)](https://docs.renovatebot.com/merge-confidence/) | --- ### Release Notes <details> <summary>googleapis/java-storage</summary> ### [`v2.17.1`](https://github.com/googleapis/java-storage/blob/HEAD/CHANGELOG.md#2171-httpsgithubcomgoogleapisjava-storagecomparev2170v2171-2023-01-13) [Compare Source](https://github.com/googleapis/java-storage/compare/v2.17.0...v2.17.1) ##### Bug Fixes - Update BaseStorageReadChannel to be left open unless explicitly closed ([#1853](https://github.com/googleapis/java-storage/issues/1853)) ([1425dd9](https://github.com/googleapis/java-storage/commit/1425dd97cb7d4a58f0bbededeca543f1a89c7d5d)) ### [`v2.17.0`](https://github.com/googleapis/java-storage/blob/HEAD/CHANGELOG.md#2170-httpsgithubcomgoogleapisjava-storagecomparev2160v2170-2023-01-12) [Compare Source](https://github.com/googleapis/java-storage/compare/v2.16.0...v2.17.0) ##### Features - Implement GrpcStorageImpl BucketAccessControl operations ([#1816](https://github.com/googleapis/java-storage/issues/1816)) ([5c52079](https://github.com/googleapis/java-storage/commit/5c52079fb5f52caf39a49ccb96df6251a9c728d3)) - Implement GrpcStorageImpl ObjectAccessControl operations ([#1818](https://github.com/googleapis/java-storage/issues/1818)) ([2eec791](https://github.com/googleapis/java-storage/commit/2eec791122bb1bb28a1ffb14beb7ce8776c5b5ec)) - Implement GrpcStorageImpl#createDefaultAcl & GrpcStorageImpl#updateDefaultAcl ([#1806](https://github.com/googleapis/java-storage/issues/1806)) ([0f24a11](https://github.com/googleapis/java-storage/commit/0f24a11c5289a4c07f27d8a3c29fab34520b036f)) - Implement GrpcStorageImpl#deleteDefaultAcl ([#1807](https://github.com/googleapis/java-storage/issues/1807)) ([c783277](https://github.com/googleapis/java-storage/commit/c78327717a7936492161ddcc64c86374db72c48c)) - Implement GrpcStorageImpl#getDefaultAcl ([#1802](https://github.com/googleapis/java-storage/issues/1802)) ([b9b7c49](https://github.com/googleapis/java-storage/commit/b9b7c49fcfcab285da156b34b186a007150e876f)) - Implement GrpcStorageImpl#listDefaultAcl ([#1805](https://github.com/googleapis/java-storage/issues/1805)) ([03c2e66](https://github.com/googleapis/java-storage/commit/03c2e6660721b4a8bfc09b241ef44f3e4e08865b)) - Improve throughput of http based storage#reader between 100 MiB/s and 200 MiB/s ([#1799](https://github.com/googleapis/java-storage/issues/1799)) ([94cd288](https://github.com/googleapis/java-storage/commit/94cd2887f22f6d1bb82f9929b388c27c63353d77)) - Update GrpcBlobReadChannel to allow seek/limit after read ([#1834](https://github.com/googleapis/java-storage/issues/1834)) ([45dc983](https://github.com/googleapis/java-storage/commit/45dc983a4af8e7feb937263ce611bd34eda37e03)) ##### Bug Fixes - Add missing preconditions and update samples ([#1753](https://github.com/googleapis/java-storage/issues/1753)) ([96beca2](https://github.com/googleapis/java-storage/commit/96beca2465158fb4633d58fe09a9776a4b171811)) - **grpc:** Fix bucket logging conversion to allow clearing ([#1822](https://github.com/googleapis/java-storage/issues/1822)) ([30e19dc](https://github.com/googleapis/java-storage/commit/30e19dc55c61917c3a73055091e9e6ca0744f172)) - Update gRPC object list implementation to include synthetic directories ([#1824](https://github.com/googleapis/java-storage/issues/1824)) ([0665c24](https://github.com/googleapis/java-storage/commit/0665c2473b5b1a18061d1e58382320ae55295520)) - Update Grpc Write implementation to allow specifying expected md5 ([#1815](https://github.com/googleapis/java-storage/issues/1815)) ([4662572](https://github.com/googleapis/java-storage/commit/46625729b6fd62d8f133c3fb2d8ee00eb64ee8e9)) - Update GrpcConversions to use Bucket.RetentionPolicy.retention_duration ([#1798](https://github.com/googleapis/java-storage/issues/1798)) ([82fb014](https://github.com/googleapis/java-storage/commit/82fb014508178e8ad3fd08e9efc757a8e47564da)) - Update GrpcStorageImpl#update to support fine-grained update of BucketInfo.labels and BlobInfo.metadata ([#1843](https://github.com/googleapis/java-storage/issues/1843)) ([c8bf3c7](https://github.com/googleapis/java-storage/commit/c8bf3c70cca81ed87a52939fe7da58889c8f55ce)) ##### Documentation - Document differing behavior of {get,list}{,default}Acl between HTTP and gRPC ([#1820](https://github.com/googleapis/java-storage/issues/1820)) ([9511b17](https://github.com/googleapis/java-storage/commit/9511b173e84d2b28ab1a1625b16e3e648c3856fb)) ##### Dependencies - Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.1.1 ([#1836](https://github.com/googleapis/java-storage/issues/1836)) ([3b71fab](https://github.com/googleapis/java-storage/commit/3b71fab11ac71039c2a9983821ce02ce25ce311d)) - Update dependency net.jqwik:jqwik to v1.7.2 ([#1833](https://github.com/googleapis/java-storage/issues/1833)) ([83bc261](https://github.com/googleapis/java-storage/commit/83bc261130e89e5994f21e32422054ef6ea2fe8e)) - Update dependency org.junit.vintage:junit-vintage-engine to v5.9.2 ([#1837](https://github.com/googleapis/java-storage/issues/1837)) ([5b38184](https://github.com/googleapis/java-storage/commit/5b381845b4f48a691aa3f0cb96599ddefc7e463f)) - Update junit-platform.version to v5.9.2 ([#1838](https://github.com/googleapis/java-storage/issues/1838)) ([372521b](https://github.com/googleapis/java-storage/commit/372521ba80b12e52c74fae5ac766dbe6610ff0b2)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://app.renovatebot.com/dashboard#github/googleapis/java-storage-nio).
googleapis · Jan 13, 2023 · 28b525d · 28b525d
1 parent 25d2367
commit 28b525d
Show file tree

Hide file tree

Showing 4 changed files with 173 additions and 7 deletions.
diff --git a/google-cloud-nio/pom.xml b/google-cloud-nio/pom.xml
@@ -42,6 +42,12 @@
       <groupId>com.google.http-client</groupId>
       <artifactId>google-http-client</artifactId>
     </dependency>
+    <!-- Used by tests, but can't be set to test scope otherwise it will be
+    stripped as a transitive dependency of storage. -->
+    <dependency>
+      <groupId>com.google.http-client</groupId>
+      <artifactId>google-http-client-gson</artifactId>
+    </dependency>
     <dependency>
       <groupId>com.google.auto.value</groupId>
       <artifactId>auto-value-annotations</artifactId>

diff --git a/...-cloud-nio/src/main/java/com/google/cloud/storage/contrib/nio/testing/FakeStorageRpc.java b/...-cloud-nio/src/main/java/com/google/cloud/storage/contrib/nio/testing/FakeStorageRpc.java
@@ -16,6 +16,13 @@
 
 package com.google.cloud.storage.contrib.nio.testing;
 
+import com.google.api.client.http.HttpRequestInitializer;
+import com.google.api.client.http.HttpTransport;
+import com.google.api.client.http.LowLevelHttpRequest;
+import com.google.api.client.http.LowLevelHttpResponse;
+import com.google.api.client.json.gson.GsonFactory;
+import com.google.api.client.testing.http.MockLowLevelHttpRequest;
+import com.google.api.client.testing.http.MockLowLevelHttpResponse;
 import com.google.api.client.util.DateTime;
 import com.google.api.services.storage.model.Bucket;
 import com.google.api.services.storage.model.ServiceAccount;
@@ -25,10 +32,15 @@
 import com.google.cloud.storage.StorageException;
 import com.google.cloud.storage.spi.v1.StorageRpc;
 import com.google.cloud.storage.testing.StorageRpcTestBase;
+import com.google.common.base.Preconditions;
+import java.io.ByteArrayInputStream;
 import java.io.IOException;
 import java.io.InputStream;
 import java.io.OutputStream;
+import java.io.UnsupportedEncodingException;
 import java.math.BigInteger;
+import java.net.URLDecoder;
+import java.nio.charset.StandardCharsets;
 import java.nio.file.FileAlreadyExistsException;
 import java.text.SimpleDateFormat;
 import java.util.ArrayList;
@@ -37,6 +49,8 @@
 import java.util.List;
 import java.util.Map;
 import java.util.concurrent.ConcurrentHashMap;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
 import javax.annotation.concurrent.NotThreadSafe;
 
 /**
@@ -76,6 +90,11 @@ class FakeStorageRpc extends StorageRpcTestBase {
   private static final SimpleDateFormat RFC_3339_FORMATTER =
       new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ssXXX");
 
+  private static final int OK = 200;
+  private static final int PARTIAL_CONTENT = 206;
+  private static final int NOT_FOUND = 404;
+  private static final byte[] EMPTY_BYTES = new byte[0];
+
   // fullname -> metadata
   Map<String, StorageObject> metadata = new ConcurrentHashMap<>();
   // fullname -> contents
@@ -244,7 +263,7 @@ public StorageObject compose(
   public byte[] load(StorageObject storageObject, Map<Option, ?> options) throws StorageException {
     String key = fullname(storageObject);
     if (!contents.containsKey(key)) {
-      throw new StorageException(404, "File not found: " + key);
+      throw new StorageException(NOT_FOUND, "File not found: " + key);
     }
     return contents.get(key);
   }
@@ -264,7 +283,7 @@ public Tuple<String, byte[]> read(
     }
     String key = fullname(from);
     if (!contents.containsKey(key)) {
-      throw new StorageException(404, "File not found: " + key);
+      throw new StorageException(NOT_FOUND, "File not found: " + key);
     }
     checkGeneration(key, generationMatch);
     long position = zposition;
@@ -299,7 +318,7 @@ public long read(
     }
     String key = fullname(from);
     if (!contents.containsKey(key)) {
-      throw new StorageException(404, "File not found: " + key);
+      throw new StorageException(NOT_FOUND, "File not found: " + key);
     }
     checkGeneration(key, generationMatch);
     if (position < 0) {
@@ -396,7 +415,7 @@ public RewriteResponse openRewrite(RewriteRequest rewriteRequest) throws Storage
 
     // a little hackish, just good enough for the tests to work.
     if (!contents.containsKey(sourceKey)) {
-      throw new StorageException(404, "File not found: " + sourceKey);
+      throw new StorageException(NOT_FOUND, "File not found: " + sourceKey);
     }
 
     // if non-null, then we check the file's at that generation.
@@ -444,7 +463,7 @@ private static DateTime now() {
   }
 
   private String fullname(StorageObject so) {
-    return (so.getBucket() + "/" + so.getName());
+    return fullname(so.getBucket(), so.getName());
   }
 
   private BigInteger size(StorageObject so) {
@@ -481,7 +500,8 @@ private void checkGeneration(String key, Long generationMatch) {
       Long generation = metadata.get(key).getGeneration();
       if (!generationMatch.equals(generation)) {
         throw new StorageException(
-            404, "Generation mismatch. Requested " + generationMatch + " but got " + generation);
+            NOT_FOUND,
+            "Generation mismatch. Requested " + generationMatch + " but got " + generation);
       }
     }
   }
@@ -517,4 +537,140 @@ private static boolean processedAsFolder(
   public ServiceAccount getServiceAccount(String projectId) {
     return null;
   }
+
+  @Override
+  public com.google.api.services.storage.Storage getStorage() {
+    HttpTransport transport = new FakeStorageRpcHttpTransport();
+    HttpRequestInitializer httpRequestInitializer = request -> {};
+    return new com.google.api.services.storage.Storage(
+        transport, new GsonFactory(), httpRequestInitializer);
+  }
+
+  private static String fullname(String bucket, String object) {
+    return String.format("%s/%s", bucket, object);
+  }
+
+  private static final String KEY_PATTERN_DEFINITION = "^.*?/b/(.*?)/o/(.*?)(?:[?].*|$)";
+  private static final Pattern KEY_PATTERN = Pattern.compile(KEY_PATTERN_DEFINITION);
+
+  MyMockLowLevelHttpRequest create(String url) {
+    Matcher m = KEY_PATTERN.matcher(url);
+    Preconditions.checkArgument(
+        m.matches(),
+        "Provided url '%s' does not match expected pattern '%s'",
+        url,
+        KEY_PATTERN_DEFINITION);
+
+    String bucket = m.group(1);
+    String object = m.group(2);
+
+    String decode = urlDecode(object);
+    String key = fullname(bucket, decode);
+    return new MyMockLowLevelHttpRequest(url, key);
+  }
+
+  private static String urlDecode(String object) {
+    try {
+      return URLDecoder.decode(object, StandardCharsets.UTF_8.name());
+    } catch (UnsupportedEncodingException e) {
+      throw new RuntimeException(e);
+    }
+  }
+
+  private class MyMockLowLevelHttpRequest extends MockLowLevelHttpRequest {
+
+    private final String key;
+
+    private MyMockLowLevelHttpRequest(String url, String key) {
+      super(url);
+      this.key = key;
+    }
+
+    /**
+     * {@link MockLowLevelHttpRequest#execute} tries to return a value, but we need to compute based
+     * upon possible request mutations. So override it to call {@link #getResponse()}.
+     */
+    @Override
+    public LowLevelHttpResponse execute() throws IOException {
+      return getResponse();
+    }
+
+    @Override
+    public MockLowLevelHttpResponse getResponse() {
+
+      MockLowLevelHttpResponse resp = new MockLowLevelHttpResponse();
+      byte[] bytes = contents.get(key);
+      if (bytes == null) {
+        resp.setStatusCode(NOT_FOUND);
+      } else {
+        int length = bytes.length;
+        Map<String, List<String>> headers = getHeaders();
+        List<String> range = headers.get("range");
+        int begin = 0;
+        int endInclusive = length - 1;
+        if (range != null && !range.isEmpty()) {
+          String rangeString = range.get(0).substring("range=".length());
+          if ("0-".equals(rangeString)) {
+            resp.setStatusCode(OK);
+          } else if (rangeString.startsWith("-")) {
+            // we don't support negative offsets yet
+            resp.setStatusCode(400);
+            return resp;
+          } else if (rangeString.endsWith("-")) {
+            // only lower bounded
+            String beginS = rangeString.substring(0, rangeString.length() - 1);
+            begin = Integer.parseInt(beginS);
+            resp.setStatusCode(PARTIAL_CONTENT);
+          } else {
+            // otherwise a lower and upper bound
+            int i = rangeString.indexOf('-');
+            if (i == -1) {
+              resp.setStatusCode(400);
+              return resp;
+            } else {
+              String beginS = rangeString.substring(0, i);
+              String endInclusiveS = rangeString.substring(i + 1);
+              begin = Integer.parseInt(beginS);
+              endInclusive = Integer.parseInt(endInclusiveS);
+              resp.setStatusCode(PARTIAL_CONTENT);
+            }
+          }
+
+          if (begin > length) {
+            resp.addHeader("Content-Range", String.format("bytes */%d", length));
+            resp.setContent(EMPTY_BYTES);
+            resp.setContent(new ByteArrayInputStream(EMPTY_BYTES));
+          } else {
+            int newLength = endInclusive - begin + 1;
+            resp.addHeader("Content-Length", String.valueOf(newLength));
+            // Content-Range: bytes 4-9/512
+            resp.addHeader(
+                "Content-Range", String.format("bytes %d-%d/%d", begin, endInclusive, length));
+            byte[] content = Arrays.copyOfRange(bytes, begin, endInclusive + 1);
+            resp.setContent(content);
+            resp.setContent(new ByteArrayInputStream(content));
+          }
+        } else {
+          resp.addHeader("Content-Length", String.valueOf(length));
+          resp.setContent(bytes);
+          resp.setContent(new ByteArrayInputStream(bytes));
+          resp.setStatusCode(OK);
+        }
+      }
+      return resp;
+    }
+  }
+
+  private class FakeStorageRpcHttpTransport extends HttpTransport {
+
+    @Override
+    public boolean supportsMethod(String method) {
+      return "GET".equals(method);
+    }
+
+    @Override
+    public LowLevelHttpRequest buildRequest(String method, String url) {
+      return create(url);
+    }
+  }
 }
diff --git a/google-cloud-nio/src/test/java/com/google/cloud/storage/contrib/nio/it/ITGcsNio.java b/google-cloud-nio/src/test/java/com/google/cloud/storage/contrib/nio/it/ITGcsNio.java
@@ -415,6 +415,7 @@ public void testReadByteChannel() throws IOException {
     ByteBuffer buf = ByteBuffer.allocate(SML_SIZE);
     int read = 0;
     while (chan.isOpen()) {
+      buf.clear();
       int rc = chan.read(buf);
       assertThat(chan.size()).isEqualTo(size);
       if (rc < 0) {
@@ -794,6 +795,7 @@ public void testFileChannelRead() throws IOException {
     ByteBuffer buf = ByteBuffer.allocate(SML_SIZE);
     int read = 0;
     while (chan.isOpen()) {
+      buf.clear();
       int rc = chan.read(buf);
       assertThat(chan.size()).isEqualTo(size);
       if (rc < 0) {
@@ -954,6 +956,7 @@ public void testFileChannelTransferFrom() throws IOException {
       ByteBuffer buf = ByteBuffer.allocate(SML_SIZE);
       int read = 0;
       while (source.isOpen()) {
+        buf.clear();
         int rc = source.read(buf);
         assertThat(source.size()).isEqualTo(size);
         if (rc < 0) {
@@ -997,6 +1000,7 @@ public void testFileChannelTransferTo() throws IOException {
       ByteBuffer buf = ByteBuffer.allocate(SML_SIZE);
       int read = 0;
       while (source.isOpen()) {
+        buf.clear();
         int rc = source.read(buf);
         assertThat(source.size()).isEqualTo(size);
         if (rc < 0) {

diff --git a/pom.xml b/pom.xml
@@ -78,7 +78,7 @@
       <dependency>
         <groupId>com.google.cloud</groupId>
         <artifactId>google-cloud-storage</artifactId>
-        <version>2.16.0</version>
+        <version>2.17.1</version>
       </dependency>
       <dependency>
         <groupId>com.google.apis</groupId>