Test fail on ITStorageTest.testUpdateBlob

[mziccard]

After merging #211 travis build for oraclejdk7 failed with an error in ITStorageTest.testUpdate:

testUpdateBlob(com.google.gcloud.storage.ITStorageTest)  Time elapsed: 10.421 sec  <<< ERROR!
com.google.gcloud.storage.StorageException: Backend Error
    at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:145)
    at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
    at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:321)
    at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1056)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
    at com.google.gcloud.spi.DefaultStorageRpc.patch(DefaultStorageRpc.java:231)
    at com.google.gcloud.storage.StorageImpl$11.call(StorageImpl.java:343)
    at com.google.gcloud.storage.StorageImpl$11.call(StorageImpl.java:340)
    at com.google.gcloud.RetryHelper.doRetry(RetryHelper.java:181)
    at com.google.gcloud.RetryHelper.runWithRetries(RetryHelper.java:247)
    at com.google.gcloud.RetryHelper.runWithRetries(RetryHelper.java:237)
    at com.google.gcloud.storage.StorageImpl.update(StorageImpl.java:340)
    at com.google.gcloud.storage.ITStorageTest.testUpdateBlob(ITStorageTest.java:150)

I never experienced this error locally and it seems not to occur always (builds for other java versions and next builds succeeded). The error message Backend Error makes me think that the problem resides outside of our library. I open this issue just to double check with you guys, can we consider this as a sporadic error?

[aozarov]

We should probably configure RemoteGcsHelper to use something better than RetryParams.noRetries().

[mziccard]

Yes this error could have probably been avoided by using RetryParams.getDefaultInstance().

[aozarov]

I think default settings are not enough.

I asked @ajkannan to change it to similar settings that we set in AE mapreduce library. See #231.

This is likely to fix the problem but if not than I am sure #232 would help.

[aozarov]

I think we still need #232 for this to be fixed.

[aozarov]

We still see this failures.

com.google.gcloud.storage.StorageException: 500 Internal Server Error
{
 "error": {
  "errors": [
   {
    "domain": "global",
    "reason": "backendError",
    "message": "Backend Error"
   }
  ],
  "code": 500,
  "message": "Backend Error"
 }
}
    at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1061)
    at com.google.gcloud.spi.DefaultStorageRpc.open(DefaultStorageRpc.java:515)
    at com.google.gcloud.storage.BlobWriteChannelImpl.<init>(BlobWriteChannelImpl.java:62)
    at com.google.gcloud.storage.StorageImpl.writer(StorageImpl.java:569)
    at com.google.gcloud.storage.StorageImpl.writer(StorageImpl.java:564)
    at com.google.gcloud.storage.ITStorageTest.testWriteChannelExistingBlob(ITStorageTest.java:483)

I wonder if we are actually not retrying in some condition where we should be.

[ajkannan]

Adding another error seen a couple times recently:

Running com.google.gcloud.storage.ITStorageTest
Tests run: 30, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 86.55 sec <<< FAILURE! - in com.google.gcloud.storage.ITStorageTest
testDeleteBlobs(com.google.gcloud.storage.ITStorageTest)  Time elapsed: 11.099 sec  <<< FAILURE!
java.lang.AssertionError: null
    at org.junit.Assert.fail(Assert.java:86)
    at org.junit.Assert.assertTrue(Assert.java:41)
    at org.junit.Assert.assertTrue(Assert.java:52)
    at com.google.gcloud.storage.ITStorageTest.testDeleteBlobs(ITStorageTest.java:563)

[aozarov]

The above error should be significantly reduced after issue #371 is fixed.

[mziccard]

This issue is very old and contains a bunch of exceptions that possibly have different causes. Some o which are already fixed. I am closing this, we should favor opening 1 issue for each flakey or failing test. Feel free to reopen if you disagree.