As of 2022-02-22 Automatic detection of requesterPays buckets is failing

[BenWhitehead]

First observed on PR #823 and visible in the integration tests related to Requester Pays (build results)

Environment details

1. this library
2. OS type and version: N/A
3. Java version: N/A
4. version(s): all released prior to 2022-02-22

Code example

java-storage-nio/google-cloud-nio/src/test/java/com/google/cloud/storage/contrib/nio/it/ITGcsNio.java

Lines 310 to 316 in 417704a

	public void testAutodetectWhenRequesterPays() throws IOException {
	CloudStorageFileSystem testRPBucket = getRequesterPaysBucket(true, project);
	Assert.assertEquals(
	"Autodetect should have detected the RP bucket",
	testRPBucket.config().userProject(),
	project);
	}

External references such as API reference guides

It seems the error message response changed subtly

- Bucket is requester pays bucket but no user project provided.
+ Bucket is a requester pays bucket but no user project provided.

According to public documentation, we should expect the reason to be userProjectMissing, however it is required as can be seen in

{
  "error": {
    "code": 400,
    "message": "Bucket is a requester pays bucket but no user project provided.",
    "errors": [
      {
        "message": "Bucket is a requester pays bucket but no user project provided.",
        "domain": "global",
        "reason": "required"
      }
    ]
  }
}

Related from NodeJS https://github.com/googleapis/nodejs-storage/pull/1791/files

[lbergelson]

I think the underlying issue is here.

java-storage-nio/google-cloud-nio/src/main/java/com/google/cloud/storage/contrib/nio/CloudStorageFileSystemProvider.java

Lines 963 to 975 in fc0f8a1

	public boolean requesterPays(String bucketName) {
	initStorage();
	try {
	// instead of true/false, this method returns true/null.
	Boolean isRP = storage.get(bucketName).requesterPays();
	return isRP != null && isRP.booleanValue();
	} catch (StorageException ex) {
	if (ex.getCode() == 400 && ex.getMessage().contains("Bucket is requester pays")) {
	return true;
	}
	throw ex;
	}
	}

It always falls into the catch block which then tries to match against that changed string. I think it shouldn't be falling into the catch block and instead the call to get(bucketName) should include an option specifying the project name which should keep it from throwing an exception. It seems like a bug that was previously covered up by the catch clause.
Boolean isRP = storage.get(bucketName).requesterPays();

[lbergelson]

Ignore that. It seems that throwing the exception is necessary because it's attempting to find requester pays status without providing the user account to avoid access charges in the case where it wasn't requester pays.

It seems like there needs to be a more robust mechanism of determining if this is a requester pays bucket than scraping the exception message after trying to connect. This mechanism also causes problems because it requires the somewhat unusual storage.buckets.get permission which is not necessary for reading the actual content of the bucket. So users need more permissions than necessary when trying to read from a requester pays bucket because the test to see if it IS a requester pays bucket requires additional permissions than actually accessing the file they want. This has caused continual confusion for our users.

Is there either:
A) a way to query the requester pays status of a bucket that doesn't require the user to be authenticated in the case where it is requester pays
B) a way to make this ignore querying requester pays status and just falling back to connecting with a user account when we get back an error that says it is requester pays?

[droazen]

This issue is unfortunately breaking large numbers of our production workflows. I hate to suggest this, but perhaps for now we could grit our teeth and add the missing "a" into the message it's looking for to get it working again, and open a separate ticket to research / implement a more robust mechanism for detecting requester pays status. Unless someone already knows of a more robust way?

Once a patch is in for this issue, would it be possible to get a release of the library with the fix?