Storage: 'test_bucket_w_retention_period' flakes with 429

[tseaver]

From Kokoro today:

______________ TestRetentionPolicy.test_bucket_w_retention_period ______________

self = <tests.system.TestRetentionPolicy testMethod=test_bucket_w_retention_period>

    def test_bucket_w_retention_period(self):
        import datetime
        from google.api_core import exceptions

        period_secs = 10

        new_bucket_name = 'w-retention-period' + unique_resource_id('-')
>       bucket = Config.CLIENT.create_bucket(new_bucket_name)

tests/system.py:1296:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
google/cloud/storage/client.py:285: in create_bucket
    bucket.create(client=self, project=project)
google/cloud/storage/bucket.py:480: in create
    data=properties, _target_object=self)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <google.cloud.storage._http.Connection object at 0x7f99c18a1210>
method = 'POST', path = '/b', query_params = {'project': 'precise-truck-742'}
data = '{"name": "w-retention-period-1539807189922"}'
content_type = 'application/json', headers = None, api_base_url = None
api_version = None, expect_json = True
_target_object = <Bucket: w-retention-period-1539807189922>

    def api_request(self, method, path, query_params=None,
                    data=None, content_type=None, headers=None,
                    api_base_url=None, api_version=None,
                    expect_json=True, _target_object=None):
        ...
        url = self.build_api_url(path=path, query_params=query_params,
                                 api_base_url=api_base_url,
                                 api_version=api_version)

        # Making the executive decision that any dictionary
        # data will be sent properly as JSON.
        if data and isinstance(data, dict):
            data = json.dumps(data)
            content_type = 'application/json'

        response = self._make_request(
            method=method, url=url, data=data, content_type=content_type,
            headers=headers, target_object=_target_object)

        if not 200 <= response.status_code < 300:
>           raise exceptions.from_http_response(response)
E           TooManyRequests: 429 POST https://www.googleapis.com/storage/v1/b?project=precise-truck-742: The project exceeded the rate limit for creating and deleting buckets.

../core/google/cloud/_http.py:293: TooManyRequests

[frankyn]

Looks like this project needs to be whitelisted as well.

@tseaver could you directly ping me the project id if this is still occurring?

[tseaver]

@frankyn this looks like actual flakiness in the system test: the call to Config.CLIENT.create_bucket needs to be wrapped in a retry_429.

[frankyn]

Agreed that it's a flaky failure based on create/delete frequency. Systems tests can have bucket create/delete quota lifted.