[Failing Test]: A recently added GCS IO IT test_create_default_bucket, seems to be flaky

[tvalentyn]

What happened?

Sample failure from https://github.com/apache/beam/actions/runs/9504236452/job/26196628403?pr=31581

=========================== short test summary info ============================
FAILED apache_beam/io/gcp/gcsio_integration_test.py::GcsIOIntegrationTest::test_create_default_bucket - google.api_core.exceptions.NotFound: 404 DELETE https://storage.googleapis.com/storage/v1/b/gcsio-it-991-us-central1-77b801c0838aee13391c0d1885860494?prettyPrint=false: The specified bucket does not exist


google.api_core.exceptions.NotFound: 404 DELETE https://storage.googleapis.com/storage/v1/b/gcsio-it-991-us-central1-77b801c0838aee13391c0d1885860494?prettyPrint=false: The specified bucket does not exist.
self = <apache_beam.io.gcp.gcsio_integration_test.GcsIOIntegrationTest testMethod=test_create_default_bucket>
mock_default_gcs_bucket_name = <MagicMock name='default_gcs_bucket_name' id='136046463836144'>

    @pytest.mark.it_postcommit
    @mock.patch('apache_beam.io.gcp.gcsio.default_gcs_bucket_name')
    def test_create_default_bucket(self, mock_default_gcs_bucket_name):
...
      existing_bucket = self.gcsio.get_bucket(overridden_bucket_name)
      if existing_bucket:
>       existing_bucket.delete()

apache_beam/io/gcp/gcsio_integration_test.py:171: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/cloud/storage/bucket.py:1607: in delete
    client._delete_resource(
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/cloud/storage/client.py:769: in _delete_resource
    return self._connection.api_request(
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/cloud/storage/_http.py:78: in api_request
    return call()
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/retry/retry_unary.py:293: in retry_wrapped_func
    return retry_target(
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/retry/retry_unary.py:153: in retry_target
    _retry_error_helper(
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/retry/retry_base.py:212: in _retry_error_helper
    raise final_exc from source_exc
../../build/gradleenv/2050596098/lib/python3.10/site-packages/google/api_core/retry/retry_unary.py:144: in retry_target
    result = target()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <google.cloud.storage._http.Connection object at 0x7bbbc9264d00>
method = 'DELETE'
path = '/b/gcsio-it-991-us-central1-77b801c0838aee13391c0d1885860494'
query_params = {}, data = None, content_type = None, headers = None
api_base_url = None, api_version = None, expect_json = True
_target_object = None, timeout = 60
extra_api_info = 'gccl-invocation-id/b6bb18e9-8db0-4b99-bd4c-94fccd0bd99d'

    def api_request(
...
    
        if not 200 <= response.status_code < 300:
>           raise exceptions.from_http_response(response)
E           google.api_core.exceptions.NotFound: 404 DELETE https://storage.googleapis.com/storage/v1/b/gcsio-it-991-us-central1-77b801c0838aee13391c0d1885860494?prettyPrint=false: The specified bucket does not exist.

Issue Failure

Failure: Test is flaky

Issue Priority

Priority: 2 (backlog / disabled test but we think the product is healthy)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[tvalentyn]

cc: @shunping - would you have time to TAL? Thanks!