Fix gcs connection failure

[korneelvdbroek]

This PR should fix issue #606 by retrying the upload to GCS in case of connection issues.
Requires version >= 1.28.0 of google.cloud.storage

[codecov-commenter]

Codecov Report

Base: 86.40% // Head: 86.36% // Decreases project coverage by -0.04% ⚠️

Coverage data is based on head (f20a27e) compared to base (da08432).
Patch coverage: 0.00% of modified lines in pull request are covered.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #686      +/-   ##
==========================================
- Coverage   86.40%   86.36%   -0.04%     
==========================================
  Files          36       36              
  Lines        2339     2340       +1     
==========================================
  Hits         2021     2021              
- Misses        318      319       +1     

Impacted Files	Coverage Δ
tensorboardX/record_writer.py	78.67% <0.00%> (-0.59%)	⬇️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

[lanpa]

As a reference:
https://cloud.google.com/python/docs/reference/storage/2.1.0/retry_timeout

Configuring Retries
NOTE: For more background on retries, see also the GCS Retry Strategies Document

Methods which invoke API methods may fail for a number of reasons, some of which represent “transient” conditions, and thus can be retried automatically. The library tries to provide a sensible default retry policy for each method, base on its semantics:

For API requests which are always idempotent, the library uses its DEFAULT_RETRY policy, which retries any API request which returns a “transient” error.

For API requests which are idempotent only if the blob has the same “generation”, the library uses its DEFAULT_RETRY_IF_GENERATION_SPECIFIED policy, which retries API requests which returns a “transient” error, but only if the original request includes an ifGenerationMatch header.

For API requests which are idempotent only if the bucket or blob has the same “metageneration”, the library uses its DEFAULT_RETRY_IF_METAGENERATION_SPECIFIED policy, which retries API requests which returns a “transient” error, but only if the original request includes an ifMetagenerationMatch header.

For API requests which are idempotent only if the bucket or blob has the same “etag”, the library uses its DEFAULT_RETRY_IF_ETAG_IN_JSON policy, which retries API requests which returns a “transient” error, but only if the original request includes an ETAG in its payload.

For those API requests which are never idempotent, the library passes retry=None by default, suppressing any retries.

[lanpa]

Thanks! @korneelvdbroek