Upload to GCS error

[mmolinac]

Hi,
It's only recently we're getting errors when uploading files to GCS, like this:

2022/09/17 01:11:21.636318 error one of upload go-routine return error: one of uploadTableData go-routine return error: can't check uploaded file: key not found

or this other one:

2022/09/20 04:10:51.581880  warn can't close writer: googleapi: Error 503: We encountered an internal error. Please try again., backendError
2022/09/20 04:10:51.644658 error can't acquire semaphore during Upload data parts: context canceled
2022/09/20 04:10:51.798495  warn can't close writer: googleapi: Error 503: We encountered an internal error. Please try again., backendError
2022/09/20 04:10:52.714160  warn can't close writer: googleapi: Error 503: We encountered an internal error. Please try again., backendError
2022/09/20 04:10:53.604868  warn can't close writer: googleapi: Error 503: We encountered an internal error. Please try again., backendError
2022/09/20 04:10:53.667549 error can't acquire semaphore during Upload data parts: context canceled
2022/09/20 04:10:54.037670  warn can't close writer: googleapi: Error 503: We encountered an internal error. Please try again., backendError
2022/09/20 04:10:54.114429 error can't acquire semaphore during Upload data parts: context canceled
2022/09/20 04:10:54.511411  warn can't close writer: googleapi: got HTTP response code 503 with body: Service Unavailable

I've seen that there's a fixed retry count for S3, as mentioned by @AlexAkulov in #36 (comment) .

I can't see if there's something similar in https://github.com/AlexAkulov/clickhouse-backup/blob/master/pkg/storage/gcs.go .

Is the app trying several times already or does it stop after first error?

Regards,
Manuel

[Slach]

it should stop after first error now
we try to add storage independent retry mechanism in 2.1.0

[Slach]

implements in altinity/clickhouse-backup:master duplicated with #397