Harden retries on tenant/timeline deletion path.

[LizardWizzard]

Originated from test failure where we hot SlowDown error from s3.
The patch generalizes download_retry to not be download specific.
Resulting retry function is moved to utils crate. download_retries
is now a thin wrapper around this retry function.

To ensure that all needed retries are in place test code now uses
test_remote_failures=1 setting.

Ref https://neondb.slack.com/archives/C059ZC138NR/p1691743624353009

[github-actions]

1588 tests run: 1511 passed, 0 failed, 77 skipped (full report)

---

Flaky tests (3)

Postgres 15

• test_get_tenant_size_with_multiple_branches: debug
• test_single_branch_get_tenant_size_grows: debug
• test_s3_wal_replay[local_fs]: debug

_{The comment gets automatically updated with the latest test results
c25a737 at 2023-08-14T14:05:47.294Z :recycle:}

[jcsp]

This PR isn't adding it but I'm curious: why do we have our own retry code instead of using a RetryConfig from aws_sdk_s3?

[LizardWizzard]

This PR isn't adding it but I'm curious: why do we have our own retry code instead of using a RetryConfig from aws_sdk_s3?

Hm, thats a good question. We wernt using aws sdk in the beginning, it is fairly new. Before that we used rusoto that is nowdeprecated.

Probably a good idea to consider switching. Also not sure if aws sdk works for other s3 vendors.

[koivunej]

I think the tests should be split but we can merge this regardless.

[hlinnaka]

This PR isn't adding it but I'm curious: why do we have our own retry code instead of using a RetryConfig from aws_sdk_s3?

Hm, thats a good question. We wernt using aws sdk in the beginning, it is fairly new. Before that we used rusoto that is nowdeprecated.

This. Also, by doing the retries ourselves, we have more control. IIRC we print a warning after some number of retries, for example.