Use curl retry features to workaround transient network problems #440
Conversation
curl after a certain version allows a number of --retry options that can improve resiliency against transient network issues. Use curl retry features and bail out if the system doesn't honor the minimum curl version. Ideally we'd also use `--retry-connrefused` but this parameter got introduced in curl 7.52 which isn't available yet on many distributions such as Ubuntu 16.04.
| local curl_patch_release=$(curl --version | head -1 | cut -d ' ' -f 2,2 | cut -d '.' -f 3,3) | ||
|
|
||
| if [[ $curl_major_version < ${MIN_CURL_VERSION[0]} ]] || \ | ||
| [[ $curl_major_version == ${MIN_CURL_VERSION[0]} && $curl_minor_version < ${MIN_CURL_VERSION[1]} ]] || \ |
There was a problem hiding this comment.
Hmm, I don't think this will work anymore when curl 8.x will be released in the future? While the check is good, I am also not sure whether we can just avoid this check completely because we have tight control over the environments in which this runs (our CI)?
There was a problem hiding this comment.
It should work (I've tested it too) because we are failing only if:
- curl < 7 or
- curl_major == 7 and curl_minor < 12 or
- curl_major == 7 and curl_minor == 12 and curl_patch < 3
everything else passes, including curl >= 8
There was a problem hiding this comment.
Ah, I missed that. Good point.
integration-test.sh
Outdated
| @@ -85,9 +99,11 @@ function set_up() { | |||
| mkdir -p .rally_it/cache | |||
| cd .rally_it/cache | |||
| if [ ! -f elasticsearch-"${ES_METRICS_STORE_VERSION}".tar.gz ]; then | |||
| curl -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-"${ES_METRICS_STORE_VERSION}".tar.gz | |||
| # If curl fails immediately, executing all retries will take up to (2**retries)-1 seconds. | |||
| curl --retry 8 -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-"${ES_METRICS_STORE_VERSION}".tar.gz || { rm elasticsearch-"${ES_METRICS_STORE_VERSION}".tar.gz; exit 1; } | |||
There was a problem hiding this comment.
Maybe it makes sense to extract the artefact name (elasticsearch-"${ES_METRICS_STORE_VERSION}".tar.gz) as a constant as it is repeated multiple times now?
There was a problem hiding this comment.
Makes sense, will address.
There was a problem hiding this comment.
I pushed 20972a2 for ^^
danielmitterdorfer
added
enhancement
Remove code duplication through the use of common variables for the elasticsearch downloaded artifact and its path. Relates elastic#440 (comment)
|
Thanks @dliappis! LGTM. |
curl after a certain version allows a number of --retry options that
can improve resiliency against transient network issues.
Use curl retry features and bail out if the system doesn't honor the
minimum curl version.
Ideally we'd also use
--retry-connrefusedbut this parameter gotintroduced in curl 7.52 which isn't available yet on many
distributions such as Ubuntu 16.04.