Download retry

[pothos]

• HTTP requests should time out and be retried on timeouts or HTTP errors (503 etc.) - DNS retries are out of scope, similar to how curl is used already: Integration of ue-rs with update-engine post-install script Flatcar#1028 (comment)
• Not a blocker, because we could run a bash loop around it but it's not nice because we don't know if it's a download or verification error, and would wastefully retry on verification errors then

[pothos]

Until then, to distinguish download from verification errors we can also do the following: Download into a .download folder and only if finished, move to the .unverified folder. This way the caller can check that downloading worked by looking if the file exists in the .unverified folder.

[pothos]

I think we should also set a connection timeout in the reqwest client builder:
https://docs.rs/reqwest/latest/reqwest/struct.ClientBuilder.html#method.connect_timeout
https://docs.rs/reqwest/latest/reqwest/struct.ClientBuilder.html#method.timeout (can be a problem on slow connections, but with a very large timeout it should be ok and help to eventually make progress)
Note that it needs the tokio timer feature.

In curl we have this:

--retry-delay 1 --retry 60 --retry-connrefused --retry-max-time 60 --connect-timeout 20

In words, we need a retry on temporary http errors, http cancelations, refused connection, and TCP connect timeouts. The read timeout in addition is nice to have when chosen very large.