Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Retry http on error #791

Closed
NobodyXu opened this issue Feb 12, 2023 · 6 comments · Fixed by #922
Closed

Retry http on error #791

NobodyXu opened this issue Feb 12, 2023 · 6 comments · Fixed by #922
Assignees
@NobodyXu
Labels
help wanted Extra attention is needed

Comments

@NobodyXu
Copy link
Member

In our CI, installing miniserve from GH release often failed due to timed out, thus I think we need to retry them on error.

Also:

http2 error: stream error received: refused stream before processing any application logic

The same stream error also happens to cargo-watch and sccache, IDK whether it is a bug or simply due to being overloaded?
@NobodyXu
Copy link
Member Author

P.S. This is related to #776

@NobodyXu NobodyXu mentioned this issue Feb 12, 2023
Merged
NobodyXu added a commit that referenced this issue Feb 13, 2023
@NobodyXu
Feature: Better retry policy in binstalk-downloader (#794)
87686cb 
Fixed #779 #791 

 - Retry request on timeout
 - Retry for `StatusCode::{REQUEST_TIMEOUT, GATEWAY_TIMEOUT}`
 - Add `DEFAULT_RETRY_DURATION_FOR_RATE_LIMIT` for 503/429
   if 503/429 does not give us a header or give us an invalid header on
   when to retry, we would default to
   `DEFAULT_RETRY_DURATION_FOR_RATE_LIMIT`.
 - Fix `Client::get_redirected_final_url`: Retry using `GET` on status code 400..405 + 410
 - Rename remote_exists => remote_gettable & support fallback to GET
   if HEAD fails due to status code 400..405 + 410.
 - Improve `Client::get_stream`: Include url & method in the err of the stream returned

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
@NobodyXu NobodyXu self-assigned this Feb 13, 2023
@NobodyXu NobodyXu added the help wanted Extra attention is needed label Feb 13, 2023
@NobodyXu
Copy link
Member Author

There are still some very strange error in our CI.

From here:

 WARN resolve: Error while checking fetcher invalid url: could not HEAD https://github.com/svenstaro/miniserve: error sending request for url (https://github.com/svenstaro/miniserve): http2 error: stream error received: refused stream before processing any application logic

@passcod Do you have any idea what could cause this?
I have no idea what is going on.

Also this:

 WARN resolve: Error while checking fetcher invalid url: could not HEAD https://github.com/svenstaro/miniserve/releases/download/v0.22.0/miniserve_x86_64-unknown-linux-gnu.tgz: error sending request for url (https://github.com/svenstaro/miniserve/releases/download/v0.22.0/miniserve_x86_64-unknown-linux-gnu.tgz): connection error: connection reset

For this connection reset, we could call fn reqwest::Error::is_connect to detect and retry this.

Also, I think we need to cache call to get_redirected_final_url which is called once for every GhCrateMeta::find.
Our CI took as long as 8m to complete the cargo binstall cargo-binstall sccache miniserve cargo-release cargo-watch, that's way too long.

@NobodyXu
Copy link
Member Author

According to this SSO answer, the "stream error received" might be caused by opening too many concurrent streams, so we should also retry on encountering this error.

@NobodyXu
Copy link
Member Author

I suppose we should also increase the interval in our rate limiting.

@NobodyXu NobodyXu mentioned this issue Feb 14, 2023
Open
@NobodyXu
Copy link
Member Author

I suppose we should also increase the interval in our rate limiting.

That works very well and our CI now finishes with in 7m.

@NobodyXu NobodyXu mentioned this issue Mar 12, 2023
Merged
@NobodyXu
Copy link
Member Author

NobodyXu commented Mar 17, 2023 •
edited
Loading

We got another http error from https://crates.io

  × For crate b3sum: crates.io API error for b3sum: could not GET https://
  │ crates.io/api/v1/crates/b3sum: error sending request for url (https://
  │ crates.io/api/v1/crates/b3sum): connection error: broken pipe
  ├─▶ crates.io API error for b3sum: could not GET https://crates.io/api/v1/
  │   crates/b3sum: error sending request for url (https://crates.io/api/v1/
  │   crates/b3sum): connection error: broken pipe
  ├─▶ could not GET https://crates.io/api/v1/crates/b3sum: error sending
  │   request for url (https://crates.io/api/v1/crates/b3sum): connection
  │   error: broken pipe
  ├─▶ error sending request for url (https://crates.io/api/v1/crates/b3sum):
  │   connection error: broken pipe
  ├─▶ connection error: broken pipe
  ╰─▶ broken pipe

looks like we still need to add more retry logic to binstalk-downloader.

NobodyXu added a commit that referenced this issue Mar 18, 2023
@NobodyXu
Retry http request on connection error
4f4fa0b 
Fixed #791

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
@NobodyXu NobodyXu mentioned this issue Mar 18, 2023
Merged
@NobodyXu NobodyXu mentioned this issue Mar 22, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@NobodyXu NobodyXu

Labels
help wanted Extra attention is needed
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Retry http request on connection error cargo-bins/cargo-binstall
1 participant
@NobodyXu