Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

storage: TestRetryConformance failed #9178

Closed
flaky-bot bot opened this issue Dec 15, 2023 · 9 comments · Fixed by googleapis/google-api-go-client#2634
Closed

storage: TestRetryConformance failed #9178

flaky-bot bot opened this issue Dec 15, 2023 · 9 comments · Fixed by googleapis/google-api-go-client#2634
Assignees
@tritone
Labels
api: storage Issues related to the Cloud Storage API. flakybot: flaky Tells the Flaky Bot not to close or comment on this issue. flakybot: issue An issue filed by the Flaky Bot. Should not be added manually. priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Comments

@flaky-bot
Copy link

flaky-bot bot commented Dec 15, 2023

Note: #8185 was also for this test, but it was closed more than 10 days ago. So, I didn't mark it flaky.

commit: cae54ed
buildURL: Build Status, Sponge
status: failed
@flaky-bot flaky-bot bot added flakybot: issue An issue filed by the Flaky Bot. Should not be added manually. priority: p1 Important issue which blocks shipping the next release. Will be fixed prior to next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. labels Dec 15, 2023
@product-auto-label product-auto-label bot added the api: storage Issues related to the Cloud Storage API. label Dec 15, 2023
@flaky-bot Flaky Bot
Copy link
Author

flaky-bot bot commented Dec 15, 2023

Test passed for commit 69c49f2 (Build Status, Sponge)! Closing this issue.

@flaky-bot flaky-bot bot closed this as completed Dec 15, 2023
@flaky-bot flaky-bot bot reopened this Dec 22, 2023
@flaky-bot flaky-bot bot added the flakybot: flaky Tells the Flaky Bot not to close or comment on this issue. label Dec 22, 2023
@flaky-bot Flaky Bot
Copy link
Author

flaky-bot bot commented Dec 22, 2023

Looks like this issue is flaky. 😟

I'm going to leave this open and stop commenting.

A human should fix and close this.

commit: ec9b526
buildURL: Build Status, Sponge
status: failed

@BrennaEpp
Copy link
Contributor 

=== RUN   TestRetryConformance/2-[return-reset-connection_return-reset-connection]-storage.objects.insert-0
    retry_conformance_test.go:753: roundtrip error (may be expected): write tcp 127.0.0.1:60652->127.0.0.1:9000: write: broken pipe
        request: POST /upload/storage/v1/b/bucket-20231222-35551646713864-0060/o?alt=json&ifGenerationMatch=0&name=new-object.txt&prettyPrint=false&projection=full&uploadType=multipart HTTP/1.1
        Host: localhost:9000
        Content-Type: ...
        
    retry_conformance_test.go:539: want success, got Writer.Close: Post "[http://localhost:9000/upload/storage/v1/b/bucket-20231222-35551646713864-0060/o?alt=json&ifGenerationMatch=0&name=new-object.txt&prettyPrint=false&projection=full&uploadType=multipart](https://www.google.com/url?q=http://localhost:9000/upload/storage/v1/b/bucket-20231222-35551646713864-0060/o?alt%3Djson%26ifGenerationMatch%3D0%26name%3Dnew-object.txt%26prettyPrint%3Dfalse%26projection%3Dfull%26uploadType%3Dmultipart&sa=D)": write tcp 127.0.0.1:60652->127.0.0.1:9000: write: broken pipe
    retry_conformance_test.go:718: test not completed; unused instructions: map[storage.objects.insert:[return-reset-connection]]

@flaky-bot Flaky Bot
Copy link
Author

flaky-bot bot commented Jan 5, 2024

Oops! Looks like this issue is still flaky. It failed again. 😬

I reopened the issue, but a human will need to close it again.

commit: 82054d0
buildURL: Build Status, Sponge
status: failed

@flaky-bot flaky-bot bot reopened this Jan 5, 2024
@BrennaEpp
Copy link
Contributor

Not sure why I originally closed this issue. Error from the latest comment is the same as above.

@BrennaEpp BrennaEpp added priority: p2 Moderately-important priority. Fix may not be included in next release. and removed priority: p1 Important issue which blocks shipping the next release. Will be fixed prior to next release. labels Jan 24, 2024
@BrennaEpp
Copy link
Contributor 

=== RUN   TestRetryConformance/http-6-[return-reset-connection_return-401]-storage.objects.insert-1
    retry_conformance_test.go:821: test not completed; unused instructions: map[storage.objects.insert:[return-401]]

^ Recently got this error as well.

tritone added a commit to tritone/google-cloud-go that referenced this issue Jun 12, 2024
@tritone
fix(storage): retry broken pipe error
85fd85b 
This is another retryable net.OpError that can arise when a
connection has been closed. So far we've only seen this in
TestRetryConformance cases rather than in customer issues, but
there is no reason not to do a retry and we think it can cut
down on flakes from this test.

I'm going to do another PR to add this to google-api-go-client
since it often occurs on uploads.

Updates googleapis#9178
@tritone tritone mentioned this issue Jun 12, 2024
Merged
tritone added a commit that referenced this issue Jun 12, 2024
@tritone
fix(storage): retry broken pipe error (#10374)
2f4daa1 
This is another retryable net.OpError that can arise when a
connection has been closed. So far we've only seen this in
TestRetryConformance cases rather than in customer issues, but
there is no reason not to do a retry and we think it can cut
down on flakes from this test.

I'm going to do another PR to add this to google-api-go-client
since it often occurs on uploads.

Updates #9178
@release-please release-please bot mentioned this issue Jun 12, 2024
Merged
tritone added a commit to tritone/google-api-go-client that referenced this issue Jun 12, 2024
@tritone
fix(storage): update shouldRetry for GCS uploads
0c0ad4d 
A few fixes to shouldRetry, which is used only for storage
uploads.

* Remove attempt to unwrap syscall errors and use string matching
instead, as is used in the veneer client layer for storage.
* Use errors.Is and appropriate sentinel for net.ErrClosed.
* Add broken pipe error, which is causing flakes in veneer layer
tests (see googleapis/google-cloud-go#10374
* Add unit test for shouldRetry.

Closes googleapis/google-cloud-go#9178
@tritone tritone mentioned this issue Jun 12, 2024
Merged
tritone added a commit to googleapis/google-api-go-client that referenced this issue Jun 12, 2024
@tritone
fix(internal/gensupport): update shouldRetry for GCS uploads (#2634)
ea513cb 
* fix(storage): update shouldRetry for GCS uploads

A few fixes to shouldRetry, which is used only for storage
uploads.

* Remove attempt to unwrap syscall errors and use string matching
instead, as is used in the veneer client layer for storage.
* Use errors.Is and appropriate sentinel for net.ErrClosed.
* Add broken pipe error, which is causing flakes in veneer layer
tests (see googleapis/google-cloud-go#10374
* Add unit test for shouldRetry.

Closes googleapis/google-cloud-go#9178

* don't use xerrors
@tritone
Copy link
Contributor

tritone commented Jun 12, 2024

Beefed up retry logic for broken pipe errors in this repo and in apiary. We should see fewer flakes on this test especially once we pick up the new release of the apiary client.

@flaky-bot flaky-bot bot reopened this Jun 17, 2024
@flaky-bot Flaky Bot
Copy link
Author

flaky-bot bot commented Jun 17, 2024

Oops! Looks like this issue is still flaky. It failed again. 😬

I reopened the issue, but a human will need to close it again.

commit: 1e2af81
buildURL: Build Status, Sponge
status: failed

@tritone
Copy link
Contributor

tritone commented Jun 17, 2024

Latest failure should be fixed by the Apiary library patch as soon as we pick up the new release.

@tritone tritone closed this as completed Jun 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tritone tritone

Labels
api: storage Issues related to the Cloud Storage API. flakybot: flaky Tells the Flaky Bot not to close or comment on this issue. flakybot: issue An issue filed by the Flaky Bot. Should not be added manually. priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(internal/gensupport): update shouldRetry for GCS uploads tritone/google-api-go-client
2 participants
@tritone @BrennaEpp