roachtest/cdc: increase cloud storage assume role acceptable latency #97394
Conversation
Previously, we would observe failures due to the latency jumping to ~1m15s. - https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/8757437 - https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/8534805 The most likely explanation for this is network blips. Previously, the maximum acceptable latency was 1 minute. This change bumps it to 90 seconds to reduce how often we see flakes. Fixes: cockroachdb#96330 Epic: none Release note: None
jayshrivastava
requested review from
herkolategan and
srosenberg
and removed request for
a team
|
This change is |
|
I'm okay bumping limit for now; but I'm not convinced if there isn't some underlying issue at play. |
|
We've been seeing such error messages since 2019: #37307. It's failed over a variety of roachtests and sinks (kafka, cloudstorage, pubsub). For this particular roachtest, we added it in July 2022. The oldest failure is August 2022. |
|
I thought catchup scans were the reason, but this failure over the weekend shows it can happen any time. I feel like every time we see this kind of issue occur, we close the issue as a flake. If we really think they are flakes, then maybe we should allow some percentage error in the latency verifier. Ex. if you specify 60 seconds, it allows max 120 seconds before failing. |
|
On second thought, we should add more logging to better understand the issue rather than naively increase the limit. |
Previously, we would observe failures due to the latency jumping to ~1m15s.
The most likely explanation for this is network blips. Previously, the maximum acceptable latency was 1 minute. This change bumps it to 90 seconds to reduce how often we see flakes.
Fixes: #96330
Epic: none
Release note: None