Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: cdc/cloud-sink-gcs/assume-role failed #89726

Closed
cockroach-teamcity opened this issue Oct 11, 2022 · 2 comments
Closed

roachtest: cdc/cloud-sink-gcs/assume-role failed #89726

cockroach-teamcity opened this issue Oct 11, 2022 · 2 comments
Assignees
@HonoreDB
Labels
C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. T-cdc
Milestone
22.2

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Oct 11, 2022 •
edited by cockroach-jira-scripts
Loading

roachtest.cdc/cloud-sink-gcs/assume-role failed with artifacts on release-22.2.0 @ f7090a81f2e603536a0c4fbaadaa759ef2b0f60a:

test artifacts and logs in: /artifacts/cdc/cloud-sink-gcs/assume-role/run_1
	cdc.go:1744,cdc.go:302,cdc.go:847,test_runner.go:930: max latency was more than allowed: 1m25.253956906s vs 1m0s

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

/cc @cockroachdb/cdc

This test on roachdash | Improve this report!

Jira issue: CRDB-20395
@cockroach-teamcity cockroach-teamcity added branch-release-22.2.0 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Oct 11, 2022
@cockroach-teamcity cockroach-teamcity added this to the 22.2 milestone Oct 11, 2022
@blathers-crl blathers-crl bot added the T-cdc label Oct 11, 2022
@HonoreDB HonoreDB removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Oct 14, 2022
@HonoreDB
Copy link
Contributor

Removing the release blocker tag. assume-role tests are occasionally failing because their latency sometimes rises above a minute twice before shrinking to an acceptable steady latency, while the test only tolerates rising once. I don't think this reflects an actual performance issue with assume role, but even if it does, it's only visible at the p99 level and shouldn't be a release blocker. I'll make a PR to reduce the test noise.

@HonoreDB HonoreDB mentioned this issue Oct 14, 2022
Open
@cockroach-teamcity cockroach-teamcity mentioned this issue Oct 18, 2022
Closed
HonoreDB added a commit to HonoreDB/cockroach that referenced this issue Oct 19, 2022
@HonoreDB
changefeedccl: run assume-role tests under initial_scan=only
cf0f02c 
The latency pattern with assume-role is causing tests to flake,
and we also don't have tests for initial_scan=only, so let's
feed two birds with one scone.

Informs cockroachdb#89726

Release note: None
@cockroach-teamcity cockroach-teamcity mentioned this issue Dec 31, 2022
Closed
@miretskiy
Copy link
Contributor

Dup of roachtest: cdc/cloud-sink-gcs/assume-role failed#86100

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@HonoreDB HonoreDB

Labels
C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. T-cdc
Projects
None yet
Milestone
22.2
Development

No branches or pull requests
3 participants
@HonoreDB @cockroach-teamcity @miretskiy