Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

K8SSAND-1885 ⁃ Stabilize the unit/integration tests flakes #761

Closed
adejanovski opened this issue Nov 10, 2022 · 0 comments · Fixed by #782
Closed

K8SSAND-1885 ⁃ Stabilize the unit/integration tests flakes #761

adejanovski opened this issue Nov 10, 2022 · 0 comments · Fixed by #782
Assignees
Labels
done Issues in the state 'done'

Comments

@adejanovski
Copy link
Contributor

adejanovski commented Nov 10, 2022

We recently started seeing more and more transient errors on the unit/integration tests.
Looking at the logs, it seems like the add-dc tests are responsible for a majority of these failures:

--- FAIL: TestK8ssandraCluster (239.38s)
    --- FAIL: TestK8ssandraCluster/AddDcToExistingCluster (54.51s)
        --- FAIL: TestK8ssandraCluster/AddDcToExistingCluster/SchemaDisagreementOnStargate (2.80s)
            add_dc_test.go:104: wait for the CassandraInitialized condition to be set
            add_dc_test.go:718: add Stargate and Reaper
            add_dc_test.go:560: check that stargate sg1 is created
            add_dc_test.go:563: update stargate sg1 status to ready
            add_dc_test.go:575: check that reaper reaper1 is created
            add_dc_test.go:578: update reaper reaper1 status to ready
            add_dc_test.go:734: add dc2 to cluster
            add_dc_test.go:759: 
                	Error Trace:	add_dc_test.go:759
                	            				add_dc_test.go:584
                	            				add_dc_test.go:223
                	Error:      	Received unexpected error:
                	            	Operation cannot be fulfilled on k8ssandraclusters.k8ssandra.io "add-dc-test": the object has been modified; please apply your changes to the latest version and try again
                	Test:       	TestK8ssandraCluster/AddDcToExistingCluster/SchemaDisagreementOnStargate
                	Messages:   	failed to add dc to K8ssandraCluster

This is starting to fail too often for us to re-run failed jobs now, and we should fix whatever race condition is happening here.

@sync-by-unito sync-by-unito bot changed the title Stabilize the unit/integration tests flakes K8SSAND-1885 ⁃ Stabilize the unit/integration tests flakes Nov 10, 2022
@adejanovski adejanovski added ready Issues in the state 'ready' in-progress Issues in the state 'in-progress' and removed ready Issues in the state 'ready' in-progress Issues in the state 'in-progress' labels Nov 16, 2022
@adejanovski adejanovski added in-progress Issues in the state 'in-progress' and removed ready Issues in the state 'ready' labels Nov 25, 2022
@adejanovski adejanovski self-assigned this Nov 25, 2022
@adejanovski adejanovski mentioned this issue Nov 25, 2022
5 tasks
@adejanovski adejanovski added ready-for-review Issues in the state 'ready-for-review' and removed in-progress Issues in the state 'in-progress' labels Nov 25, 2022
@adejanovski adejanovski added done Issues in the state 'done' and removed ready-for-review Issues in the state 'ready-for-review' labels Nov 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
done Issues in the state 'done'
Projects
No open projects
Archived in project
Development

Successfully merging a pull request may close this issue.

1 participant