Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] test framework to test concurrent controllers with linearization #1713

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

tkashem
Copy link
Contributor

@tkashem tkashem commented Apr 7, 2024

No description provided.

@openshift-ci openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 7, 2024
@openshift-ci openshift-ci bot requested review from deads2k and stlaz April 7, 2024 17:21
Copy link
Contributor

openshift-ci bot commented Apr 7, 2024

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: tkashem
Once this PR has been reviewed and has the lgtm label, please assign deads2k for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@tkashem tkashem force-pushed the concurrency-test branch 2 times, most recently from 729d0d1 to fdab34a Compare April 21, 2024 13:35
@@ -183,7 +183,7 @@ func getValidityFromAnnotations(annotations map[string]string) (notBefore time.T

// setSigningCertKeyPairSecret creates a new signing cert/key pair and sets them in the secret
func setSigningCertKeyPairSecret(signingCertKeyPairSecret *corev1.Secret, validity time.Duration) error {
signerName := fmt.Sprintf("%s_%s@%d", signingCertKeyPairSecret.Namespace, signingCertKeyPairSecret.Name, time.Now().Unix())
signerName := fmt.Sprintf("%s_%s@%d", signingCertKeyPairSecret.Namespace, signingCertKeyPairSecret.Name, time.Now().UnixNano())
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That seems to be required for tests only, right? I don't think its suitable for production though, as it may grow too large?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, for tests only, if we want to make sure the annotation changes as expected, otherwise we have to add sleep to the test

)

/*
A controller has this sequence: Get, Create, and Update
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Create here assumes "Delete and Create" on type change or its purely "Create" when the item doesn't exist?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is just an example to demonstrate how the framework generates the API call sequence, we could replace the Create/Delete with DoFoo/DoBar here

d *dispatcher
}

func (a actor) Start() <-chan struct{} {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we have "start" as separate action? Its not a kube-apiserver request, so we don't need it on the graph

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's so that we can linearize the startup logic as well, the test here doesn't instrument the lister logic and a controller can start with a get from the lister.
A lister reacts to the state of the storage, so as long as we linearize all the API invocations we are good, this also means that the startup logic must me linearized as well.

Copy link
Contributor

openshift-ci bot commented Apr 27, 2024

@tkashem: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/verify-deps 246c2d7 link true /test verify-deps
ci/prow/unit 246c2d7 link true /test unit

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@tkashem
Copy link
Contributor Author

tkashem commented Jun 11, 2024

graphic visualization of an example run, where two controller instances {0,1} run concurrently, each path from the start node to a leaf node represents a linearized run where the API calls from the two instances are interleaved. The path marked in red shows a failed run.

image

@vrutkovs
Copy link
Member

This may be useful in case we won't be able to avoid having a proper leader election for copies of the controller - or we find a resource managed by multiple controllers.

@openshift-bot
Copy link

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

@openshift-ci openshift-ci bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 11, 2024
@openshift-merge-robot openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 11, 2024
@openshift-merge-robot
Copy link
Contributor

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants