-
Notifications
You must be signed in to change notification settings - Fork 154
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Make the operator idempotent: resilient to state changes #138
Comments
In k6-operator, wait.PollImmediate is used to achieve timeout with initialize/start/finish, but I think this method is not so good because it blocks the reconsiliation loop. k6-operator/controllers/k6_finish.go Line 34 in 0abaa3a
I think that timeout can be realized without blocking the reconsiliation loop by annotating k6 CR with a timeout time, and each action returns an error if the time exceeds the value of the timeout time. And I think it is possible to requeu until the timeout time by returning the following return value in the part that used wait.PollImmediate.
|
I am running k6 operator in AWS EKS with Fargate and was facing issue #128. Mainly because Fargate takes more than a minute for a pod to get assigned to a node. I increased the timeout to 3 mins in initialise and start. Now it seems to have solved the problem. I understand that it's not a perfect fix because in case of an error in the initialising phase it would still not be able to recover from it. Also, I can run tests simultaneously, so i am not sure what #126 means. |
Lately this issue has been prioritized internally, in part thanks to PR #169 🎉 There has been some confusion around this issue, as in what exactly it involves, as well as lack of clarity around the term "idempotency". So I'll try to clarify a bit 🙂 AFAIK, "idempotency" is often familiar in Kubernetes community but it may not always have exactly the same meaning as math term idempotency. Regardless, it definitely is meant in a broader sense here, in k6-operator: there's a number of inter-connected problems that are all related to how the operator treats the state. Some of them has already been linked here. I nicknamed them "idempotency issues" simply due to lack of another term (and lack of time to search for another term). In order to be more clear, both with the problem and with current expectations, I've made a list of test cases that I look into to pinpoint the exact problems we need to solve. If there's another, significantly different test case that should be added, please feel free to write in the comments. A list of test cases, together with some notes on current behaviour:
These cases are considered in two possible configurations: OSS test run and k6 Cloud (cloud output) test run. At the moment of writing, most of these cases are supported for OSS test runs, with a few notable exceptions. Current conclusion The main problem is with fixing major pain points with k6 Cloud test run. Another problem: the operator does not have any timeout mechanism at the moment. If a test run enters any infinite execution scenario, the operator will wait for completion without timing out. Fixing and clarifying these two problems are considered the main priorities at the moment. Just in case, I don't think it should be a goal to support all of the above test cases but as many as possible and, hopefully, to be clear on what is not going to work. We need to also take into account probability of a test case in real situation VS the effort required to resolve it. |
After reading the issues described above, I think "idempotency" is not a proper term and may result confusing. I suggest using a more generic and possibly more neutral term such as "state handling". |
Thanks for feedback @pablochacin! I kind of agree with you 😄 That's how the label "FSM" appeared after all. Probably the more correct terminology would be ~ "resilient to state changes" which includes idempotency. But the term "idempotency" is rather distinctive even if not exhaustive in this case and it'd be equally confusing to remove it at this point, IMO. |
The main goals of this issue were achieved: the operator acts as "idempotent operator" in most cases. If there are edge cases, it makes sense to open additional issues with specific instructions on how to repeat them. As a follow-up of the list above, the following issues exist: |
This is an umbrella issue for WIP investigation that is being done to resolve the problems mentioned below.
There are two major issues in the k6 Operator that came up recently in different contexts. Firstly, there are hard to repeat bugs like #128 It is likely an indicator that the execution flow itself has a bug to it. One of the possible culprits is repetition of
Reconcile
call: this repetition is an expected behaviour but it doesn't seem that current setup deals with it properly. Essentially, the operator is not being idempotent.Second issue is with the blocked simultaneous test run executions which came up in #126. It must be said that it was not an officially declared feature but apparently, the operator was indeed used in this way, and now it became impossible due to the need to store the additional state vars here:
k6-operator/controllers/k6_initialize.go
Lines 29 to 31 in 0abaa3a
More precisely, duration of the test run is blocking since #112 so this change must be reverted or, better yet, the duration must be stored as part of status to avoid the need to block in the first place.
Both of these issues are tied to how operator handles the state of a test run. Currently operator relies on the string field
stage
of CRD's status to direct execution flow. Given the above, it no longer seems as the correct solution. The Kubernetes API convention essentially promises eventual consistency of spec and status of the resource. So the execution flow of the operator must be updated to handle the declarative approach of Kubernetes API correctly.Additional related info. The conditions enhancement can be referenced to add the informational part of the status of CRD.
Relevant comments would be welcome.
The text was updated successfully, but these errors were encountered: