Leader remove runlocal #7

jmrodri · 2020-07-22T22:50:49Z

No description provided.

coveralls · 2020-07-22T22:53:48Z

Pull Request Test Coverage Report for Build 181321825

23 of 36 (63.89%) changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-25.9%) to 74.074%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
leader/leader.go	23	36	63.89%

Totals
Change from base Build 180379840:	-25.9%
Covered Lines:	220
Relevant Lines:	297

💛 - Coveralls

leader/leader.go

leader/leader_test.go

joelanford

One more review round (sorry!)

And another thing that just popped into my head. It occurs to me that controller-runtime has some pre- and post- leader-election runnables as part of the manager. I wonder if it would be possible to shim this leader election implementation into controller-runtime. This point is not a PR a blocker, but it would be nice to handle in a follow-up if it is possible. I'll investigate and report back.

joelanford · 2020-07-24T04:08:19Z

leader/leader_test.go

+var _ = Describe("Leader", func() {
+


I think this Describe layer is unnecessary. Every test in this suite (i.e. package) is a leader test.

If I remove it, it doesn't seem to work. I renamed it to Leader election to better describe it. I would say, it works so I'd rather not change this.

joelanford · 2020-07-24T04:16:22Z

leader/leader.go

 var log = logf.Log.WithName("leader")

 // maxBackoffInterval defines the maximum amount of time to wait between
 // attempts to become the leader.
 const maxBackoffInterval = time.Second * 16

+type Option func(*config) error


Ah crap. I totally messed this up. Now that I'm seeing this, I feel like it's a little user-hostile. If we don't export Config, users can't provide their own Option implementations.

Given that we only have one field to set, it might not seem important now, but if we add more fields in the future, we want to give users the flexibility to write their own Options.

Opening this up later would be a breaking change to the Option definition.

So given all of this, I think we want:

type Option func(*Config) error type Config struct { ... } // now that Config is exported, setDefaults would need // to be unexported since this only needs to be called // by us after all options have run. func (c *Config) setDefaults() error { ... }

Sorry for the back and forth!

Reverted the commit that unexported Config. I then added a new commit to unexport just setDefaults

joelanford · 2020-07-24T05:10:21Z

Following up about controller-runtime manager integration.

It doesn't seem immediately do-able given controller-runtime's direct use of resourcelock.Interface, which seems to have a hardcoded expectation of using leases.

Here's where the manager starts the pre-leader-election runnables vs the post-leader-election runnables: https://github.com/kubernetes-sigs/controller-runtime/blob/ab02953d0bcb6467333fcc5809eb1df83a8d89c4/pkg/manager/internal.go#L465-L478.

The only way I can see this working would be for controller-runtime to give callers a hook into startLeaderElection to override the default. Perhaps it could abstract the leader election implementation to something that allows a LeaderCallbacks to be injected and implements Run(context.Context). So perhaps:

type LeaderElector interface {
    Run(context.Context)
}

And then a new manager Option field:

type Option struct {
	// NewLeaderElector overrides the default lease-based leader elector
	// and allows callers to use their own leader election implementation.
	NewLeaderElector func(leaderelection.LeaderCallbacks) LeaderElector
}

This would definitely require a controller-runtime proposal and thorough discussion, so not anything that could happen anytime really soon.

camilamacedo86 · 2020-07-24T10:33:48Z

leader/leader.go

 // Become ensures that the current pod is the leader within its namespace. If
 // run outside a cluster, it will skip leader election and return nil. It
 // continuously tries to create a ConfigMap with the provided name and the
 // current pod set as the owner reference. Only one can exist at a time with
 // the same name, so the pod that successfully creates the ConfigMap is the
 // leader. Upon termination of that pod, the garbage collector will delete the
 // ConfigMap, enabling a different pod to become the leader.
-func Become(ctx context.Context, lockName string) error {
+func Become(ctx context.Context, lockName string, opts ...Option) error {


We replaced the sdk impl for the controller-runtime one. So, I do not think that we need to keep it at all.

@joelanford why we will keep our implementation since all opers now are using the controller-runtime one? If yes, i need to revert the change made in operator-framework/operator-sdk#3514.

c/c @jmrodri

@camilamacedo86 my understanding was that this is a leader for life and we kept it for someone that wanted to have leader for life. It will not be used in the SDK but could be used by an operator developer.

A lot of existing operators are using the leader-for-life implementation. They'll either need a way to continue using that, or will need an easy path to migrating to the lease-based election.

These days the lease-based election is easy to use and robust, which hasn't always been the case. But even still, each pattern has advantages and disadvantages.

joelanford · 2020-07-24T14:27:22Z

leader/leader_test.go

+
+			err := Become(context.TODO(), "leader-test", WithClient(client))
+			Expect(err).Should(BeNil())
+		})


Can we add a test case for returning ErrNoNamespace from Become and using errors.Is(err, ErrNoNamespace)?

joelanford

/lgtm

openshift-ci-robot · 2020-07-24T15:07:40Z

New changes are detected. LGTM label has been removed.

openshift-ci-robot requested review from estroz and varshaprasad96 July 22, 2020 22:50

jmrodri changed the title ~~Leader remove runlocal~~ WIP: Leader remove runlocal Jul 22, 2020

openshift-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 22, 2020

joelanford mentioned this pull request Jul 22, 2020

Operator SDK 1.0 Meta-issue operator-framework/operator-sdk#3327

Closed

92 tasks

joelanford reviewed Jul 23, 2020

View reviewed changes

leader/leader.go Outdated Show resolved Hide resolved

leader/leader.go Outdated Show resolved Hide resolved

jmrodri force-pushed the leader-remove-runlocal branch from 1b6db72 to 9eabf9d Compare July 23, 2020 20:30

jmrodri changed the title ~~WIP: Leader remove runlocal~~ Leader remove runlocal Jul 23, 2020

openshift-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 23, 2020

jmrodri added 8 commits July 23, 2020 17:31

Remove runlocal bits

eb4eaf2

Add ginkgo tests for leader election

909be0b

Replace defaultNS with a function for better testing

34e0a50

Export ErrNoNamespace

ed7bb5a

Add test for getPod

380e99f

Add myOwnerRef test

2aa15df

Move Unsetenv into the It test

32ab0e9

Added options to Become to make it more testable

affe8aa

jmrodri force-pushed the leader-remove-runlocal branch from 9eabf9d to affe8aa Compare July 23, 2020 21:33

joelanford reviewed Jul 23, 2020

View reviewed changes

leader/leader.go Outdated Show resolved Hide resolved

joelanford reviewed Jul 23, 2020

View reviewed changes

leader/leader_test.go Outdated Show resolved Hide resolved

joelanford reviewed Jul 24, 2020

View reviewed changes

camilamacedo86 reviewed Jul 24, 2020

View reviewed changes

joelanford reviewed Jul 24, 2020

View reviewed changes

jmrodri added 2 commits July 24, 2020 10:28

simplify readNamespace

3127638

change description

dac301c

jmrodri force-pushed the leader-remove-runlocal branch from 769d3ca to dac301c Compare July 24, 2020 14:40

unexport setDefaults

81cc900

joelanford approved these changes Jul 24, 2020

View reviewed changes

openshift-ci-robot assigned joelanford Jul 24, 2020

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Jul 24, 2020

Ensure Become will return ErrNoNamespace

b98972b

openshift-ci-robot removed the lgtm Indicates that a PR is ready to be merged. label Jul 24, 2020

jmrodri merged commit dca3fec into operator-framework:main Jul 24, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Leader remove runlocal #7

Leader remove runlocal #7

jmrodri commented Jul 22, 2020

coveralls commented Jul 22, 2020 •

edited

Loading

joelanford left a comment

joelanford Jul 24, 2020

jmrodri Jul 24, 2020

joelanford Jul 24, 2020

jmrodri Jul 24, 2020

joelanford commented Jul 24, 2020

camilamacedo86 Jul 24, 2020 •

edited

Loading

jmrodri Jul 24, 2020

mhrivnak Jul 24, 2020

joelanford Jul 24, 2020

joelanford left a comment

openshift-ci-robot commented Jul 24, 2020

Leader remove runlocal #7

Leader remove runlocal #7

Conversation

jmrodri commented Jul 22, 2020

coveralls commented Jul 22, 2020 • edited Loading

Pull Request Test Coverage Report for Build 181321825

💛 - Coveralls

joelanford left a comment

Choose a reason for hiding this comment

joelanford Jul 24, 2020

Choose a reason for hiding this comment

jmrodri Jul 24, 2020

Choose a reason for hiding this comment

joelanford Jul 24, 2020

Choose a reason for hiding this comment

jmrodri Jul 24, 2020

Choose a reason for hiding this comment

joelanford commented Jul 24, 2020

camilamacedo86 Jul 24, 2020 • edited Loading

Choose a reason for hiding this comment

jmrodri Jul 24, 2020

Choose a reason for hiding this comment

mhrivnak Jul 24, 2020

Choose a reason for hiding this comment

joelanford Jul 24, 2020

Choose a reason for hiding this comment

joelanford left a comment

Choose a reason for hiding this comment

openshift-ci-robot commented Jul 24, 2020

coveralls commented Jul 22, 2020 •

edited

Loading

camilamacedo86 Jul 24, 2020 •

edited

Loading