leader: check the node status for the leader-election (#24) #25

HyungJune · 2020-08-06T06:50:15Z

Description of the change:
node-check for leader re-election #24

Motivation for the change:
#24
Check the condition of the node where the leader pod is running with [Node.Type == "NodeReady" && Node.Status != "ConditionTrue"] and when the node has been failed, delete the leaderPod (only mark the pod with 'terminating' because the node where the pod is running has been failed) and the configmap lock whose OwnerReference is leaderPod).

This approach can bring more reliability

…work#24)

coveralls · 2020-08-06T06:53:24Z

Pull Request Test Coverage Report for Build 232104749

32 of 40 (80.0%) changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.005%) to 80.047%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
leader/leader.go	32	40	80.0%

Totals
Change from base Build 195168809:	-0.005%
Covered Lines:	341
Relevant Lines:	426

💛 - Coveralls

camilamacedo86 · 2020-08-24T12:14:16Z

leader/leader.go

@@ -167,6 +167,7 @@ func Become(ctx context.Context, lockName string, opts ...Option) error {
 				log.Info("Leader lock configmap owner reference must be a pod.", "OwnerReference", existingOwners[0])
 			default:
 				leaderPod := &corev1.Pod{}
+				// leaderNode := &corev1.Node{}


we should not merge code commented

@camilamacedo86 thanks, I will delete the comment at the new commit.

camilamacedo86 · 2020-08-24T12:15:56Z

leader/leader.go

@@ -167,6 +167,7 @@ func Become(ctx context.Context, lockName string, opts ...Option) error {
 				log.Info("Leader lock configmap owner reference must be a pod.", "OwnerReference", existingOwners[0])


We need to add the logic of the fragments to here as well for we are able to generate the changelog.

c/c @jmrodri

There's really no changelog fragment for the operator-lib. The releases are just a signed tag.

IMO we should have since we should have a way to know what changed in each version/release.

camilamacedo86 · 2020-08-24T12:19:15Z

leader/leader.go

+					log.Info("Deleting the leader.")
+
+					//Mark the termainating status to the leaderPod and Delete the configmap lock
+					if err := deleteLeader(ctx, config.Client, leaderPod, existing); err != nil {


Should we not use the same func across all implementation? Also, why is required to remove the configMap?

@camilamacedo86

The reason i added the function(delete Leader) is for the lint-test. golangci-lint limits Cyclomatic complexity in the function. so we have to divide the code into multiple functions.

if i do not remove the configmap, leader election will not occur imediately when the node where the leader pod is running was 'not ready' status. Because the leader pod was just marked as 'terminating'.(the garbage collector do not delete the configmap until the leader pod is actually deleted)

camilamacedo86 · 2020-08-24T12:20:06Z

leader/leader_test.go

@@ -209,4 +209,137 @@ var _ = Describe("Leader election", func() {
 			Expect(pod.TypeMeta.Kind).To(Equal("Pod"))
 		})
 	})
+


PS.: see that is not all changes covered by the tests. its % decreased.

@camilamacedo86 i will enhance the coverage in the new commit. Do i need to make a new pull-request?

@kasonglee I see you already added tests that is just fine. There is no need for a new commit.

jmrodri · 2020-08-24T13:39:23Z

/assign jmrodri

jmrodri · 2020-08-24T13:57:51Z

/hold until I can review

joelanford · 2020-08-28T14:08:55Z

/test test-all

jmrodri

two minor issue but I will not hold this PR up for it. I can fix those later. So far this looks fine to me.

jmrodri · 2020-10-10T01:39:54Z

leader/leader.go

+					log.Info("the status of the node where operator pod with leader lock was running has been 'notReady'")
+					log.Info("Deleting the leader.")
+
+					//Mark the termainating status to the leaderPod and Delete the configmap lock


Nit: should have a space after // : // Mark
Nit: typo termainating should be terminating.

jmrodri · 2020-10-10T01:40:38Z

leader/leader.go

+func getNode(ctx context.Context, client crclient.Client, nodeName string, node *corev1.Node) error {
+	key := crclient.ObjectKey{Namespace: "", Name: nodeName}
+	err := client.Get(ctx, key, node)
+	if err != nil {
+		log.Error(err, "Failed to get Node", "Node.Name", nodeName)
+		return err
+	}
+	return nil
+}
+
+func isNotReadyNode(ctx context.Context, client crclient.Client, nodeName string) bool {
+	leaderNode := &corev1.Node{}
+	if err := getNode(ctx, client, nodeName, leaderNode); err != nil {
+		return false
+	}
+	for _, condition := range leaderNode.Status.Conditions {
+		if condition.Type == corev1.NodeReady && condition.Status != corev1.ConditionTrue {
+			return true
+		}
+	}
+	return false
+
+}
+
+func deleteLeader(ctx context.Context, client crclient.Client, leaderPod *corev1.Pod, existing *corev1.ConfigMap) error {
+	err := client.Delete(ctx, leaderPod)
+	if err != nil {
+		log.Error(err, "Leader pod could not be deleted.")
+		return err


This is all good. Pretty clean and easy to read.

jmrodri · 2020-10-10T01:41:41Z

leader/leader_test.go

@@ -209,4 +209,137 @@ var _ = Describe("Leader election", func() {
 			Expect(pod.TypeMeta.Kind).To(Equal("Pod"))
 		})
 	})
+


@kasonglee I see you already added tests that is just fine. There is no need for a new commit.

jmrodri · 2020-10-10T01:44:51Z

/hold cancel

jmrodri · 2020-10-10T01:44:58Z

/lgtm
/approve

openshift-ci-robot · 2020-10-10T01:45:01Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jmrodri

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [jmrodri]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

leader: check the node status for the leader-election (operator-frame…

64e709a

…work#24)

openshift-ci-robot requested review from camilamacedo86 and jmrodri August 6, 2020 06:50

camilamacedo86 reviewed Aug 24, 2020

View reviewed changes

openshift-ci-robot assigned jmrodri Aug 24, 2020

openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 24, 2020

leader: enhance the coverage

f362dee

jmrodri approved these changes Oct 10, 2020

View reviewed changes

openshift-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 10, 2020

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Oct 10, 2020

openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 10, 2020

openshift-merge-robot merged commit d8d1759 into operator-framework:main Oct 10, 2020

varshaprasad96 mentioned this pull request Jan 11, 2021

Check the node status when leader re-election. #24

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

leader: check the node status for the leader-election (#24) #25

leader: check the node status for the leader-election (#24) #25

HyungJune commented Aug 6, 2020

coveralls commented Aug 6, 2020 •

edited

Loading

camilamacedo86 Aug 24, 2020

kasonglee Aug 31, 2020 •

edited

Loading

camilamacedo86 Aug 24, 2020

jmrodri Aug 24, 2020

camilamacedo86 Aug 24, 2020

camilamacedo86 Aug 24, 2020

kasonglee Aug 31, 2020 •

edited

Loading

camilamacedo86 Aug 24, 2020

kasonglee Aug 31, 2020

jmrodri Oct 10, 2020

jmrodri commented Aug 24, 2020

jmrodri commented Aug 24, 2020

joelanford commented Aug 28, 2020

jmrodri left a comment

jmrodri Oct 10, 2020

jmrodri Oct 10, 2020

jmrodri Oct 10, 2020

jmrodri commented Oct 10, 2020

jmrodri commented Oct 10, 2020

openshift-ci-robot commented Oct 10, 2020

		@@ -167,6 +167,7 @@ func Become(ctx context.Context, lockName string, opts ...Option) error {
		log.Info("Leader lock configmap owner reference must be a pod.", "OwnerReference", existingOwners[0])

leader: check the node status for the leader-election (#24) #25

leader: check the node status for the leader-election (#24) #25

Conversation

HyungJune commented Aug 6, 2020

coveralls commented Aug 6, 2020 • edited Loading

Pull Request Test Coverage Report for Build 232104749

💛 - Coveralls

Choose a reason for hiding this comment

kasonglee Aug 31, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kasonglee Aug 31, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jmrodri commented Aug 24, 2020

jmrodri commented Aug 24, 2020

joelanford commented Aug 28, 2020

jmrodri left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jmrodri commented Oct 10, 2020

jmrodri commented Oct 10, 2020

openshift-ci-robot commented Oct 10, 2020

coveralls commented Aug 6, 2020 •

edited

Loading

kasonglee Aug 31, 2020 •

edited

Loading

kasonglee Aug 31, 2020 •

edited

Loading