🐛 Update Machine Status with Error Reason and Error Message if infra object is deleted #1458

kashifest · 2019-09-27T12:51:01Z

What this PR does / why we need it:
Updates the machine status with error reason and error message when machine is in running phase and the infrastructure object is deleted
Which issue(s) this PR fixes (optional, in fixes #(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #1398

k8s-ci-robot · 2019-09-27T12:51:15Z

Hi @kashifest. Thanks for your PR.

I'm waiting for a kubernetes-sigs or kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

kashifest · 2019-09-27T12:52:28Z

/assign @ncdc

detiber · 2019-09-27T13:33:07Z

controllers/machine_controller_phases.go

@@ -78,6 +78,13 @@ func (r *MachineReconciler) reconcileExternal(ctx context.Context, m *clusterv1.
 	obj, err := external.Get(r.Client, ref, m.Namespace)
 	if err != nil {
 		if apierrors.IsNotFound(err) {
+			if m.Status.InfrastructureReady && m.Status.GetTypedPhase() == clusterv1.MachinePhaseRunning {


We shouldn't have logic comparing against the Machine "Phase", it is meant to only be a user friendly display value.

This check should also probably be pushed up to where reconcileExternal is called for the InfrastructureRef, since a missing Bootstrap.ConfigRef should not result in an error.

I checked the machine phase to make sure this block of code is executed only once. If it is not recommended, I can remove that

ncdc

Once we get it sorted where this logic should live, could you please make sure to add test case(s) for it? We have e.g. TestReconcileInfrastructure and TestReconcileBootstrap.

ncdc · 2019-09-27T13:27:59Z

controllers/machine_controller_phases.go

@@ -78,6 +78,13 @@ func (r *MachineReconciler) reconcileExternal(ctx context.Context, m *clusterv1.
 	obj, err := external.Get(r.Client, ref, m.Namespace)
 	if err != nil {
 		if apierrors.IsNotFound(err) {
+			if m.Status.InfrastructureReady && m.Status.GetTypedPhase() == clusterv1.MachinePhaseRunning {
+				// Infra object went missing after the machine was up and running
+				klog.Infof("Infrastructure object %v %q went missing. Updating Machine Status", ref.GroupVersionKind(), ref.Name)


We don't want to add any new raw klog calls. Please use r.Log instead.

ncdc · 2019-09-27T13:28:42Z

controllers/machine_controller_phases.go

@@ -78,6 +78,13 @@ func (r *MachineReconciler) reconcileExternal(ctx context.Context, m *clusterv1.
 	obj, err := external.Get(r.Client, ref, m.Namespace)
 	if err != nil {
 		if apierrors.IsNotFound(err) {
+			if m.Status.InfrastructureReady && m.Status.GetTypedPhase() == clusterv1.MachinePhaseRunning {


We should not check the phase - that is for human consumption. Checking InfrastructureReady is sufficient.

If we place this logic here, it applies to all external references - both the bootstrap ref and the infrastructure ref. @vincepri do you this is appropriate?

I checked the machine phase to make sure this block of code is executed only once. If it is not recommended, I can remove that

If we are not currently short-circuiting machine reconciliation if the error reason/error message are set, we need to be doing that.

@ncdc I am not sure if I understood correctly, this check was allowing to setting the error reason/message and logging only once, but it did not short-circuit reconciliation

In what situation would it be executing this more than once?

In each call to reconcileExternal in my current version of the code?

Ok, once you move this to reconcileInfrastructure, I think we also need to add code toward the top of Reconcile that checks machine.status.errorReason/errorMessage, and if either is set, it stops. This would need to be before the call to patch.NewHelper. Do you agree @vincepri?

Yes, the short-circuit behavior is definitely missing. Let's follow-up with a different PR marked with ⚠️ for that

controllers/machine_controller_phases.go

ncdc · 2019-09-27T13:29:54Z

controllers/machine_controller_phases.go

+			if m.Status.InfrastructureReady && m.Status.GetTypedPhase() == clusterv1.MachinePhaseRunning {
+				// Infra object went missing after the machine was up and running
+				klog.Infof("Infrastructure object %v %q went missing. Updating Machine Status", ref.GroupVersionKind(), ref.Name)
+				machineStatusError := capierrors.MachineStatusError("Invalid Configuration")


@vincepri do we have any guidance on the content and format for error reasons?

We should avoid creating new ErrorReason(s) dynamically. We can add new ones to the constants defined in the package. Let's focus on having a good error message in ErrorMessage though

ncdc · 2019-09-27T13:30:16Z

controllers/machine_controller_phases.go

+				klog.Infof("Infrastructure object %v %q went missing. Updating Machine Status", ref.GroupVersionKind(), ref.Name)
+				machineStatusError := capierrors.MachineStatusError("Invalid Configuration")
+				m.Status.ErrorReason = &machineStatusError
+				m.Status.ErrorMessage = pointer.StringPtr("Invalid Configuration")


This should be descriptive and include the GVK (GroupVersionKind) and name of the resource that went missing.

kashifest · 2019-09-27T14:13:39Z

@ncdc I can add the test cases also. Now I am waiting to know where should I put the check in reconcileExternal or reoncileInfrastructure

ncdc · 2019-09-27T14:36:14Z

+1 to reconcileInfrastructure

vincepri · 2019-09-27T15:08:43Z

Before merging, let's find a PR title that communicates better the changes we're making. PR titles go in release notes for users to read

vincepri · 2019-09-27T15:08:51Z

/ok-to-test

kashifest · 2019-09-27T21:08:21Z

@ncdc @vincepri @detiber I have addressed most of the issues commented above. I did not move the code to reconcileInfrastructure though, since reconcileExternal changes the error cause for reconciliation. This makes it a bit tricky to get the actual error in ```reconcileInfrastructure``.

detiber · 2019-09-30T14:30:34Z

controllers/machine_controller_phases.go

@@ -197,6 +197,14 @@ func (r *MachineReconciler) reconcileInfrastructure(ctx context.Context, m *clus
 	if infraConfig == nil && err == nil {
 		return nil
 	} else if err != nil {
+		if m.Status.InfrastructureReady {


This should probably also test apierrors.IsNotFound(errors.Cause(err)), to ensure that the error is related to the resource being missing rather than another type of error.

I tried that. It seems reconcileExternal is consuming that error and only returning the requeue error

Arrgh, indeed it is... I'm wondering if we need to refactor reconcileExternal a bit so that we are not returning a wrapped RequeueAfter error there.... It would probably require registering the watcher for the resource prior to actually retrieving one, though... @vincepri @ncdc thoughts?

We could either move the requeue error up to the calling functions, or (less preferred, but quicker) string match "not found" on the incoming error

@vincepri @detiber I chose the quicker but not preferred way ... the placement of the requeue error in calling function is bit of a design issue since it also requires change in reconcileBootstrap , which I thought is bit out of scope of this bug, shall we follow this in another PR?

I'm good with this as a short term solution as long as a tracking issue is created to clean up later.

controllers/machine_controller_phases.go

vincepri · 2019-09-30T18:58:53Z

controllers/machine_controller_phases.go

+			// Infra object went missing after the machine was up and running
+			r.Log.Error(err, "Machine infrastructure reference has been deleted after being ready, setting failure state")
+			machineStatusError := capierrors.InvalidConfigurationMachineError
+			m.Status.ErrorReason = &machineStatusError


Suggested change

m.Status.ErrorReason = &machineStatusError

m.Status.ErrorReason = capierrors.ClusterStatusErrorPtr(capierrors.InvalidConfigurationMachineError)

@vincepri Thanks. Didn't know this exist. Shouldn't it be MachineStatusErrorPtr instead of ClusterStatusErrorPtr

vincepri

/approve
/assign @detiber

k8s-ci-robot · 2019-09-30T19:40:01Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kashifest, vincepri

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [vincepri]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

detiber · 2019-09-30T20:56:11Z

/lgtm

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Sep 27, 2019

k8s-ci-robot requested review from detiber and ncdc September 27, 2019 12:51

k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Sep 27, 2019

k8s-ci-robot assigned ncdc Sep 27, 2019

detiber reviewed Sep 27, 2019

View reviewed changes

ncdc reviewed Sep 27, 2019

View reviewed changes

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Sep 27, 2019

kashifest changed the title ~~🐛 Update Machine Status if infra object is deleted~~ 🐛 Update Machine Status with Error Reason and Error Message if infra object is deleted Sep 27, 2019

kashifest force-pushed the bugfix/infraObject-deletion-Kashif branch 2 times, most recently from 29f3724 to 166cc55 Compare September 27, 2019 21:03

kashifest changed the title ~~🐛 Update Machine Status with Error Reason and Error Message if infra object is deleted~~ 🐛 Update Machine Status with Error Reason and Error Message if infra/bootstrap object is deleted Sep 27, 2019

kashifest force-pushed the bugfix/infraObject-deletion-Kashif branch 3 times, most recently from b779cfb to 0a59225 Compare September 30, 2019 14:18

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Sep 30, 2019

kashifest changed the title ~~🐛 Update Machine Status with Error Reason and Error Message if infra/bootstrap object is deleted~~ 🐛 Update Machine Status with Error Reason and Error Message if infra object is deleted Sep 30, 2019

kashifest requested review from ncdc, detiber and vincepri September 30, 2019 14:25

detiber reviewed Sep 30, 2019

View reviewed changes

vincepri reviewed Sep 30, 2019

View reviewed changes

kashifest force-pushed the bugfix/infraObject-deletion-Kashif branch from 0a59225 to 160b664 Compare September 30, 2019 18:55

vincepri reviewed Sep 30, 2019

View reviewed changes

Set Machine Error Status if infra object deleted

c62129c

kashifest force-pushed the bugfix/infraObject-deletion-Kashif branch from 160b664 to c62129c Compare September 30, 2019 19:14

kashifest requested review from detiber and vincepri September 30, 2019 19:19

vincepri approved these changes Sep 30, 2019

View reviewed changes

k8s-ci-robot assigned detiber Sep 30, 2019

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 30, 2019

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 30, 2019

k8s-ci-robot merged commit 3046e2f into kubernetes-sigs:master Sep 30, 2019

kashifest deleted the bugfix/infraObject-deletion-Kashif branch October 1, 2019 06:16

	m.Status.ErrorReason = &machineStatusError
	m.Status.ErrorReason = capierrors.ClusterStatusErrorPtr(capierrors.InvalidConfigurationMachineError)

🐛 Update Machine Status with Error Reason and Error Message if infra object is deleted #1458

🐛 Update Machine Status with Error Reason and Error Message if infra object is deleted #1458

Conversation

kashifest commented Sep 27, 2019

k8s-ci-robot commented Sep 27, 2019

kashifest commented Sep 27, 2019

Choose a reason for hiding this comment

kashifest Sep 27, 2019 • edited Loading

Choose a reason for hiding this comment

ncdc left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kashifest Sep 27, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kashifest Sep 27, 2019 • edited Loading

Choose a reason for hiding this comment

kashifest commented Sep 27, 2019

ncdc commented Sep 27, 2019

vincepri commented Sep 27, 2019

vincepri commented Sep 27, 2019

kashifest commented Sep 27, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kashifest Sep 30, 2019 • edited Loading

Choose a reason for hiding this comment

vincepri left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Sep 30, 2019

detiber commented Sep 30, 2019

kashifest Sep 27, 2019 •

edited

Loading

kashifest Sep 27, 2019 •

edited

Loading

kashifest Sep 27, 2019 •

edited

Loading

kashifest Sep 30, 2019 •

edited

Loading