[0.1] Delete machines #1180

liztio · 2019-07-22T19:58:08Z

What this PR does / why we need it:
Some pieces of cluster infrastructure (i.e. VPCs for CAPA) cannot be successfully deleted until after after the machines in the cluster are all deleted. This pull requests blocks cluster deletion until all owned resources are deleted.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Part of #985 but it likely won't fix it until/unless we fix #1190

Special notes for your reviewer:
This only covers Machines, MachineSets, and MachineDeployments

Please confirm that if this PR changes any image versions, then that's the sole change this PR makes.

Release note:

Deleting a cluster will block until all owned MachineDeployments, MachineSets, and Machines are deleted.

vincepri

Looks good so far! Few comments

pkg/controller/cluster/cluster_controller.go

liztio · 2019-07-23T15:17:05Z

/assign @ncdc

ncdc · 2019-07-23T15:45:43Z

Currently reviewing.

/hold

pending comments

pkg/controller/cluster/cluster_controller.go

pkg/controller/cluster/cluster_controller_test.go

pkg/controller/cluster/cluster_controller.go

liztio · 2019-07-23T21:35:55Z

/test pull-cluster-api-test

vincepri · 2019-07-23T23:06:58Z

/test pull-cluster-api-test

liztio · 2019-07-24T13:59:26Z

/test pull-cluster-api-test

liztio · 2019-07-24T14:06:43Z

well it passes now 🤷‍♀️

pkg/controller/cluster/cluster_controller.go

ncdc · 2019-07-24T15:27:27Z

pkg/controller/cluster/cluster_controller.go

+			for _, child := range children {
+				accessor, err := meta.Accessor(child)
+				if err != nil {
+					return reconcile.Result{}, errors.Wrapf(err, "couldn't create accessor for %T", child)


I agree that Accessor is unlikely to fail, but I'd still like to include it in errList instead of returning early.

pkg/controller/cluster/cluster_controller.go

ncdc · 2019-07-24T15:44:34Z

pkg/controller/cluster/cluster_controller.go

+		children, err := r.listChildren(context.Background(), cluster)
+		if err != nil {
+			klog.Infof("Failed to list dependent objects of cluster %s/%s: %v", cluster.ObjectMeta.Namespace, cluster.ObjectMeta.Name, err)
+			return reconcile.Result{}, errors.WithStack(err)


I don't think we need WithStack and instead we can return err directly because r.listChildren is our function, and it properly wraps/stacks all the errors it returns. WDYT?

I'll be honest, I have no idea what WithStack does, I added it because you suggested it a few blocks up

I guess you only need to add it once?

By default, errors in Go do not include any stack trace information. Without it, it makes it very difficult to determine the source of an error (i.e. source code file & line), especially if the same error can be created in multiple places. github.com/pkg/errors makes it much easier to embed that stack information, and it includes a special custom format printer to print out the stack trace information.

The general guidance I've used in the past is that every line of code you write in a project that returns an error must either already have stack info, or you must wrap it. For first class functions (those defined in your project), you can/should assume that any errors they return are already wrapped. When you are invoking functions from vendored libraries, you can either check to see if they use github.com/pkg/errors, or you can just wrap.

pkg/controller/cluster/cluster_controller.go

ncdc · 2019-07-24T16:19:57Z

1 last comment, then LGTM. I'd like 1 more review:
/assign @vincepri

pkg/controller/cluster/cluster_controller.go

vincepri · 2019-07-24T16:26:35Z

Mostly nits, which can be fixed later.

/lgtm
/hold

Feel free to unhold when ready

pkg/controller/cluster/cluster_controller.go

Co-Authored-By: Andy Goldstein <goldsteina@vmware.com>

ncdc · 2019-07-24T18:02:52Z

/lgtm
/approve

@liztio please remove the hold after you retest against CAPA.

liztio · 2019-07-24T18:49:44Z

I0724 18:48:29.221278       1 scope.go:219] [cluster-actuator]/cluster.k8s.io/v1alpha1/default/whinnyapolis "level"=1 "msg"="updating cluster status"  
I0724 18:48:29.314440       1 cluster_controller.go:196] Cluster object whinnyapolis deletion successful, removing finalizer.

liztio · 2019-07-24T18:49:55Z

/hold cancel

vincepri

/approve

k8s-ci-robot · 2019-07-24T18:50:52Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: liztio, ncdc, vincepri

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [vincepri]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Delete machines before deleting clusters

0ff2de4

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jul 22, 2019

k8s-ci-robot requested review from detiber and justinsb July 22, 2019 19:58

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jul 22, 2019

vincepri reviewed Jul 22, 2019

View reviewed changes

Logging and tests

1b198d7

liztio force-pushed the delete-machines branch 5 times, most recently from 2e61dc8 to ca9ee02 Compare July 23, 2019 14:45

Paginate children and search by cluster name

f8ce245

liztio force-pushed the delete-machines branch from ca9ee02 to f8ce245 Compare July 23, 2019 15:06

k8s-ci-robot assigned ncdc Jul 23, 2019

Delay before requeuing child deletion checks

e109098

liztio force-pushed the delete-machines branch from 48d58a5 to e109098 Compare July 23, 2019 15:18

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 23, 2019

ncdc suggested changes Jul 23, 2019

View reviewed changes

Enhanced error handling

76b6839

ncdc reviewed Jul 23, 2019

View reviewed changes

pkg/controller/cluster/cluster_controller.go Outdated Show resolved Hide resolved

pkg/controller/cluster/cluster_controller.go Outdated Show resolved Hide resolved

pkg/controller/cluster/cluster_controller.go Show resolved Hide resolved

Use EachListItem to iteratate owned objects during cluster deletion

335c65e

ncdc mentioned this pull request Jul 24, 2019

[0.1] MachineDeployments, MachineSets, and Machines should always have a Cluster owner reference if a Cluster exists #1190

Closed

ncdc changed the title ~~Delete machines~~ [0.1] Delete machines Jul 24, 2019

ncdc reviewed Jul 24, 2019

View reviewed changes

liztio force-pushed the delete-machines branch 2 times, most recently from 57fabbe to e6d4114 Compare July 24, 2019 15:57

ncdc reviewed Jul 24, 2019

View reviewed changes

pkg/controller/cluster/cluster_controller.go Outdated Show resolved Hide resolved

Roll-up multiple deletion errors

349628c

liztio force-pushed the delete-machines branch from e6d4114 to 349628c Compare July 24, 2019 16:01

k8s-ci-robot assigned vincepri Jul 24, 2019

ncdc mentioned this pull request Jul 24, 2019

Cluster controller and types for v1alpha2 #1177

Merged

vincepri reviewed Jul 24, 2019

View reviewed changes

pkg/controller/cluster/cluster_controller.go Outdated Show resolved Hide resolved

pkg/controller/cluster/cluster_controller.go Outdated Show resolved Hide resolved

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 24, 2019

Move PointsTo function to util

ac37029

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 24, 2019

ncdc reviewed Jul 24, 2019

View reviewed changes

pkg/controller/cluster/cluster_controller.go Outdated Show resolved Hide resolved

liztio and others added 2 commits July 24, 2019 13:54

Log and skip over failures to create accessors

53b6bf6

return nil instead of an empty list

6b8bb8e

Co-Authored-By: Andy Goldstein <goldsteina@vmware.com>

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 24, 2019

liztio mentioned this pull request Jul 24, 2019

Final control plane node can never be deleted kubernetes-retired/cluster-api-provider-docker#109

Closed

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 24, 2019

vincepri approved these changes Jul 24, 2019

View reviewed changes

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 24, 2019

k8s-ci-robot merged commit e818908 into kubernetes-sigs:release-0.1 Jul 24, 2019

ncdc mentioned this pull request Jul 24, 2019

[0.1] Deleting Cluster does not delete #985

Closed

This was referenced Aug 1, 2019

Cannot delete a machine if there is no cluster #1218

Closed

Cluster: delete children first. #1225

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[0.1] Delete machines #1180

[0.1] Delete machines #1180

liztio commented Jul 22, 2019 •

edited by ncdc

Loading

vincepri left a comment

liztio commented Jul 23, 2019

ncdc commented Jul 23, 2019

liztio commented Jul 23, 2019

vincepri commented Jul 23, 2019

liztio commented Jul 24, 2019

liztio commented Jul 24, 2019

ncdc Jul 24, 2019

ncdc Jul 24, 2019

liztio Jul 24, 2019

liztio Jul 24, 2019

ncdc Jul 24, 2019

ncdc commented Jul 24, 2019

vincepri commented Jul 24, 2019

ncdc commented Jul 24, 2019

liztio commented Jul 24, 2019

liztio commented Jul 24, 2019

vincepri left a comment

k8s-ci-robot commented Jul 24, 2019

[0.1] Delete machines #1180

[0.1] Delete machines #1180

Conversation

liztio commented Jul 22, 2019 • edited by ncdc Loading

vincepri left a comment

Choose a reason for hiding this comment

liztio commented Jul 23, 2019

ncdc commented Jul 23, 2019

liztio commented Jul 23, 2019

vincepri commented Jul 23, 2019

liztio commented Jul 24, 2019

liztio commented Jul 24, 2019

ncdc Jul 24, 2019

Choose a reason for hiding this comment

ncdc Jul 24, 2019

Choose a reason for hiding this comment

liztio Jul 24, 2019

Choose a reason for hiding this comment

liztio Jul 24, 2019

Choose a reason for hiding this comment

ncdc Jul 24, 2019

Choose a reason for hiding this comment

ncdc commented Jul 24, 2019

vincepri commented Jul 24, 2019

ncdc commented Jul 24, 2019

liztio commented Jul 24, 2019

liztio commented Jul 24, 2019

vincepri left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Jul 24, 2019

liztio commented Jul 22, 2019 •

edited by ncdc

Loading