fix: annotate nodes for reboot before aborting due to blocked

[jackfrancis]

This PR moves the checking of "reboot blockers" (e.g., matching podSelectors running on the node to-be-rebooted) further down in the conditional reboot flow so that those checks happen after we (conditionally) annotate the nodes.

This enables the following when a reboot is detected (by default the presence of the /var/run/reboot-required file):

1. If --annotate-nodes is true, we annotate the nodes with the following annotations:

• "weave.works/kured-reboot-in-progress" (indicates that this node will be rebooted at some point in the near future)
• "weave.works/kured-most-recent-reboot-needed" (marks a timestamp of when this need for reboot was detected by kured)

2. After we annotate nodes, add a "prefer no schedule taint" to communicate a preference (though not a strict requirement) to the scheduler not to consider this node for future pod scheduling
3. We check to see if any prometheus or pod selector blockers prevent an immediate reboot, if so we short-circuit and requeue for a later reboot attempt when blockers and give the green light
4. If we're not blocked, we cordon+drain node
5. We reboot

I'm simplifying a bit above, there is a bit more complexity, but the key point is that we are now marking "this node is definitely going to be reboote" prior to blockers. This allows other, complementary tooling to know about the state of the node ("gonna be rebooted soon") and do things like extra-manual stateful application migration off of that node, add additional taints to better prevent future scheduling, etc.

Fixes #702

[jackfrancis]

cc @timo-42

[jackfrancis]

/assign @ckotzbauer

[ckotzbauer]

LGTM

[timo-42]

@jackfrancis @ckotzbauer Please merge this PR.

[ckotzbauer]

We're just discussing if this should land now right before the upcoming 1.13.0 release.