Soft-fail on node password verification if the secret cannot be created

[brandond]

Proposed Changes

Soft-fail on node password verification if the secret cannot be created

Allows nodes to join the cluster during a webhook outage. This also enhances auditability by creating Kubernetes events for the deferred verification.

This approach was discussed with @macedogm at rancher/rke2#4266 (comment)

Types of Changes

bugfix

Verification

See linked issue

Testing

Linked Issues

• Failing secret webhook causes unexpected behavior #7654
• Failing secret webhook causes unexpected behavior rancher/rke2#4266

User-Facing Change

K3s now allows nodes to join the cluster even if the node password secret cannot be created at the time the node joins. The secret create will be retried in the background.
    This resolves a potential deadlock created by fail-closed validating webhooks that block secret creation, where the webhook is unavailable until new nodes join the cluster to run the webhook pod.

Further Comments

[codecov]

Codecov Report

Patch coverage: 48.57% and project coverage change: +2.47 🎉

Comparison is base (b64a226) 45.00% compared to head (cdff3d9) 47.47%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #7655      +/-   ##
==========================================
+ Coverage   45.00%   47.47%   +2.47%     
==========================================
  Files         138      140       +2     
  Lines       14262    14343      +81     
==========================================
+ Hits         6419     6810     +391     
+ Misses       6757     6446     -311     
- Partials     1086     1087       +1     

Flag	Coverage Δ
inttests	44.84% <47.14%> (-0.17%)	⬇️
unittests	19.86% <60.71%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
pkg/daemons/config/types.go	84.44% <ø> (+14.44%)	⬆️
pkg/server/router.go	49.00% <37.50%> (+0.28%)	⬆️
pkg/nodepassword/nodepassword.go	73.68% <60.71%> (+27.96%)	⬆️
pkg/server/context.go	65.71% <100.00%> (+1.00%)	⬆️
pkg/server/server.go	56.98% <100.00%> (+0.09%)	⬆️

... and 24 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.