Restarting rke2-server.service with multiple controlplane nodes doesn't always cause updates from rke2 secrets-encrypt rotate to take effect

[jakefhyde]

Environmental Info:
RKE2 Version: v1.23.6+rke2r2

Cluster Configuration:

3 servers (rancher provisioned, but reproducible standalone)

Describe the bug:

In a 3 node all roles cluster, restarting the rke2-server.service following the rke2 secrets-encrypt rotate command does not cause the other controlplane nodes to advance to the rotate stage.

Steps To Reproduce:

• Installed RKE2: Rancher provisioned, but also reproducible standalone
• Create a script that does the following:
  • Run rke2 secrets-encrypt prepare
  • Scrape rke2 secrets-encrypt status to verify that the current stage is prepare
  • Restart the node using systemctl restart rke2-server.service
  • Restart the other controlplane nodes using systemctl restart rke2-server.service
  • Run rke2 secrets-encrypt rotate
  • Scrape rke2 secrets-encrypt status to verify that the current stage is rotate
  • Restart the node using systemctl restart rke2-server.service
  • Restart the other controlplane nodes using systemctl restart rke2-server.service

Expected behavior:

Other controlplane nodes successfully restart following rke2 secrets-encrypt rotate

Actual behavior:

rke2-server.service on subsequent controlplane nodes successfully actives after running systemctl restart rke2-server, but does not have the correct status.

Additional context / logs:

The same workaround described in this issue was attempted here as well to some moderate success however is also not an option for the same reasons listed there.

[jakefhyde]

Closing this issue as rancher has migrated away from using the killall script.