Bruop error when trying to upgrade nodes #179

pmacieje · 2022-04-04T08:19:24Z

**Image I'm using:**bottlerocket/bottlerocket-update-operator:v0.2.0

Issue or Feature Request: I've implementet bruop on EKS cluster, but there only one (of six) node was updated. Rest of nodes got error like below.

Agent logs:
{"v":0,"name":"agent","msg":"[UPDATE_BOTTLEROCKET_SHADOW - START]","level":30,"hostname":"brupop-agent-4z2dr","pid":1,"time":"2022-04-04T07:50:55.176421185+00:00","target":"apiserver::client::webclient","line":155,"file":"apiserver/src/client/webclient.rs","self":"K8SAPIServerClient { k8s_projected_token_path: "/var/run/secrets/tokens/bottlerocket-agent-service-account-token" }","req":"UpdateBottlerocketShadowRequest { node_selector: BottlerocketShadowSelector { node_name: "ip-172-22-49-202.aws-ec2", node_uid: "1c491162-0d6f-4517-a594-e2ab72f4bc14" }, node_status: BottlerocketShadowStatus { current_version: "1.6.2", target_version: "1.6.2", current_state: Idle } }"}
{"v":0,"name":"agent","msg":"[UPDATE_BOTTLEROCKET_SHADOW - END]","level":30,"hostname":"brupop-agent-4z2dr","pid":1,"time":"2022-04-04T07:51:12.631267091+00:00","target":"apiserver::client::webclient","line":155,"file":"apiserver/src/client/webclient.rs","elapsed_milliseconds":17454,"self":"K8SAPIServerClient { k8s_projected_token_path: "/var/run/secrets/tokens/bottlerocket-agent-service-account-token" }","req":"UpdateBottlerocketShadowRequest { node_selector: BottlerocketShadowSelector { node_name: "ip-172-22-49-202.aws-ec2", node_uid: "1c491162-0d6f-4517-a594-e2ab72f4bc14" }, node_status: BottlerocketShadowStatus { current_version: "1.6.2", target_version: "1.6.2", current_state: Idle } }"}
{"v":0,"name":"agent","msg":"[UPDATE_METADATA_CUSTOM_RESOURCE - EVENT] agent::agentclient","level":50,"hostname":"brupop-agent-4z2dr","pid":1,"time":"2022-04-04T07:51:12.631362459+00:00","target":"agent::agentclient","line":206,"file":"agent/src/agentclient.rs","error":"Unable to update the custom resource associated with this node: 'Unable to update BottlerocketShadow status (ip-172-22-49-202.aws-ec2, 1c491162-0d6f-4517-a594-e2ab72f4bc14): 'API server responded with an error status code 500 Internal Server Error: 'Error patching BottlerocketShadow: 'Unable to update BottlerocketShadow status (ip-172-22-49-202.aws-ec2, 1c491162-0d6f-4517-a594-e2ab72f4bc14): 'ApiError: BottlerocketShadow.brupop.bottlerocket.aws "brs-ip-172-22-49-202.aws-ec2" is invalid: status.crash_count: Required value: Invalid (ErrorResponse { status: "Failure", message: "BottlerocketShadow.brupop.bottlerocket.aws \"brs-ip-172-22-49-202.aws-ec2\" is invalid: status.crash_count: Required value", reason: "Invalid", code: 422 })'''''"}

Monitoring Custom Resources:
$ kubectl get brs --namespace brupop-bottlerocket-aws
NAME STATE VERSION TARGET STATE TARGET VERSION CRASH COUNT
brs-ip-172-22-40-11.aws-ec2 Idle
brs-ip-172-22-43-11.aws-ec2 Idle
brs-ip-172-22-45-41.aws-ec2 Idle
brs-ip-172-22-45-87.aws-ec2 Idle
brs-ip-172-22-49-202.aws-ec2 Idle
brs-ip-172-22-50-13.aws-ec2 Idle

Bruop stack deployed:
kubectl get pods -n brupop-bottlerocket-aws
NAME READY STATUS RESTARTS AGE
brupop-agent-4z2dr 1/1 Running 0 3d21h
brupop-agent-dfztv 1/1 Running 0 3d21h
brupop-agent-hzd7d 1/1 Running 0 3d21h
brupop-agent-lz9xq 1/1 Running 0 3d21h
brupop-agent-vb2w8 1/1 Running 0 3d21h
brupop-agent-zc6rt 1/1 Running 1 3d21h
brupop-apiserver-745c5cffd9-bdts7 1/1 Running 1 3d21h
brupop-apiserver-745c5cffd9-ssvbx 1/1 Running 0 3d21h
brupop-apiserver-745c5cffd9-wht6g 1/1 Running 0 3d21h
brupop-controller-deployment-8545559bc7-pqkrs 1/1 Running 1 3d21h

All nodes are labeled with bottlerocket.aws/updater-interface-version=2.0.0

Could enyone help with this issue ?

cbgbt · 2022-04-04T17:30:12Z

Thanks for opening this issue. I believe this is an issue with the installation instructions: the instructions use the .yaml file from our develop branch, which has had a few changes to it since; however, the images we've deployed are still 0.2.0. We're waiting for a few more features to land before cutting a new release.

To fix this, we need to lock the installation instructions to the latest version.

You can fix this in your cluster by clearing your resources in the brupop namespace, and re-installing those resources from the 0.2.0 tag for the time being:
https://github.com/bottlerocket-os/bottlerocket-update-operator/tree/v0.2.0

I'll have a stab at correcting the instructions.

cbgbt · 2022-04-04T19:44:37Z

#180 has fixed the install instructions to use the versioned artifacts. Would you mind checking to see if this resolves your issue?

To fix this in a better way long-term, we have #126, as well as #161 which would make newer CRDs for the shadow objects backwards compatible. We're intending to use the solution in #161 to fix the current compatibility issue in develop with 0.2.0 before rolling out the next version.

pmacieje · 2022-04-05T08:05:26Z

Looks much better:

$kubectl get brs --namespace brupop-bottlerocket-aws
NAME STATE VERSION TARGET STATE TARGET VERSION
brs-ip-172-22-43-11.aws-ec2 Idle 1.6.2 Idle
brs-ip-172-22-45-41 Idle 1.6.2 Idle
brs-ip-172-22-45-87 Idle 1.6.2 Idle
brs-ip-172-22-49-202 Idle 1.6.2 StagedUpdate 1.7.0
brs-ip-172-22-50-13 Idle 1.7.0 Idle

But there is still some problem with upgrade:

Logs from agent of 172-22-49-202 node:

{"v":0,"name":"agent","msg":"[RUN - EVENT] Detected drift between spec state and current state. Requesting node to take action","level":30,"hostname":"brupop-agent-fxq6d","pid":1,"time":"2022-04-05T07:57:55.244687734+00:00","target":"agent::agentclient","line":322,"file":"agent/src/agentclient.rs","action":"StagedUpdate","brs_name":"Some("brs-ip-172-22-49-202.aws-ec2")"}
{"v":0,"name":"agent","msg":"[RUN - EVENT] Preparing update","level":30,"hostname":"brupop-agent-fxq6d","pid":1,"time":"2022-04-05T07:57:55.244705606+00:00","target":"agent::agentclient","line":337,"file":"agent/src/agentclient.rs"}
{"v":0,"name":"agent","msg":"[RUN - EVENT] API server busy, retrying later ...","level":30,"hostname":"brupop-agent-fxq6d","pid":1,"time":"2022-04-05T07:57:55.255168626+00:00","target":"agent::apiclient","line":140,"file":"agent/src/apiclient.rs"}
{"v":0,"name":"agent","msg":"[RUN - EVENT] agent::agentclient","level":50,"hostname":"brupop-agent-fxq6d","pid":1,"time":"2022-04-05T07:58:05.264579472+00:00","target":"agent::agentclient","line":261,"file":"agent/src/agentclient.rs","error":"Unable to take action 'Prepare': 'Unexpected update state: Ready, expecting state to be Available or Staged. Update action performed out of band?'"}
{"v":0,"name":"agent","msg":"[RUN - END]","level":30,"hostname":"brupop-agent-fxq6d","pid":1,"time":"2022-04-05T07:58:05.264648063+00:00","target":"agent::agentclient","line":260,"file":"agent/src/agentclient.rs","elapsed_milliseconds":10039}

cbgbt · 2022-04-05T18:51:55Z

I misread your message, apologies for closing.

So the error message is:

{"v":0,"name":"agent","msg":"[RUN - EVENT] agent::agentclient","level":50,"hostname":"brupop-agent-fxq6d","pid":1,"time":"2022-04-05T07:58:05.264579472+00:00","target":"agent::agentclient","line":261,"file":"agent/src/agentclient.rs","error":"Unable to take action 'Prepare': 'Unexpected update state: Ready, expecting state to be Available or Staged. Update action performed out of band?'"}

Which is being thrown here.

Brupop attempts to "prepare" an update image, and then writes back to the k8s API that the preparation is ready. If, during preparation, it notices an image is already prepped, then it raises an error -- the rationale being that a user may be attempting to manually install a different update ("out of band"), and Brupop should not intervene.

I'm noticing the IP of the affected host is the same -- so I think what has happened is:

The previously-installed Brupop readied a new image
The previously-installed Brupop attempted to write back to the k8s API that a new image was ready, but this write failed due to the initial issue that you hit with the CRD schema.
After re-installing Brupop, the preparation was attempted again, but the image that was readied last time is now triggering the logic I mentioned above, causing it to refuse to proceed.

If you have admin access to this node, then manually triggering the update this once should "un-stick" Brupop. From a usability perspective, I think it might make sense for Brupop to attempt to perform it's configured update regardless. Admins trying to perform an out-of-band update should probably disable Brupop on the node first. We can look at changing this behavior to prevent this error case.

cbgbt · 2022-04-05T18:54:42Z

To apply the update I'm mentioning on this node, you'd want to follow these steps.

I would suggest using kubectl drain to remove the node from service first, then using kubectl uncordon to bring it back after the update is finished.

pmacieje · 2022-04-08T12:14:19Z

Thanks for suggestion. I've updated manually problematic host. Bow it's another problem ... Bruop try to update next host, state is StagedUpdate, host is cordoned but it's not drained ans stand in this state for last 24h ...

$ kubectl get brs --namespace brupop-bottlerocket-aws
NAME STATE VERSION TARGET STATE TARGET VERSION
brs-ip-172-22-40-11 Idle 1.6.2 Idle
brs-ip-172-22-42-116 Idle 1.5.2 Idle
brs-ip-172-22-45-41 Idle 1.6.2 StagedUpdate 1.7.0
brs-ip-172-22-45-87 Idle 1.6.2 Idle
brs-ip-172-22-47-39 Idle 1.5.2 Idle
brs-ip-172-22-49-202 Idle 1.7.0 Idle
brs-ip-172-22-50-13 Idle 1.7.0 Idle

Agent logs from host 172-22-45-41:

{"v":0,"name":"agent","msg":"[FETCH_CUSTOM_RESOURCE - START]","level":30,"hostname":"brupop-agent-bqxz6","pid":1,"time":"2022-04-07T20:34:33.367421300+00:00","target":"agent::agentclient","line":140,"file":"agent/src/agentclient.rs"}
{"v":0,"name":"agent","msg":"[FETCH_CUSTOM_RESOURCE - END]","level":30,"hostname":"brupop-agent-bqxz6","pid":1,"time":"2022-04-07T20:34:33.367455131+00:00","target":"agent::agentclient","line":140,"file":"agent/src/agentclient.rs","elapsed_milliseconds":0}
{"v":0,"name":"agent","msg":"[CHECK_CUSTOM_RESOURCE_STATUS_EXISTS - END]","level":30,"hostname":"brupop-agent-bqxz6","pid":1,"time":"2022-04-07T20:34:33.367467999+00:00","target":"agent::agentclient","line":108,"file":"agent/src/agentclient.rs","elapsed_milliseconds":0}
{"v":0,"name":"agent","msg":"[FETCH_CUSTOM_RESOURCE - START]","level":30,"hostname":"brupop-agent-bqxz6","pid":1,"time":"2022-04-07T20:34:33.367479359+00:00","target":"agent::agentclient","line":140,"file":"agent/src/agentclient.rs"}
{"v":0,"name":"agent","msg":"[FETCH_CUSTOM_RESOURCE - END]","level":30,"hostname":"brupop-agent-bqxz6","pid":1,"time":"2022-04-07T20:34:33.367514056+00:00","target":"agent::agentclient","line":140,"file":"agent/src/agentclient.rs","elapsed_milliseconds":0}
{"v":0,"name":"agent","msg":"[RUN - EVENT] Detected drift between spec state and current state. Requesting node to take action","level":30,"hostname":"brupop-agent-bqxz6","pid":1,"time":"2022-04-07T20:34:33.367538347+00:00","target":"agent::agentclient","line":322,"file":"agent/src/agentclient.rs","action":"StagedUpdate","brs_name":"Some("brs-ip-172-22-45-41.")"}
{"v":0,"name":"agent","msg":"[RUN - EVENT] Preparing update","level":30,"hostname":"brupop-agent-bqxz6","pid":1,"time":"2022-04-07T20:34:33.367550257+00:00","target":"agent::agentclient","line":337,"file":"agent/src/agentclient.rs"}
{"v":0,"name":"agent","msg":"[RUN - EVENT] API server busy, retrying later ...","level":30,"hostname":"brupop-agent-bqxz6","pid":1,"time":"2022-04-07T20:34:33.385496908+00:00","target":"agent::apiclient","line":140,"file":"agent/src/apiclient.rs"}
{"v":0,"name":"agent","msg":"[RUN - EVENT] API server busy, retrying later ...","level":30,"hostname":"brupop-agent-bqxz6","pid":1,"time":"2022-04-07T20:34:43.411666641+00:00","target":"agent::apiclient","line":140,"file":"agent/src/apiclient.rs"}
{"v":0,"name":"agent","msg":"[RUN - EVENT] API server busy, retrying later ...","level":30,"hostname":"brupop-agent-bqxz6","pid":1,"time":"2022-04-07T20:34:53.430894722+00:00","target":"agent::apiclient","line":140,"file":"agent/src/apiclient.rs"}
{"v":0,"name":"agent","msg":"[CORDON_AND_DRAIN - START]","level":30,"hostname":"brupop-agent-bqxz6","pid":1,"time":"2022-04-07T20:35:03.437700594+00:00","target":"agent::agentclient","line":232,"file":"agent/src/agentclient.rs"}
{"v":0,"name":"agent","msg":"[GET_NODE_SELECTOR - START]","level":30,"hostname":"brupop-agent-bqxz6","pid":1,"time":"2022-04-07T20:35:03.437928758+00:00","target":"agent::agentclient","line":113,"file":"agent/src/agentclient.rs"}
{"v":0,"name":"agent","msg":"[GET_NODE_SELECTOR - END]","level":30,"hostname":"brupop-agent-bqxz6","pid":1,"time":"2022-04-07T20:35:03.438329067+00:00","target":"agent::agentclient","line":113,"file":"agent/src/agentclient.rs","elapsed_milliseconds":0}
{"v":0,"name":"agent","msg":"[CORDON_AND_DRAIN_NODE - START]","level":30,"hostname":"brupop-agent-bqxz6","pid":1,"time":"2022-04-07T20:35:03.438483325+00:00","target":"apiserver::client::webclient","line":209,"file":"apiserver/src/client/webclient.rs","self":"K8SAPIServerClient { k8s_projected_token_path: "/var/run/secrets/tokens/bottlerocket-agent-service-account-token" }","req":"CordonAndDrainBottlerocketShadowRequest { node_selector: BottlerocketShadowSelector { node_name: "ip-172-22-45-41", node_uid: "85bee1f8-badd-48b6-98e5-d147a05050a7" }

API server logs:

cbgbt · 2022-04-08T12:33:03Z

Thanks for the report and logs. Are you using StatefulSet deployments? We recently merged a PR to fix #168, which sounds like the same problem. We're planning to release an updated Brupop with the fix.

pmacieje · 2022-04-11T08:15:50Z

Yeap, I use almost all Kinds of objects on the cluster, StateFullSet as well.

somnusfish · 2022-04-20T21:42:32Z

Hi, the release v0.2.1 includes the fix for StateFullSet. Could you try and let us know if you still meet the same issue?

somnusfish · 2022-05-02T17:07:14Z

Hello pmacieje@, haven't heard from you for a long time. I would assume our new release solved the issue you mentioned here. I am going to close this issue. Please feel free to open a new issue if you have any questions or concerns.

pmacieje · 2022-05-08T13:08:28Z

Hi, there smonusfish, works better now, no more hanging issue, but I saw that there was no drain during update process of my bottlerockets, there was only apply new os version and api reboot. Is that correct ?

somnusfish · 2022-05-08T21:17:52Z

The drain won't show up in STATE or TARGET STATE if you check BotttlerocketShadow using

kubectl get brs

The drain actually happens during the state change Idle-> StagedUpdate. You will be able to see the logs similar to the following in one of the api-server logs that actually handle the drain requests:

kubectl logs brupop-apiserver-7d7fb58566-htclc --follow

...
{"v":0,"name":"apiserver","msg":"[CORDON_NODE - START]","level":30,"hostname":"brupop-apiserver-7d7fb58566-qsrj4","pid":1,"time":"2022-05-08T21:02:20.906750984+00:00","target":"models::node::client","line":265,"file":"models/src/node/client.rs","http.target":"/bottlerocket-node-resource/cordon-and-drain","node_name":"ip-192-168-47-149.us-west-2.compute.internal","http.host":"brupop-apiserver.brupop-bottlerocket-aws.svc.cluster.local","http.user_agent":"","http.route":"/bottlerocket-node-resource/cordon-and-drain","http.flavor":"1.1","otel.kind":"server","request_id":"61cc250d-066c-4d63-969f-24a1d63237b4","selector":"BottlerocketShadowSelector { node_name: \"ip-192-168-47-149.us-west-2.compute.internal\", node_uid: \"8a949c0e-741d-4e6e-83d8-a234dc0e6ad9\" }","http.client_ip":"192.168.62.14:38524","http.method":"POST","http.scheme":"http"}
{"v":0,"name":"apiserver","msg":"[CORDON_NODE - END]","level":30,"hostname":"brupop-apiserver-7d7fb58566-qsrj4","pid":1,"time":"2022-05-08T21:02:20.918336671+00:00","target":"models::node::client","line":265,"file":"models/src/node/client.rs","http.target":"/bottlerocket-node-resource/cordon-and-drain","node_name":"ip-192-168-47-149.us-west-2.compute.internal","http.host":"brupop-apiserver.brupop-bottlerocket-aws.svc.cluster.local","elapsed_milliseconds":11,"http.user_agent":"","http.route":"/bottlerocket-node-resource/cordon-and-drain","http.flavor":"1.1","otel.kind":"server","request_id":"61cc250d-066c-4d63-969f-24a1d63237b4","selector":"BottlerocketShadowSelector { node_name: \"ip-192-168-47-149.us-west-2.compute.internal\", node_uid: \"8a949c0e-741d-4e6e-83d8-a234dc0e6ad9\" }","http.client_ip":"192.168.62.14:38524","http.method":"POST","http.scheme":"http"}
{"v":0,"name":"apiserver","msg":"[DRAIN_NODE - START]","level":30,"hostname":"brupop-apiserver-7d7fb58566-qsrj4","pid":1,"time":"2022-05-08T21:02:20.919590517+00:00","target":"models::node::client","line":280,"file":"models/src/node/client.rs","http.target":"/bottlerocket-node-resource/cordon-and-drain","node_name":"ip-192-168-47-149.us-west-2.compute.internal","http.host":"brupop-apiserver.brupop-bottlerocket-aws.svc.cluster.local","http.user_agent":"","http.route":"/bottlerocket-node-resource/cordon-and-drain","http.flavor":"1.1","otel.kind":"server","request_id":"61cc250d-066c-4d63-969f-24a1d63237b4","selector":"BottlerocketShadowSelector { node_name: \"ip-192-168-47-149.us-west-2.compute.internal\", node_uid: \"8a949c0e-741d-4e6e-83d8-a234dc0e6ad9\" }","http.client_ip":"192.168.62.14:38524","http.method":"POST","http.scheme":"http"}
{"v":0,"name":"apiserver","msg":"[DRAIN_NODE - START]","level":30,"hostname":"brupop-apiserver-7d7fb58566-qsrj4","pid":1,"time":"2022-05-08T21:02:20.919815245+00:00","target":"models::node::drain","line":80,"file":"models/src/node/drain.rs","http.target":"/bottlerocket-node-resource/cordon-and-drain","node_name":"ip-192-168-47-149.us-west-2.compute.internal","http.host":"brupop-apiserver.brupop-bottlerocket-aws.svc.cluster.local","http.user_agent":"","http.route":"/bottlerocket-node-resource/cordon-and-drain","http.flavor":"1.1","otel.kind":"server","request_id":"61cc250d-066c-4d63-969f-24a1d63237b4","selector":"BottlerocketShadowSelector { node_name: \"ip-192-168-47-149.us-west-2.compute.internal\", node_uid: \"8a949c0e-741d-4e6e-83d8-a234dc0e6ad9\" }","http.client_ip":"192.168.62.14:38524","http.method":"POST","http.scheme":"http"}
{"v":0,"name":"apiserver","msg":"[FIND_TARGET_PODS - START]","level":30,"hostname":"brupop-apiserver-7d7fb58566-qsrj4","pid":1,"time":"2022-05-08T21:02:20.919883222+00:00","target":"models::node::drain","line":112,"file":"models/src/node/drain.rs","http.target":"/bottlerocket-node-resource/cordon-and-drain","node_name":"ip-192-168-47-149.us-west-2.compute.internal","http.host":"brupop-apiserver.brupop-bottlerocket-aws.svc.cluster.local","http.user_agent":"","http.route":"/bottlerocket-node-resource/cordon-and-drain","http.flavor":"1.1","otel.kind":"server","request_id":"61cc250d-066c-4d63-969f-24a1d63237b4","selector":"BottlerocketShadowSelector { node_name: \"ip-192-168-47-149.us-west-2.compute.internal\", node_uid: \"8a949c0e-741d-4e6e-83d8-a234dc0e6ad9\" }","http.client_ip":"192.168.62.14:38524","http.method":"POST","http.scheme":"http"}
{"v":0,"name":"apiserver","msg":"[FIND_TARGET_PODS - END]","level":30,"hostname":"brupop-apiserver-7d7fb58566-qsrj4","pid":1,"time":"2022-05-08T21:02:20.929653441+00:00","target":"models::node::drain","line":112,"file":"models/src/node/drain.rs","http.target":"/bottlerocket-node-resource/cordon-and-drain","node_name":"ip-192-168-47-149.us-west-2.compute.internal","http.host":"brupop-apiserver.brupop-bottlerocket-aws.svc.cluster.local","elapsed_milliseconds":9,"http.user_agent":"","http.route":"/bottlerocket-node-resource/cordon-and-drain","http.flavor":"1.1","otel.kind":"server","request_id":"61cc250d-066c-4d63-969f-24a1d63237b4","selector":"BottlerocketShadowSelector { node_name: \"ip-192-168-47-149.us-west-2.compute.internal\", node_uid: \"8a949c0e-741d-4e6e-83d8-a234dc0e6ad9\" }","http.client_ip":"192.168.62.14:38524","http.method":"POST","http.scheme":"http"}
{"v":0,"name":"apiserver","msg":"[DRAIN_NODE - EVENT] Not draining Pod 'brupop-agent-gq25c': Pod is member of a DaemonSet","level":30,"hostname":"brupop-apiserver-7d7fb58566-qsrj4","pid":1,"time":"2022-05-08T21:02:20.929879959+00:00","target":"models::node::drain","line":151,"file":"models/src/node/drain.rs","http.target":"/bottlerocket-node-resource/cordon-and-drain","node_name":"ip-192-168-47-149.us-west-2.compute.internal","http.host":"brupop-apiserver.brupop-bottlerocket-aws.svc.cluster.local","http.user_agent":"","http.route":"/bottlerocket-node-resource/cordon-and-drain","http.flavor":"1.1","otel.kind":"server","request_id":"61cc250d-066c-4d63-969f-24a1d63237b4","selector":"BottlerocketShadowSelector { node_name: \"ip-192-168-47-149.us-west-2.compute.internal\", node_uid: \"8a949c0e-741d-4e6e-83d8-a234dc0e6ad9\" }","http.client_ip":"192.168.62.14:38524","http.method":"POST","http.scheme":"http"}
{"v":0,"name":"apiserver","msg":"[EVICT_POD - START]","level":30,"hostname":"brupop-apiserver-7d7fb58566-qsrj4","pid":1,"time":"2022-05-08T21:02:20.930435705+00:00","target":"models::node::drain","line":176,"file":"models/src/node/drain.rs","http.target":"/bottlerocket-node-resource/cordon-and-drain","node_name":"ip-192-168-47-149.us-west-2.compute.internal","http.host":"brupop-apiserver.brupop-bottlerocket-aws.svc.cluster.local","http.user_agent":"","http.route":"/bottlerocket-node-resource/cordon-and-drain","http.flavor":"1.1","otel.kind":"server","request_id":"61cc250d-066c-4d63-969f-24a1d63237b4","selector":"BottlerocketShadowSelector { node_name: \"ip-192-168-47-149.us-west-2.compute.internal\", node_uid: \"8a949c0e-741d-4e6e-83d8-a234dc0e6ad9\" }","http.client_ip":"192.168.62.14:38524","http.method":"POST","http.scheme":"http"}
{"v":0,"name":"apiserver","msg":"[EVICT_POD - EVENT] Attempting to evict pod brupop-apiserver-7d7fb58566-qsrj4","level":30,"hostname":"brupop-apiserver-7d7fb58566-qsrj4","pid":1,"time":"2022-05-08T21:02:20.930522405+00:00","target":"models::node::drain","line":193,"file":"models/src/node/drain.rs","http.target":"/bottlerocket-node-resource/cordon-and-drain","node_name":"ip-192-168-47-149.us-west-2.compute.internal","http.host":"brupop-apiserver.brupop-bottlerocket-aws.svc.cluster.local","http.user_agent":"","http.route":"/bottlerocket-node-resource/cordon-and-drain","http.flavor":"1.1","otel.kind":"server","request_id":"61cc250d-066c-4d63-969f-24a1d63237b4","selector":"BottlerocketShadowSelector { node_name: \"ip-192-168-47-149.us-west-2.compute.internal\", node_uid: \"8a949c0e-741d-4e6e-83d8-a234dc0e6ad9\" }","http.client_ip":"192.168.62.14:38524","http.method":"POST","http.scheme":"http"}
{"v":0,"name":"apiserver","msg":"[EVICT_POD - START]","level":30,"hostname":"brupop-apiserver-7d7fb58566-qsrj4","pid":1,"time":"2022-05-08T21:02:20.931230026+00:00","target":"models::node::drain","line":176,"file":"models/src/node/drain.rs","http.target":"/bottlerocket-node-resource/cordon-and-drain","node_name":"ip-192-168-47-149.us-west-2.compute.internal","http.host":"brupop-apiserver.brupop-bottlerocket-aws.svc.cluster.local","http.user_agent":"","http.route":"/bottlerocket-node-resource/cordon-and-drain","http.flavor":"1.1","otel.kind":"server","request_id":"61cc250d-066c-4d63-969f-24a1d63237b4","selector":"BottlerocketShadowSelector { node_name: \"ip-192-168-47-149.us-west-2.compute.internal\", node_uid: \"8a949c0e-741d-4e6e-83d8-a234dc0e6ad9\" }","http.client_ip":"192.168.62.14:38524","http.method":"POST","http.scheme":"http"}
{"v":0,"name":"apiserver","msg":"[EVICT_POD - EVENT] Attempting to evict pod cert-manager-b4d6fd99b-gb6h2","level":30,"hostname":"brupop-apiserver-7d7fb58566-qsrj4","pid":1,"time":"2022-05-08T21:02:20.931288296+00:00","target":"models::node::drain","line":193,"file":"models/src/node/drain.rs","http.target":"/bottlerocket-node-resource/cordon-and-drain","node_name":"ip-192-168-47-149.us-west-2.compute.internal","http.host":"brupop-apiserver.brupop-bottlerocket-aws.svc.cluster.local","http.user_agent":"","http.route":"/bottlerocket-node-resource/cordon-and-drain","http.flavor":"1.1","otel.kind":"server","request_id":"61cc250d-066c-4d63-969f-24a1d63237b4","selector":"BottlerocketShadowSelector { node_name: \"ip-192-168-47-149.us-west-2.compute.internal\", node_uid: \"8a949c0e-741d-4e6e-83d8-a234dc0e6ad9\" }","http.client_ip":"192.168.62.14:38524","http.method":"POST","http.scheme":"http"}
{"v":0,"name":"apiserver","msg":"[EVICT_POD - START]","level":30,"hostname":"brupop-apiserver-7d7fb58566-qsrj4","pid":1,"time":"2022-05-08T21:02:20.932070728+00:00","target":"models::node::drain","line":176,"file":"models/src/node/drain.rs","http.target":"/bottlerocket-node-resource/cordon-and-drain","node_name":"ip-192-168-47-149.us-west-2.compute.internal","http.host":"brupop-apiserver.brupop-bottlerocket-aws.svc.cluster.local","http.user_agent":"","http.route":"/bottlerocket-node-resource/cordon-and-drain","http.flavor":"1.1","otel.kind":"server","request_id":"61cc250d-066c-4d63-969f-24a1d63237b4","selector":"BottlerocketShadowSelector { node_name: \"ip-192-168-47-149.us-west-2.compute.internal\", node_uid: \"8a949c0e-741d-4e6e-83d8-a234dc0e6ad9\" }","http.client_ip":"192.168.62.14:38524","http.method":"POST","http.scheme":"http"}
{"v":0,"name":"apiserver","msg":"[EVICT_POD - EVENT] Attempting to evict pod cert-manager-cainjector-74bfccdfdf-xzntw","level":30,"hostname":"brupop-apiserver-7d7fb58566-qsrj4","pid":1,"time":"2022-05-08T21:02:20.932111017+00:00","target":"models::node::drain","line":193,"file":"models/src/node/drain.rs","http.target":"/bottlerocket-node-resource/cordon-and-drain","node_name":"ip-192-168-47-149.us-west-2.compute.internal","http.host":"brupop-apiserver.brupop-bottlerocket-aws.svc.cluster.local","http.user_agent":"","http.route":"/bottlerocket-node-resource/cordon-and-drain","http.flavor":"1.1","otel.kind":"server","request_id":"61cc250d-066c-4d63-969f-24a1d63237b4","selector":"BottlerocketShadowSelector { node_name: \"ip-192-168-47-149.us-west-2.compute.internal\", node_uid: \"8a949c0e-741d-4e6e-83d8-a234dc0e6ad9\" }","http.client_ip":"192.168.62.14:38524","http.method":"POST","http.scheme":"http"}
...

Since api-server can be restarted during the update, one possible missing logs could be check the log after apiserver restarting.

The other possible could be since Kubernetes API doesn't provide an implementation of drain, Brupop uses Pod Deletion and Eviction to remove all Pods from a give node, kubectl by default will not evict nodes under some criteria, so by default, we ignore:

DaemonSet Pods - The DaemonSet controller will not respect node cordons, so we don't battle it.
Mirror Pods - These are static and cannot be controlled.

If you are talking about pods belong to the above set, it's not drained as expected.

cbgbt mentioned this issue Apr 4, 2022

Use versioned links for installation instructions #180

Merged

cbgbt added type/bug type/documentation labels Apr 4, 2022

cbgbt closed this as completed Apr 5, 2022

cbgbt reopened this Apr 5, 2022

cbgbt self-assigned this Apr 5, 2022

cbgbt added this to the brupop 0.2.1 milestone Apr 5, 2022

somnusfish closed this as completed May 2, 2022

somnusfish reopened this May 8, 2022

pmacieje closed this as completed May 30, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bruop error when trying to upgrade nodes #179

Bruop error when trying to upgrade nodes #179

pmacieje commented Apr 4, 2022

cbgbt commented Apr 4, 2022

cbgbt commented Apr 4, 2022

pmacieje commented Apr 5, 2022

cbgbt commented Apr 5, 2022

cbgbt commented Apr 5, 2022

pmacieje commented Apr 8, 2022 •

edited

Loading

cbgbt commented Apr 8, 2022

pmacieje commented Apr 11, 2022

somnusfish commented Apr 20, 2022

somnusfish commented May 2, 2022

pmacieje commented May 8, 2022

somnusfish commented May 8, 2022

Bruop error when trying to upgrade nodes #179

Bruop error when trying to upgrade nodes #179

Comments

pmacieje commented Apr 4, 2022

cbgbt commented Apr 4, 2022

cbgbt commented Apr 4, 2022

pmacieje commented Apr 5, 2022

cbgbt commented Apr 5, 2022

cbgbt commented Apr 5, 2022

pmacieje commented Apr 8, 2022 • edited Loading

cbgbt commented Apr 8, 2022

pmacieje commented Apr 11, 2022

somnusfish commented Apr 20, 2022

somnusfish commented May 2, 2022

pmacieje commented May 8, 2022

somnusfish commented May 8, 2022

pmacieje commented Apr 8, 2022 •

edited

Loading