Webhook conversion: Support v2 to v1

[jpmcb]

Issue number:

Fixes #247

Description of changes:

This patch sets a path for v2 to v1 conversions via the established conversion webhook:

• Implements From blocks for type conversion from BottlerocketShadow v2 to v1
  • Tests for these conversions
• Creates path in webhook for conversion from v2 to v1
  • Establishes the idea of using a pinwheel converter for future implementations
  • Tests for this webhook conversion

The idea that we need to support a downgrade seems abit counter intuitive, but since we are shipping both the v1 and v2 CRD, we also need to support conversion between both: users may query either crd version and the kube-api server needs to know how to resolve the difference in what is in it's "Stored Version" (which by default will be v2). Further, since there the kube-api server has a watcher on both the v1 and v2 resources, without this patch, it can't convert v1 queries into v2 (and dumps alot of logs in the kube-api server and the bottlerocket apiserver)

References:

• https://book.kubebuilder.io/multiversion-tutorial/conversion-concepts.html
• https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definition-versioning/#webhook-conversion

Testing done:

Unit tests pass.

Tested on a new cluster with a new image and logs are much more quite. Doesn't affect brs coming up or upgrades.

Ran through integration tests on ipv4 cluster and ipv6 cluster using custom build image 👍🏼

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

[jpmcb]

The logs from the brupop api-server are also interseting:

Details

❯ k logs -n brupop-bottlerocket-aws brupop-apiserver-568bdf6f46-2lhsp
2022-10-21T17:44:01.358864Z INFO models::node::drain: Not draining Pod 'brupop-agent-7j6ml': Pod is member of a DaemonSet
at models/src/node/drain.rs:150
in models::node::drain::drain_node with node_name: "ip-192-168-153-70.us-west-2.compute.internal"
in models::node::client::drain_node with selector: BottlerocketShadowSelector { node_name: "ip-192-168-153-70.us-west-2.compute.internal", node_uid: "0ef71b7e-2319-4363-94a5-fe6c9986ded4" }
in apiserver::telemetry::HTTP request with http.method: POST, http.route: /bottlerocket-node-resource/cordon-and-drain, http.flavor: 1.1, http.scheme: https, http.host: brupop-apiserver.brupop-bottlerocket-aws.svc.cluster.local, http.client_ip: [2600:1f13:5ba:2304:c6a6::4]:40548, http.user_agent: , http.target: /bottlerocket-node-resource/cordon-and-drain, otel.kind: "server", request_id: 03362a19-79a2-436a-9c57-d1dd8d858704, node_name: "ip-192-168-153-70.us-west-2.compute.internal"

2022-10-21T17:44:01.359057Z INFO models::node::drain: Attempting to evict pod brupop-apiserver-568bdf6f46-9dgpl
at models/src/node/drain.rs:192
in models::node::drain::evict_pod
in models::node::drain::drain_node with node_name: "ip-192-168-153-70.us-west-2.compute.internal"
in models::node::client::drain_node with selector: BottlerocketShadowSelector { node_name: "ip-192-168-153-70.us-west-2.compute.internal", node_uid: "0ef71b7e-2319-4363-94a5-fe6c9986ded4" }
in apiserver::telemetry::HTTP request with http.method: POST, http.route: /bottlerocket-node-resource/cordon-and-drain, http.flavor: 1.1, http.scheme: https, http.host: brupop-apiserver.brupop-bottlerocket-aws.svc.cluster.local, http.client_ip: [2600:1f13:5ba:2304:c6a6::4]:40548, http.user_agent: , http.target: /bottlerocket-node-resource/cordon-and-drain, otel.kind: "server", request_id: 03362a19-79a2-436a-9c57-d1dd8d858704, node_name: "ip-192-168-153-70.us-west-2.compute.internal"

2022-10-21T17:44:01.359196Z INFO models::node::drain: Not draining Pod 'aws-node-jgzvw': Pod is member of a DaemonSet
at models/src/node/drain.rs:150
in models::node::drain::drain_node with node_name: "ip-192-168-153-70.us-west-2.compute.internal"
in models::node::client::drain_node with selector: BottlerocketShadowSelector { node_name: "ip-192-168-153-70.us-west-2.compute.internal", node_uid: "0ef71b7e-2319-4363-94a5-fe6c9986ded4" }
in apiserver::telemetry::HTTP request with http.method: POST, http.route: /bottlerocket-node-resource/cordon-and-drain, http.flavor: 1.1, http.scheme: https, http.host: brupop-apiserver.brupop-bottlerocket-aws.svc.cluster.local, http.client_ip: [2600:1f13:5ba:2304:c6a6::4]:40548, http.user_agent: , http.target: /bottlerocket-node-resource/cordon-and-drain, otel.kind: "server", request_id: 03362a19-79a2-436a-9c57-d1dd8d858704, node_name: "ip-192-168-153-70.us-west-2.compute.internal"

2022-10-21T17:44:01.359291Z INFO models::node::drain: Not draining Pod 'kube-proxy-n9mf8': Pod is member of a DaemonSet
at models/src/node/drain.rs:150
in models::node::drain::drain_node with node_name: "ip-192-168-153-70.us-west-2.compute.internal"
in models::node::client::drain_node with selector: BottlerocketShadowSelector { node_name: "ip-192-168-153-70.us-west-2.compute.internal", node_uid: "0ef71b7e-2319-4363-94a5-fe6c9986ded4" }
in apiserver::telemetry::HTTP request with http.method: POST, http.route: /bottlerocket-node-resource/cordon-and-drain, http.flavor: 1.1, http.scheme: https, http.host: brupop-apiserver.brupop-bottlerocket-aws.svc.cluster.local, http.client_ip: [2600:1f13:5ba:2304:c6a6::4]:40548, http.user_agent: , http.target: /bottlerocket-node-resource/cordon-and-drain, otel.kind: "server", request_id: 03362a19-79a2-436a-9c57-d1dd8d858704, node_name: "ip-192-168-153-70.us-west-2.compute.internal"

2022-10-21T17:44:01.367934Z INFO models::node::drain: Successfully evicted Pod 'brupop-apiserver-568bdf6f46-9dgpl'
at models/src/node/drain.rs:197
in models::node::drain::evict_pod
in models::node::drain::drain_node with node_name: "ip-192-168-153-70.us-west-2.compute.internal"
in models::node::client::drain_node with selector: BottlerocketShadowSelector { node_name: "ip-192-168-153-70.us-west-2.compute.internal", node_uid: "0ef71b7e-2319-4363-94a5-fe6c9986ded4" }
in apiserver::telemetry::HTTP request with http.method: POST, http.route: /bottlerocket-node-resource/cordon-and-drain, http.flavor: 1.1, http.scheme: https, http.host: brupop-apiserver.brupop-bottlerocket-aws.svc.cluster.local, http.client_ip: [2600:1f13:5ba:2304:c6a6::4]:40548, http.user_agent: , http.target: /bottlerocket-node-resource/cordon-and-drain, otel.kind: "server", request_id: 03362a19-79a2-436a-9c57-d1dd8d858704, node_name: "ip-192-168-153-70.us-west-2.compute.internal"

2022-10-21T17:44:06.389463Z INFO models::node::drain: Pod brupop-apiserver-568bdf6f46-9dgpl deleted.
at models/src/node/drain.rs:287
in models::node::drain::wait_for_deletion
in models::node::drain::drain_node with node_name: "ip-192-168-153-70.us-west-2.compute.internal"
in models::node::client::drain_node with selector: BottlerocketShadowSelector { node_name: "ip-192-168-153-70.us-west-2.compute.internal", node_uid: "0ef71b7e-2319-4363-94a5-fe6c9986ded4" }
in apiserver::telemetry::HTTP request with http.method: POST, http.route: /bottlerocket-node-resource/cordon-and-drain, http.flavor: 1.1, http.scheme: https, http.host: brupop-apiserver.brupop-bottlerocket-aws.svc.cluster.local, http.client_ip: [2600:1f13:5ba:2304:c6a6::4]:40548, http.user_agent: , http.target: /bottlerocket-node-resource/cordon-and-drain, otel.kind: "server", request_id: 03362a19-79a2-436a-9c57-d1dd8d858704, node_name: "ip-192-168-153-70.us-west-2.compute.internal"

2022-10-21T17:48:04.071093Z INFO models::node::drain: Not draining Pod 'brupop-agent-hwv9g': Pod is member of a DaemonSet
at models/src/node/drain.rs:150
in models::node::drain::drain_node with node_name: "ip-192-168-134-119.us-west-2.compute.internal"
in models::node::client::drain_node with selector: BottlerocketShadowSelector { node_name: "ip-192-168-134-119.us-west-2.compute.internal", node_uid: "23bd80ae-aeff-42d3-a9fd-6bce8d518149" }
in apiserver::telemetry::HTTP request with http.method: POST, http.route: /bottlerocket-node-resource/cordon-and-drain, http.flavor: 1.1, http.scheme: https, http.host: brupop-apiserver.brupop-bottlerocket-aws.svc.cluster.local, http.client_ip: [2600:1f13:5ba:2304:3a50::4]:41450, http.user_agent: , http.target: /bottlerocket-node-resource/cordon-and-drain, otel.kind: "server", request_id: c7256d05-c468-46a5-9d64-05079b2a3c8c, node_name: "ip-192-168-134-119.us-west-2.compute.internal"

2022-10-21T17:48:04.071200Z INFO models::node::drain: Attempting to evict pod brupop-apiserver-568bdf6f46-fs8mc
at models/src/node/drain.rs:192
in models::node::drain::evict_pod
in models::node::drain::drain_node with node_name: "ip-192-168-134-119.us-west-2.compute.internal"
in models::node::client::drain_node with selector: BottlerocketShadowSelector { node_name: "ip-192-168-134-119.us-west-2.compute.internal", node_uid: "23bd80ae-aeff-42d3-a9fd-6bce8d518149" }
in apiserver::telemetry::HTTP request with http.method: POST, http.route: /bottlerocket-node-resource/cordon-and-drain, http.flavor: 1.1, http.scheme: https, http.host: brupop-apiserver.brupop-bottlerocket-aws.svc.cluster.local, http.client_ip: [2600:1f13:5ba:2304:3a50::4]:41450, http.user_agent: , http.target: /bottlerocket-node-resource/cordon-and-drain, otel.kind: "server", request_id: c7256d05-c468-46a5-9d64-05079b2a3c8c, node_name: "ip-192-168-134-119.us-west-2.compute.internal"

2022-10-21T17:48:04.071253Z INFO models::node::drain: Attempting to evict pod brupop-apiserver-568bdf6f46-zxtcf
at models/src/node/drain.rs:192
in models::node::drain::evict_pod
in models::node::drain::drain_node with node_name: "ip-192-168-134-119.us-west-2.compute.internal"
in models::node::client::drain_node with selector: BottlerocketShadowSelector { node_name: "ip-192-168-134-119.us-west-2.compute.internal", node_uid: "23bd80ae-aeff-42d3-a9fd-6bce8d518149" }
in apiserver::telemetry::HTTP request with http.method: POST, http.route: /bottlerocket-node-resource/cordon-and-drain, http.flavor: 1.1, http.scheme: https, http.host: brupop-apiserver.brupop-bottlerocket-aws.svc.cluster.local, http.client_ip: [2600:1f13:5ba:2304:3a50::4]:41450, http.user_agent: , http.target: /bottlerocket-node-resource/cordon-and-drain, otel.kind: "server", request_id: c7256d05-c468-46a5-9d64-05079b2a3c8c, node_name: "ip-192-168-134-119.us-west-2.compute.internal"

2022-10-21T17:48:04.071424Z INFO models::node::drain: Attempting to evict pod brupop-controller-deployment-b8b577957-5ln5q
at models/src/node/drain.rs:192
in models::node::drain::evict_pod
in models::node::drain::drain_node with node_name: "ip-192-168-134-119.us-west-2.compute.internal"
in models::node::client::drain_node with selector: BottlerocketShadowSelector { node_name: "ip-192-168-134-119.us-west-2.compute.internal", node_uid: "23bd80ae-aeff-42d3-a9fd-6bce8d518149" }
in apiserver::telemetry::HTTP request with http.method: POST, http.route: /bottlerocket-node-resource/cordon-and-drain, http.flavor: 1.1, http.scheme: https, http.host: brupop-apiserver.brupop-bottlerocket-aws.svc.cluster.local, http.client_ip: [2600:1f13:5ba:2304:3a50::4]:41450, http.user_agent: , http.target: /bottlerocket-node-resource/cordon-and-drain, otel.kind: "server", request_id: c7256d05-c468-46a5-9d64-05079b2a3c8c, node_name: "ip-192-168-134-119.us-west-2.compute.internal"

2022-10-21T17:48:04.071559Z INFO models::node::drain: Attempting to evict pod nginx-test-85dd6f4cc-cxqxh
at models/src/node/drain.rs:192
in models::node::drain::evict_pod
in models::node::drain::drain_node with node_name: "ip-192-168-134-119.us-west-2.compute.internal"
in models::node::client::drain_node with selector: BottlerocketShadowSelector { node_name: "ip-192-168-134-119.us-west-2.compute.internal", node_uid: "23bd80ae-aeff-42d3-a9fd-6bce8d518149" }
in apiserver::telemetry::HTTP request with http.method: POST, http.route: /bottlerocket-node-resource/cordon-and-drain, http.flavor: 1.1, http.scheme: https, http.host: brupop-apiserver.brupop-bottlerocket-aws.svc.cluster.local, http.client_ip: [2600:1f13:5ba:2304:3a50::4]:41450, http.user_agent: , http.target: /bottlerocket-node-resource/cordon-and-drain, otel.kind: "server", request_id: c7256d05-c468-46a5-9d64-05079b2a3c8c, node_name: "ip-192-168-134-119.us-west-2.compute.internal"

2022-10-21T17:48:04.071900Z INFO models::node::drain: Attempting to evict pod nginx-test-85dd6f4cc-g6tsx
at models/src/node/drain.rs:192
in models::node::drain::evict_pod
in models::node::drain::drain_node with node_name: "ip-192-168-134-119.us-west-2.compute.internal"
in models::node::client::drain_node with selector: BottlerocketShadowSelector { node_name: "ip-192-168-134-119.us-west-2.compute.internal", node_uid: "23bd80ae-aeff-42d3-a9fd-6bce8d518149" }
in apiserver::telemetry::HTTP request with http.method: POST, http.route: /bottlerocket-node-resource/cordon-and-drain, http.flavor: 1.1, http.scheme: https, http.host: brupop-apiserver.brupop-bottlerocket-aws.svc.cluster.local, http.client_ip: [2600:1f13:5ba:2304:3a50::4]:41450, http.user_agent: , http.target: /bottlerocket-node-resource/cordon-and-drain, otel.kind: "server", request_id: c7256d05-c468-46a5-9d64-05079b2a3c8c, node_name: "ip-192-168-134-119.us-west-2.compute.internal"

2022-10-21T17:48:04.091186Z INFO models::node::drain: Successfully evicted Pod 'brupop-apiserver-568bdf6f46-fs8mc'
at models/src/node/drain.rs:197
in models::node::drain::evict_pod
in models::node::drain::drain_node with node_name: "ip-192-168-134-119.us-west-2.compute.internal"
in models::node::client::drain_node with selector: BottlerocketShadowSelector { node_name: "ip-192-168-134-119.us-west-2.compute.internal", node_uid: "23bd80ae-aeff-42d3-a9fd-6bce8d518149" }
in apiserver::telemetry::HTTP request with http.method: POST, http.route: /bottlerocket-node-resource/cordon-and-drain, http.flavor: 1.1, http.scheme: https, http.host: brupop-apiserver.brupop-bottlerocket-aws.svc.cluster.local, http.client_ip: [2600:1f13:5ba:2304:3a50::4]:41450, http.user_agent: , http.target: /bottlerocket-node-resource/cordon-and-drain, otel.kind: "server", request_id: c7256d05-c468-46a5-9d64-05079b2a3c8c, node_name: "ip-192-168-134-119.us-west-2.compute.internal"

2022-10-21T17:48:04.109350Z INFO models::node::drain: Successfully evicted Pod 'nginx-test-85dd6f4cc-cxqxh'
at models/src/node/drain.rs:197
in models::node::drain::evict_pod
in models::node::drain::drain_node with node_name: "ip-192-168-134-119.us-west-2.compute.internal"
in models::node::client::drain_node with selector: BottlerocketShadowSelector { node_name: "ip-192-168-134-119.us-west-2.compute.internal", node_uid: "23bd80ae-aeff-42d3-a9fd-6bce8d518149" }
in apiserver::telemetry::HTTP request with http.method: POST, http.route: /bottlerocket-node-resource/cordon-and-drain, http.flavor: 1.1, http.scheme: https, http.host: brupop-apiserver.brupop-bottlerocket-aws.svc.cluster.local, http.client_ip: [2600:1f13:5ba:2304:3a50::4]:41450, http.user_agent: , http.target: /bottlerocket-node-resource/cordon-and-drain, otel.kind: "server", request_id: c7256d05-c468-46a5-9d64-05079b2a3c8c, node_name: "ip-192-168-134-119.us-west-2.compute.internal"

2022-10-21T17:48:04.111638Z INFO models::node::drain: Successfully evicted Pod 'brupop-apiserver-568bdf6f46-zxtcf'
at models/src/node/drain.rs:197
in models::node::drain::evict_pod
in models::node::drain::drain_node with node_name: "ip-192-168-134-119.us-west-2.compute.internal"
in models::node::client::drain_node with selector: BottlerocketShadowSelector { node_name: "ip-192-168-134-119.us-west-2.compute.internal", node_uid: "23bd80ae-aeff-42d3-a9fd-6bce8d518149" }
in apiserver::telemetry::HTTP request with http.method: POST, http.route: /bottlerocket-node-resource/cordon-and-drain, http.flavor: 1.1, http.scheme: https, http.host: brupop-apiserver.brupop-bottlerocket-aws.svc.cluster.local, http.client_ip: [2600:1f13:5ba:2304:3a50::4]:41450, http.user_agent: , http.target: /bottlerocket-node-resource/cordon-and-drain, otel.kind: "server", request_id: c7256d05-c468-46a5-9d64-05079b2a3c8c, node_name: "ip-192-168-134-119.us-west-2.compute.internal"

2022-10-21T17:48:04.185767Z INFO models::node::drain: Successfully evicted Pod 'brupop-controller-deployment-b8b577957-5ln5q'
at models/src/node/drain.rs:197
in models::node::drain::evict_pod
in models::node::drain::drain_node with node_name: "ip-192-168-134-119.us-west-2.compute.internal"
in models::node::client::drain_node with selector: BottlerocketShadowSelector { node_name: "ip-192-168-134-119.us-west-2.compute.internal", node_uid: "23bd80ae-aeff-42d3-a9fd-6bce8d518149" }
in apiserver::telemetry::HTTP request with http.method: POST, http.route: /bottlerocket-node-resource/cordon-and-drain, http.flavor: 1.1, http.scheme: https, http.host: brupop-apiserver.brupop-bottlerocket-aws.svc.cluster.local, http.client_ip: [2600:1f13:5ba:2304:3a50::4]:41450, http.user_agent: , http.target: /bottlerocket-node-resource/cordon-and-drain, otel.kind: "server", request_id: c7256d05-c468-46a5-9d64-05079b2a3c8c, node_name: "ip-192-168-134-119.us-west-2.compute.internal"

2022-10-21T17:48:04.729305Z INFO models::node::drain: Successfully evicted Pod 'nginx-test-85dd6f4cc-g6tsx'
at models/src/node/drain.rs:197
in models::node::drain::evict_pod
in models::node::drain::drain_node with node_name: "ip-192-168-134-119.us-west-2.compute.internal"
in models::node::client::drain_node with selector: BottlerocketShadowSelector { node_name: "ip-192-168-134-119.us-west-2.compute.internal", node_uid: "23bd80ae-aeff-42d3-a9fd-6bce8d518149" }
in apiserver::telemetry::HTTP request with http.method: POST, http.route: /bottlerocket-node-resource/cordon-and-drain, http.flavor: 1.1, http.scheme: https, http.host: brupop-apiserver.brupop-bottlerocket-aws.svc.cluster.local, http.client_ip: [2600:1f13:5ba:2304:3a50::4]:41450, http.user_agent: , http.target: /bottlerocket-node-resource/cordon-and-drain, otel.kind: "server", request_id: c7256d05-c468-46a5-9d64-05079b2a3c8c, node_name: "ip-192-168-134-119.us-west-2.compute.internal"

2022-10-21T17:48:09.120164Z INFO models::node::drain: Pod nginx-test-85dd6f4cc-cxqxh deleted.
at models/src/node/drain.rs:287
in models::node::drain::wait_for_deletion
in models::node::drain::drain_node with node_name: "ip-192-168-134-119.us-west-2.compute.internal"
in models::node::client::drain_node with selector: BottlerocketShadowSelector { node_name: "ip-192-168-134-119.us-west-2.compute.internal", node_uid: "23bd80ae-aeff-42d3-a9fd-6bce8d518149" }
in apiserver::telemetry::HTTP request with http.method: POST, http.route: /bottlerocket-node-resource/cordon-and-drain, http.flavor: 1.1, http.scheme: https, http.host: brupop-apiserver.brupop-bottlerocket-aws.svc.cluster.local, http.client_ip: [2600:1f13:5ba:2304:3a50::4]:41450, http.user_agent: , http.target: /bottlerocket-node-resource/cordon-and-drain, otel.kind: "server", request_id: c7256d05-c468-46a5-9d64-05079b2a3c8c, node_name: "ip-192-168-134-119.us-west-2.compute.internal"

2022-10-21T17:48:09.120256Z INFO models::node::drain: Attempting to evict pod nginx-test-85dd6f4cc-p5j75
at models/src/node/drain.rs:192
in models::node::drain::evict_pod
in models::node::drain::drain_node with node_name: "ip-192-168-134-119.us-west-2.compute.internal"
in models::node::client::drain_node with selector: BottlerocketShadowSelector { node_name: "ip-192-168-134-119.us-west-2.compute.internal", node_uid: "23bd80ae-aeff-42d3-a9fd-6bce8d518149" }
in apiserver::telemetry::HTTP request with http.method: POST, http.route: /bottlerocket-node-resource/cordon-and-drain, http.flavor: 1.1, http.scheme: https, http.host: brupop-apiserver.brupop-bottlerocket-aws.svc.cluster.local, http.client_ip: [2600:1f13:5ba:2304:3a50::4]:41450, http.user_agent: , http.target: /bottlerocket-node-resource/cordon-and-drain, otel.kind: "server", request_id: c7256d05-c468-46a5-9d64-05079b2a3c8c, node_name: "ip-192-168-134-119.us-west-2.compute.internal"

2022-10-21T17:48:09.156143Z INFO models::node::drain: Successfully evicted Pod 'nginx-test-85dd6f4cc-p5j75'
at models/src/node/drain.rs:197
in models::node::drain::evict_pod
in models::node::drain::drain_node with node_name: "ip-192-168-134-119.us-west-2.compute.internal"
in models::node::client::drain_node with selector: BottlerocketShadowSelector { node_name: "ip-192-168-134-119.us-west-2.compute.internal", node_uid: "23bd80ae-aeff-42d3-a9fd-6bce8d518149" }
in apiserver::telemetry::HTTP request with http.method: POST, http.route: /bottlerocket-node-resource/cordon-and-drain, http.flavor: 1.1, http.scheme: https, http.host: brupop-apiserver.brupop-bottlerocket-aws.svc.cluster.local, http.client_ip: [2600:1f13:5ba:2304:3a50::4]:41450, http.user_agent: , http.target: /bottlerocket-node-resource/cordon-and-drain, otel.kind: "server", request_id: c7256d05-c468-46a5-9d64-05079b2a3c8c, node_name: "ip-192-168-134-119.us-west-2.compute.internal"

2022-10-21T17:48:09.217593Z INFO models::node::drain: Pod brupop-controller-deployment-b8b577957-5ln5q deleted.
at models/src/node/drain.rs:287
in models::node::drain::wait_for_deletion
in models::node::drain::drain_node with node_name: "ip-192-168-134-119.us-west-2.compute.internal"
in models::node::client::drain_node with selector: BottlerocketShadowSelector { node_name: "ip-192-168-134-119.us-west-2.compute.internal", node_uid: "23bd80ae-aeff-42d3-a9fd-6bce8d518149" }
in apiserver::telemetry::HTTP request with http.method: POST, http.route: /bottlerocket-node-resource/cordon-and-drain, http.flavor: 1.1, http.scheme: https, http.host: brupop-apiserver.brupop-bottlerocket-aws.svc.cluster.local, http.client_ip: [2600:1f13:5ba:2304:3a50::4]:41450, http.user_agent: , http.target: /bottlerocket-node-resource/cordon-and-drain, otel.kind: "server", request_id: c7256d05-c468-46a5-9d64-05079b2a3c8c, node_name: "ip-192-168-134-119.us-west-2.compute.internal"

2022-10-21T17:48:09.217770Z INFO models::node::drain: Attempting to evict pod web-test-1
at models/src/node/drain.rs:192
in models::node::drain::evict_pod
in models::node::drain::drain_node with node_name: "ip-192-168-134-119.us-west-2.compute.internal"
in models::node::client::drain_node with selector: BottlerocketShadowSelector { node_name: "ip-192-168-134-119.us-west-2.compute.internal", node_uid: "23bd80ae-aeff-42d3-a9fd-6bce8d518149" }
in apiserver::telemetry::HTTP request with http.method: POST, http.route: /bottlerocket-node-resource/cordon-and-drain, http.flavor: 1.1, http.scheme: https, http.host: brupop-apiserver.brupop-bottlerocket-aws.svc.cluster.local, http.client_ip: [2600:1f13:5ba:2304:3a50::4]:41450, http.user_agent: , http.target: /bottlerocket-node-resource/cordon-and-drain, otel.kind: "server", request_id: c7256d05-c468-46a5-9d64-05079b2a3c8c, node_name: "ip-192-168-134-119.us-west-2.compute.internal"

2022-10-21T17:48:09.281139Z INFO models::node::drain: Successfully evicted Pod 'web-test-1'
at models/src/node/drain.rs:197
in models::node::drain::evict_pod
in models::node::drain::drain_node with node_name: "ip-192-168-134-119.us-west-2.compute.internal"
in models::node::client::drain_node with selector: BottlerocketShadowSelector { node_name: "ip-192-168-134-119.us-west-2.compute.internal", node_uid: "23bd80ae-aeff-42d3-a9fd-6bce8d518149" }
in apiserver::telemetry::HTTP request with http.method: POST, http.route: /bottlerocket-node-resource/cordon-and-drain, http.flavor: 1.1, http.scheme: https, http.host: brupop-apiserver.brupop-bottlerocket-aws.svc.cluster.local, http.client_ip: [2600:1f13:5ba:2304:3a50::4]:41450, http.user_agent: , http.target: /bottlerocket-node-resource/cordon-and-drain, otel.kind: "server", request_id: c7256d05-c468-46a5-9d64-05079b2a3c8c, node_name: "ip-192-168-134-119.us-west-2.compute.internal"

2022-10-21T17:48:09.751231Z INFO models::node::drain: Pod nginx-test-85dd6f4cc-g6tsx deleted.
at models/src/node/drain.rs:287
in models::node::drain::wait_for_deletion
in models::node::drain::drain_node with node_name: "ip-192-168-134-119.us-west-2.compute.internal"
in models::node::client::drain_node with selector: BottlerocketShadowSelector { node_name: "ip-192-168-134-119.us-west-2.compute.internal", node_uid: "23bd80ae-aeff-42d3-a9fd-6bce8d518149" }
in apiserver::telemetry::HTTP request with http.method: POST, http.route: /bottlerocket-node-resource/cordon-and-drain, http.flavor: 1.1, http.scheme: https, http.host: brupop-apiserver.brupop-bottlerocket-aws.svc.cluster.local, http.client_ip: [2600:1f13:5ba:2304:3a50::4]:41450, http.user_agent: , http.target: /bottlerocket-node-resource/cordon-and-drain, otel.kind: "server", request_id: c7256d05-c468-46a5-9d64-05079b2a3c8c, node_name: "ip-192-168-134-119.us-west-2.compute.internal"

2022-10-21T17:48:09.751367Z INFO models::node::drain: Not draining Pod 'aws-node-87nlt': Pod is member of a DaemonSet
at models/src/node/drain.rs:150
in models::node::drain::drain_node with node_name: "ip-192-168-134-119.us-west-2.compute.internal"
in models::node::client::drain_node with selector: BottlerocketShadowSelector { node_name: "ip-192-168-134-119.us-west-2.compute.internal", node_uid: "23bd80ae-aeff-42d3-a9fd-6bce8d518149" }
in apiserver::telemetry::HTTP request with http.method: POST, http.route: /bottlerocket-node-resource/cordon-and-drain, http.flavor: 1.1, http.scheme: https, http.host: brupop-apiserver.brupop-bottlerocket-aws.svc.cluster.local, http.client_ip: [2600:1f13:5ba:2304:3a50::4]:41450, http.user_agent: , http.target: /bottlerocket-node-resource/cordon-and-drain, otel.kind: "server", request_id: c7256d05-c468-46a5-9d64-05079b2a3c8c, node_name: "ip-192-168-134-119.us-west-2.compute.internal"

2022-10-21T17:48:09.751523Z INFO models::node::drain: Attempting to evict pod coredns-85d5b4454c-f4t2m
at models/src/node/drain.rs:192
in models::node::drain::evict_pod
in models::node::drain::drain_node with node_name: "ip-192-168-134-119.us-west-2.compute.internal"
in models::node::client::drain_node with selector: BottlerocketShadowSelector { node_name: "ip-192-168-134-119.us-west-2.compute.internal", node_uid: "23bd80ae-aeff-42d3-a9fd-6bce8d518149" }
in apiserver::telemetry::HTTP request with http.method: POST, http.route: /bottlerocket-node-resource/cordon-and-drain, http.flavor: 1.1, http.scheme: https, http.host: brupop-apiserver.brupop-bottlerocket-aws.svc.cluster.local, http.client_ip: [2600:1f13:5ba:2304:3a50::4]:41450, http.user_agent: , http.target: /bottlerocket-node-resource/cordon-and-drain, otel.kind: "server", request_id: c7256d05-c468-46a5-9d64-05079b2a3c8c, node_name: "ip-192-168-134-119.us-west-2.compute.internal"

2022-10-21T17:48:09.768456Z INFO models::node::drain: Successfully evicted Pod 'coredns-85d5b4454c-f4t2m'
at models/src/node/drain.rs:197
in models::node::drain::evict_pod
in models::node::drain::drain_node with node_name: "ip-192-168-134-119.us-west-2.compute.internal"
in models::node::client::drain_node with selector: BottlerocketShadowSelector { node_name: "ip-192-168-134-119.us-west-2.compute.internal", node_uid: "23bd80ae-aeff-42d3-a9fd-6bce8d518149" }
in apiserver::telemetry::HTTP request with http.method: POST, http.route: /bottlerocket-node-resource/cordon-and-drain, http.flavor: 1.1, http.scheme: https, http.host: brupop-apiserver.brupop-bottlerocket-aws.svc.cluster.local, http.client_ip: [2600:1f13:5ba:2304:3a50::4]:41450, http.user_agent: , http.target: /bottlerocket-node-resource/cordon-and-drain, otel.kind: "server", request_id: c7256d05-c468-46a5-9d64-05079b2a3c8c, node_name: "ip-192-168-134-119.us-west-2.compute.internal"

2022-10-21T17:48:19.132940Z INFO models::node::drain: Pod brupop-apiserver-568bdf6f46-fs8mc deleted.
at models/src/node/drain.rs:287
in models::node::drain::wait_for_deletion
in models::node::drain::drain_node with node_name: "ip-192-168-134-119.us-west-2.compute.internal"
in models::node::client::drain_node with selector: BottlerocketShadowSelector { node_name: "ip-192-168-134-119.us-west-2.compute.internal", node_uid: "23bd80ae-aeff-42d3-a9fd-6bce8d518149" }
in apiserver::telemetry::HTTP request with http.method: POST, http.route: /bottlerocket-node-resource/cordon-and-drain, http.flavor: 1.1, http.scheme: https, http.host: brupop-apiserver.brupop-bottlerocket-aws.svc.cluster.local, http.client_ip: [2600:1f13:5ba:2304:3a50::4]:41450, http.user_agent: , http.target: /bottlerocket-node-resource/cordon-and-drain, otel.kind: "server", request_id: c7256d05-c468-46a5-9d64-05079b2a3c8c, node_name: "ip-192-168-134-119.us-west-2.compute.internal"

2022-10-21T17:48:19.133026Z INFO models::node::drain: Not draining Pod 'kube-proxy-vstbj': Pod is member of a DaemonSet
at models/src/node/drain.rs:150
in models::node::drain::drain_node with node_name: "ip-192-168-134-119.us-west-2.compute.internal"
in models::node::client::drain_node with selector: BottlerocketShadowSelector { node_name: "ip-192-168-134-119.us-west-2.compute.internal", node_uid: "23bd80ae-aeff-42d3-a9fd-6bce8d518149" }
in apiserver::telemetry::HTTP request with http.method: POST, http.route: /bottlerocket-node-resource/cordon-and-drain, http.flavor: 1.1, http.scheme: https, http.host: brupop-apiserver.brupop-bottlerocket-aws.svc.cluster.local, http.client_ip: [2600:1f13:5ba:2304:3a50::4]:41450, http.user_agent: , http.target: /bottlerocket-node-resource/cordon-and-drain, otel.kind: "server", request_id: c7256d05-c468-46a5-9d64-05079b2a3c8c, node_name: "ip-192-168-134-119.us-west-2.compute.internal"

2022-10-21T17:48:19.143614Z INFO models::node::drain: Pod brupop-apiserver-568bdf6f46-zxtcf deleted.
at models/src/node/drain.rs:287
in models::node::drain::wait_for_deletion
in models::node::drain::drain_node with node_name: "ip-192-168-134-119.us-west-2.compute.internal"
in models::node::client::drain_node with selector: BottlerocketShadowSelector { node_name: "ip-192-168-134-119.us-west-2.compute.internal", node_uid: "23bd80ae-aeff-42d3-a9fd-6bce8d518149" }
in apiserver::telemetry::HTTP request with http.method: POST, http.route: /bottlerocket-node-resource/cordon-and-drain, http.flavor: 1.1, http.scheme: https, http.host: brupop-apiserver.brupop-bottlerocket-aws.svc.cluster.local, http.client_ip: [2600:1f13:5ba:2304:3a50::4]:41450, http.user_agent: , http.target: /bottlerocket-node-resource/cordon-and-drain, otel.kind: "server", request_id: c7256d05-c468-46a5-9d64-05079b2a3c8c, node_name: "ip-192-168-134-119.us-west-2.compute.internal"

2022-10-21T17:48:19.179463Z INFO models::node::drain: Pod nginx-test-85dd6f4cc-p5j75 deleted.
at models/src/node/drain.rs:287
in models::node::drain::wait_for_deletion
in models::node::drain::drain_node with node_name: "ip-192-168-134-119.us-west-2.compute.internal"
in models::node::client::drain_node with selector: BottlerocketShadowSelector { node_name: "ip-192-168-134-119.us-west-2.compute.internal", node_uid: "23bd80ae-aeff-42d3-a9fd-6bce8d518149" }
in apiserver::telemetry::HTTP request with http.method: POST, http.route: /bottlerocket-node-resource/cordon-and-drain, http.flavor: 1.1, http.scheme: https, http.host: brupop-apiserver.brupop-bottlerocket-aws.svc.cluster.local, http.client_ip: [2600:1f13:5ba:2304:3a50::4]:41450, http.user_agent: , http.target: /bottlerocket-node-resource/cordon-and-drain, otel.kind: "server", request_id: c7256d05-c468-46a5-9d64-05079b2a3c8c, node_name: "ip-192-168-134-119.us-west-2.compute.internal"

2022-10-21T17:48:19.312806Z INFO models::node::drain: Pod web-test-1 deleted.
at models/src/node/drain.rs:292
in models::node::drain::wait_for_deletion
in models::node::drain::drain_node with node_name: "ip-192-168-134-119.us-west-2.compute.internal"
in models::node::client::drain_node with selector: BottlerocketShadowSelector { node_name: "ip-192-168-134-119.us-west-2.compute.internal", node_uid: "23bd80ae-aeff-42d3-a9fd-6bce8d518149" }
in apiserver::telemetry::HTTP request with http.method: POST, http.route: /bottlerocket-node-resource/cordon-and-drain, http.flavor: 1.1, http.scheme: https, http.host: brupop-apiserver.brupop-bottlerocket-aws.svc.cluster.local, http.client_ip: [2600:1f13:5ba:2304:3a50::4]:41450, http.user_agent: , http.target: /bottlerocket-node-resource/cordon-and-drain, otel.kind: "server", request_id: c7256d05-c468-46a5-9d64-05079b2a3c8c, node_name: "ip-192-168-134-119.us-west-2.compute.internal"

2022-10-21T17:48:19.794820Z INFO models::node::drain: Pod coredns-85d5b4454c-f4t2m deleted.
at models/src/node/drain.rs:287
in models::node::drain::wait_for_deletion
in models::node::drain::drain_node with node_name: "ip-192-168-134-119.us-west-2.compute.internal"
in models::node::client::drain_node with selector: BottlerocketShadowSelector { node_name: "ip-192-168-134-119.us-west-2.compute.internal", node_uid: "23bd80ae-aeff-42d3-a9fd-6bce8d518149" }
in apiserver::telemetry::HTTP request with http.method: POST, http.route: /bottlerocket-node-resource/cordon-and-drain, http.flavor: 1.1, http.scheme: https, http.host: brupop-apiserver.brupop-bottlerocket-aws.svc.cluster.local, http.client_ip: [2600:1f13:5ba:2304:3a50::4]:41450, http.user_agent: , http.target: /bottlerocket-node-resource/cordon-and-drain, otel.kind: "server", request_id: c7256d05-c468-46a5-9d64-05079b2a3c8c, node_name: "ip-192-168-134-119.us-west-2.compute.internal"

2022-10-21T17:49:39.636441Z INFO apiserver::api::node: Original conversion request, conversion_req: Json(ConversionRequest { kind: "ConversionReview", api_version: "apiextensions.k8s.io/v1", request: Request { uid: "db02c621-d917-4a19-9a20-fbd16d38fd14", desired_api_version: "brupop.bottlerocket.aws/v1", objects: [Object {"apiVersion": String("brupop.bottlerocket.aws/v2"), "kind": String("BottlerocketShadow"), "metadata": Object {"creationTimestamp": String("2022-10-21T17:34:26Z"), "generation": Number(3), "managedFields": Array [Object {"apiVersion": String("brupop.bottlerocket.aws/v2"), "fieldsType": String("FieldsV1"), "fieldsV1": Object {"f:metadata": Object {"f:ownerReferences": Object {".": Object {}, "k:{"uid":"23bd80ae-aeff-42d3-a9fd-6bce8d518149"}": Object {".": Object {}, "f:apiVersion": Object {}, "f:kind": Object {}, "f:name": Object {}, "f:uid": Object {}}}}, "f:spec": Object {".": Object {}, "f:state": Object {}, "f:state_transition_timestamp": Object {}, "f:version": Object {}}, "f:status": Object {".": Object {}, "f:crash_count": Object {}, "f:current_state": Object {}, "f:current_version": Object {}, "f:target_version": Object {}}}, "manager": String("unknown"), "operation": String("Update"), "time": String("2022-10-21T17:34:46Z")}], "name": String("brs-ip-192-168-134-119.us-west-2.compute.internal"), "namespace": String("brupop-bottlerocket-aws"), "ownerReferences": Array [Object {"apiVersion": String("v1"), "kind": String("Node"), "name": String("ip-192-168-134-119.us-west-2.compute.internal"), "uid": String("23bd80ae-aeff-42d3-a9fd-6bce8d518149")}], "uid": String("261bcda8-ff86-451c-b098-902e53ff83ed")}, "spec": Object {"state": String("RebootedIntoUpdate"), "state_transition_timestamp": String("2022-10-21T17:47:59.392863704+00:00"), "version": String("1.10.1")}, "status": Object {"crash_count": Number(0), "current_state": String("RebootedIntoUpdate"), "current_version": String("1.10.1"), "target_version": String("1.10.1")}}] } })
at apiserver/src/api/node.rs:53
in apiserver::telemetry::HTTP request with http.method: POST, http.route: /crdconvert, http.flavor: 1.1, http.scheme: https, http.host: brupop-apiserver.brupop-bottlerocket-aws.svc:443, http.client_ip: [2600:1f13:5ba:2303:7fca:952:3ffc:8c55]:50314, http.user_agent: kube-apiserver-admission, http.target: /crdconvert?timeout=30s, otel.kind: "server", request_id: 7ccda13f-0de9-4b57-ba4d-4299515fb3b2

2022-10-21T17:49:39.636693Z INFO apiserver::api::node: Converted response:, response_string: "{"kind":"ConversionReview","apiVersion":"apiextensions.k8s.io/v1","response":{"uid":"db02c621-d917-4a19-9a20-fbd16d38fd14","result":{"status":"Success","message":null},"convertedObjects":[{"apiVersion":"brupop.bottlerocket.aws/v1","kind":"BottlerocketShadow","metadata":{"name":"brs-ip-192-168-134-119.us-west-2.compute.internal","namespace":"brupop-bottlerocket-aws","ownerReferences":[{"apiVersion":"v1","kind":"Node","name":"ip-192-168-134-119.us-west-2.compute.internal","uid":"23bd80ae-aeff-42d3-a9fd-6bce8d518149"}],"uid":"261bcda8-ff86-451c-b098-902e53ff83ed"},"spec":{"state":"RebootedIntoUpdate","state_transition_timestamp":"2022-10-21T17:47:59.392863704+00:00","version":"1.10.1"},"status":{"current_version":"1.10.1","target_version":"1.10.1","current_state":"RebootedIntoUpdate"}}]}}"
at apiserver/src/api/node.rs:56
in apiserver::telemetry::HTTP request with http.method: POST, http.route: /crdconvert, http.flavor: 1.1, http.scheme: https, http.host: brupop-apiserver.brupop-bottlerocket-aws.svc:443, http.client_ip: [2600:1f13:5ba:2303:7fca:952:3ffc:8c55]:50314, http.user_agent: kube-apiserver-admission, http.target: /crdconvert?timeout=30s, otel.kind: "server", request_id: 7ccda13f-0de9-4b57-ba4d-4299515fb3b2

2022-10-21T17:49:40.736035Z INFO apiserver::api::node: Original conversion request, conversion_req: Json(ConversionRequest { kind: "ConversionReview", api_version: "apiextensions.k8s.io/v1", request: Request { uid: "72139a98-53d1-4671-b91a-c4dac8eb68da", desired_api_version: "brupop.bottlerocket.aws/v1", objects: [Object {"apiVersion": String("brupop.bottlerocket.aws/v2"), "kind": String("BottlerocketShadow"), "metadata": Object {"creationTimestamp": String("2022-10-21T17:34:26Z"), "generation": Number(4), "managedFields": Array [Object {"apiVersion": String("brupop.bottlerocket.aws/v2"), "fieldsType": String("FieldsV1"), "fieldsV1": Object {"f:metadata": Object {"f:ownerReferences": Object {".": Object {}, "k:{"uid":"23bd80ae-aeff-42d3-a9fd-6bce8d518149"}": Object {".": Object {}, "f:apiVersion": Object {}, "f:kind": Object {}, "f:name": Object {}, "f:uid": Object {}}}}, "f:spec": Object {".": Object {}, "f:state": Object {}, "f:state_transition_timestamp": Object {}, "f:version": Object {}}, "f:status": Object {".": Object {}, "f:crash_count": Object {}, "f:current_state": Object {}, "f:current_version": Object {}, "f:target_version": Object {}}}, "manager": String("unknown"), "operation": String("Update"), "time": String("2022-10-21T17:34:46Z")}], "name": String("brs-ip-192-168-134-119.us-west-2.compute.internal"), "namespace": String("brupop-bottlerocket-aws"), "ownerReferences": Array [Object {"apiVersion": String("v1"), "kind": String("Node"), "name": String("ip-192-168-134-119.us-west-2.compute.internal"), "uid": String("23bd80ae-aeff-42d3-a9fd-6bce8d518149")}], "uid": String("261bcda8-ff86-451c-b098-902e53ff83ed")}, "spec": Object {"state": String("MonitoringUpdate"), "state_transition_timestamp": String("2022-10-21T17:49:40.714189187+00:00"), "version": String("1.10.1")}, "status": Object {"crash_count": Number(0), "current_state": String("RebootedIntoUpdate"), "current_version": String("1.10.1"), "target_version": String("1.10.1")}}] } })
at apiserver/src/api/node.rs:53
in apiserver::telemetry::HTTP request with http.method: POST, http.route: /crdconvert, http.flavor: 1.1, http.scheme: https, http.host: brupop-apiserver.brupop-bottlerocket-aws.svc:443, http.client_ip: [2600:1f13:5ba:2303:7fca:952:3ffc:8c55]:50314, http.user_agent: kube-apiserver-admission, http.target: /crdconvert?timeout=30s, otel.kind: "server", request_id: 4d04c75b-40de-434c-a269-d6a79d5b9bfa

2022-10-21T17:49:40.736312Z INFO apiserver::api::node: Converted response:, response_string: "{"kind":"ConversionReview","apiVersion":"apiextensions.k8s.io/v1","response":{"uid":"72139a98-53d1-4671-b91a-c4dac8eb68da","result":{"status":"Success","message":null},"convertedObjects":[{"apiVersion":"brupop.bottlerocket.aws/v1","kind":"BottlerocketShadow","metadata":{"name":"brs-ip-192-168-134-119.us-west-2.compute.internal","namespace":"brupop-bottlerocket-aws","ownerReferences":[{"apiVersion":"v1","kind":"Node","name":"ip-192-168-134-119.us-west-2.compute.internal","uid":"23bd80ae-aeff-42d3-a9fd-6bce8d518149"}],"uid":"261bcda8-ff86-451c-b098-902e53ff83ed"},"spec":{"state":"MonitoringUpdate","state_transition_timestamp":"2022-10-21T17:49:40.714189187+00:00","version":"1.10.1"},"status":{"current_version":"1.10.1","target_version":"1.10.1","current_state":"RebootedIntoUpdate"}}]}}"
at apiserver/src/api/node.rs:56
in apiserver::telemetry::HTTP request with http.method: POST, http.route: /crdconvert, http.flavor: 1.1, http.scheme: https, http.host: brupop-apiserver.brupop-bottlerocket-aws.svc:443, http.client_ip: [2600:1f13:5ba:2303:7fca:952:3ffc:8c55]:50314, http.user_agent: kube-apiserver-admission, http.target: /crdconvert?timeout=30s, otel.kind: "server", request_id: 4d04c75b-40de-434c-a269-d6a79d5b9bfa

2022-10-21T17:50:25.036039Z INFO apiserver::api::node: Original conversion request, conversion_req: Json(ConversionRequest { kind: "ConversionReview", api_version: "apiextensions.k8s.io/v1", request: Request { uid: "cc0b65b8-c0db-4cc1-ac31-f0ce9b9edf0b", desired_api_version: "brupop.bottlerocket.aws/v1", objects: [Object {"apiVersion": String("brupop.bottlerocket.aws/v2"), "kind": String("BottlerocketShadow"), "metadata": Object {"creationTimestamp": String("2022-10-21T17:34:26Z"), "generation": Number(4), "managedFields": Array [Object {"apiVersion": String("brupop.bottlerocket.aws/v2"), "fieldsType": String("FieldsV1"), "fieldsV1": Object {"f:metadata": Object {"f:ownerReferences": Object {".": Object {}, "k:{"uid":"23bd80ae-aeff-42d3-a9fd-6bce8d518149"}": Object {".": Object {}, "f:apiVersion": Object {}, "f:kind": Object {}, "f:name": Object {}, "f:uid": Object {}}}}, "f:spec": Object {".": Object {}, "f:state": Object {}, "f:state_transition_timestamp": Object {}, "f:version": Object {}}, "f:status": Object {".": Object {}, "f:crash_count": Object {}, "f:current_state": Object {}, "f:current_version": Object {}, "f:target_version": Object {}}}, "manager": String("unknown"), "operation": String("Update"), "time": String("2022-10-21T17:34:46Z")}], "name": String("brs-ip-192-168-134-119.us-west-2.compute.internal"), "namespace": String("brupop-bottlerocket-aws"), "ownerReferences": Array [Object {"apiVersion": String("v1"), "kind": String("Node"), "name": String("ip-192-168-134-119.us-west-2.compute.internal"), "uid": String("23bd80ae-aeff-42d3-a9fd-6bce8d518149")}], "uid": String("261bcda8-ff86-451c-b098-902e53ff83ed")}, "spec": Object {"state": String("MonitoringUpdate"), "state_transition_timestamp": String("2022-10-21T17:49:40.714189187+00:00"), "version": String("1.10.1")}, "status": Object {"crash_count": Number(0), "current_state": String("MonitoringUpdate"), "current_version": String("1.10.1"), "target_version": String("1.10.1")}}] } })
at apiserver/src/api/node.rs:53
in apiserver::telemetry::HTTP request with http.method: POST, http.route: /crdconvert, http.flavor: 1.1, http.scheme: https, http.host: brupop-apiserver.brupop-bottlerocket-aws.svc:443, http.client_ip: [2600:1f13:5ba:2303:7fca:952:3ffc:8c55]:50934, http.user_agent: kube-apiserver-admission, http.target: /crdconvert?timeout=30s, otel.kind: "server", request_id: 4fae519a-c44f-431f-ae15-1b87eeb30b88

2022-10-21T17:50:25.036306Z INFO apiserver::api::node: Converted response:, response_string: "{"kind":"ConversionReview","apiVersion":"apiextensions.k8s.io/v1","response":{"uid":"cc0b65b8-c0db-4cc1-ac31-f0ce9b9edf0b","result":{"status":"Success","message":null},"convertedObjects":[{"apiVersion":"brupop.bottlerocket.aws/v1","kind":"BottlerocketShadow","metadata":{"name":"brs-ip-192-168-134-119.us-west-2.compute.internal","namespace":"brupop-bottlerocket-aws","ownerReferences":[{"apiVersion":"v1","kind":"Node","name":"ip-192-168-134-119.us-west-2.compute.internal","uid":"23bd80ae-aeff-42d3-a9fd-6bce8d518149"}],"uid":"261bcda8-ff86-451c-b098-902e53ff83ed"},"spec":{"state":"MonitoringUpdate","state_transition_timestamp":"2022-10-21T17:49:40.714189187+00:00","version":"1.10.1"},"status":{"current_version":"1.10.1","target_version":"1.10.1","current_state":"MonitoringUpdate"}}]}}"
at apiserver/src/api/node.rs:56
in apiserver::telemetry::HTTP request with http.method: POST, http.route: /crdconvert, http.flavor: 1.1, http.scheme: https, http.host: brupop-apiserver.brupop-bottlerocket-aws.svc:443, http.client_ip: [2600:1f13:5ba:2303:7fca:952:3ffc:8c55]:50934, http.user_agent: kube-apiserver-admission, http.target: /crdconvert?timeout=30s, otel.kind: "server", request_id: 4fae519a-c44f-431f-ae15-1b87eeb30b88

2022-10-21T17:50:26.796844Z INFO apiserver::api::node: Original conversion request, conversion_req: Json(ConversionRequest { kind: "ConversionReview", api_version: "apiextensions.k8s.io/v1", request: Request { uid: "d351fb7b-5167-4f25-985f-1b636df7b1b1", desired_api_version: "brupop.bottlerocket.aws/v1", objects: [Object {"apiVersion": String("brupop.bottlerocket.aws/v2"), "kind": String("BottlerocketShadow"), "metadata": Object {"creationTimestamp": String("2022-10-21T17:34:26Z"), "generation": Number(5), "managedFields": Array [Object {"apiVersion": String("brupop.bottlerocket.aws/v2"), "fieldsType": String("FieldsV1"), "fieldsV1": Object {"f:metadata": Object {"f:ownerReferences": Object {".": Object {}, "k:{"uid":"23bd80ae-aeff-42d3-a9fd-6bce8d518149"}": Object {".": Object {}, "f:apiVersion": Object {}, "f:kind": Object {}, "f:name": Object {}, "f:uid": Object {}}}}, "f:spec": Object {".": Object {}, "f:state": Object {}, "f:state_transition_timestamp": Object {}, "f:version": Object {}}, "f:status": Object {".": Object {}, "f:crash_count": Object {}, "f:current_state": Object {}, "f:current_version": Object {}, "f:target_version": Object {}}}, "manager": String("unknown"), "operation": String("Update"), "time": String("2022-10-21T17:34:46Z")}], "name": String("brs-ip-192-168-134-119.us-west-2.compute.internal"), "namespace": String("brupop-bottlerocket-aws"), "ownerReferences": Array [Object {"apiVersion": String("v1"), "kind": String("Node"), "name": String("ip-192-168-134-119.us-west-2.compute.internal"), "uid": String("23bd80ae-aeff-42d3-a9fd-6bce8d518149")}], "uid": String("261bcda8-ff86-451c-b098-902e53ff83ed")}, "spec": Object {"state": String("Idle"), "state_transition_timestamp": String("2022-10-21T17:50:26.775740057+00:00"), "version": String("1.10.1")}, "status": Object {"crash_count": Number(0), "current_state": String("MonitoringUpdate"), "current_version": String("1.10.1"), "target_version": String("1.10.1")}}] } })
at apiserver/src/api/node.rs:53
in apiserver::telemetry::HTTP request with http.method: POST, http.route: /crdconvert, http.flavor: 1.1, http.scheme: https, http.host: brupop-apiserver.brupop-bottlerocket-aws.svc:443, http.client_ip: [2600:1f13:5ba:2303:7fca:952:3ffc:8c55]:50934, http.user_agent: kube-apiserver-admission, http.target: /crdconvert?timeout=30s, otel.kind: "server", request_id: 6bd3d706-22db-4006-bcee-4dc2e2664983

2022-10-21T17:50:26.797175Z INFO apiserver::api::node: Converted response:, response_string: "{"kind":"ConversionReview","apiVersion":"apiextensions.k8s.io/v1","response":{"uid":"d351fb7b-5167-4f25-985f-1b636df7b1b1","result":{"status":"Success","message":null},"convertedObjects":[{"apiVersion":"brupop.bottlerocket.aws/v1","kind":"BottlerocketShadow","metadata":{"name":"brs-ip-192-168-134-119.us-west-2.compute.internal","namespace":"brupop-bottlerocket-aws","ownerReferences":[{"apiVersion":"v1","kind":"Node","name":"ip-192-168-134-119.us-west-2.compute.internal","uid":"23bd80ae-aeff-42d3-a9fd-6bce8d518149"}],"uid":"261bcda8-ff86-451c-b098-902e53ff83ed"},"spec":{"state":"Idle","state_transition_timestamp":"2022-10-21T17:50:26.775740057+00:00","version":"1.10.1"},"status":{"current_version":"1.10.1","target_version":"1.10.1","current_state":"MonitoringUpdate"}}]}}"
at apiserver/src/api/node.rs:56
in apiserver::telemetry::HTTP request with http.method: POST, http.route: /crdconvert, http.flavor: 1.1, http.scheme: https, http.host: brupop-apiserver.brupop-bottlerocket-aws.svc:443, http.client_ip: [2600:1f13:5ba:2303:7fca:952:3ffc:8c55]:50934, http.user_agent: kube-apiserver-admission, http.target: /crdconvert?timeout=30s, otel.kind: "server", request_id: 6bd3d706-22db-4006-bcee-4dc2e2664983

2022-10-21T17:50:50.179924Z INFO apiserver::api::node: Original conversion request, conversion_req: Json(ConversionRequest { kind: "ConversionReview", api_version: "apiextensions.k8s.io/v1", request: Request { uid: "77476601-5752-4ba1-b43a-d17a191f5f9e", desired_api_version: "brupop.bottlerocket.aws/v1", objects: [Object {"apiVersion": String("brupop.bottlerocket.aws/v2"), "kind": String("BottlerocketShadow"), "metadata": Object {"creationTimestamp": String("2022-10-21T17:34:26Z"), "generation": Number(5), "managedFields": Array [Object {"apiVersion": String("brupop.bottlerocket.aws/v2"), "fieldsType": String("FieldsV1"), "fieldsV1": Object {"f:metadata": Object {"f:ownerReferences": Object {".": Object {}, "k:{"uid":"23bd80ae-aeff-42d3-a9fd-6bce8d518149"}": Object {".": Object {}, "f:apiVersion": Object {}, "f:kind": Object {}, "f:name": Object {}, "f:uid": Object {}}}}, "f:spec": Object {".": Object {}, "f:state": Object {}, "f:state_transition_timestamp": Object {}, "f:version": Object {}}, "f:status": Object {".": Object {}, "f:crash_count": Object {}, "f:current_state": Object {}, "f:current_version": Object {}, "f:target_version": Object {}}}, "manager": String("unknown"), "operation": String("Update"), "time": String("2022-10-21T17:34:46Z")}], "name": String("brs-ip-192-168-134-119.us-west-2.compute.internal"), "namespace": String("brupop-bottlerocket-aws"), "ownerReferences": Array [Object {"apiVersion": String("v1"), "kind": String("Node"), "name": String("ip-192-168-134-119.us-west-2.compute.internal"), "uid": String("23bd80ae-aeff-42d3-a9fd-6bce8d518149")}], "uid": String("261bcda8-ff86-451c-b098-902e53ff83ed")}, "spec": Object {"state": String("Idle"), "state_transition_timestamp": String("2022-10-21T17:50:26.775740057+00:00"), "version": String("1.10.1")}, "status": Object {"crash_count": Number(0), "current_state": String("Idle"), "current_version": String("1.10.1"), "target_version": String("1.10.1")}}] } })
at apiserver/src/api/node.rs:53
in apiserver::telemetry::HTTP request with http.method: POST, http.route: /crdconvert, http.flavor: 1.1, http.scheme: https, http.host: brupop-apiserver.brupop-bottlerocket-aws.svc:443, http.client_ip: [2600:1f13:5ba:2303:7fca:952:3ffc:8c55]:46064, http.user_agent: kube-apiserver-admission, http.target: /crdconvert?timeout=30s, otel.kind: "server", request_id: fe08d99c-6d91-49f1-8600-32876fe6ae64

2022-10-21T17:50:50.180210Z INFO apiserver::api::node: Converted response:, response_string: "{"kind":"ConversionReview","apiVersion":"apiextensions.k8s.io/v1","response":{"uid":"77476601-5752-4ba1-b43a-d17a191f5f9e","result":{"status":"Success","message":null},"convertedObjects":[{"apiVersion":"brupop.bottlerocket.aws/v1","kind":"BottlerocketShadow","metadata":{"name":"brs-ip-192-168-134-119.us-west-2.compute.internal","namespace":"brupop-bottlerocket-aws","ownerReferences":[{"apiVersion":"v1","kind":"Node","name":"ip-192-168-134-119.us-west-2.compute.internal","uid":"23bd80ae-aeff-42d3-a9fd-6bce8d518149"}],"uid":"261bcda8-ff86-451c-b098-902e53ff83ed"},"spec":{"state":"Idle","state_transition_timestamp":"2022-10-21T17:50:26.775740057+00:00","version":"1.10.1"},"status":{"current_version":"1.10.1","target_version":"1.10.1","current_state":"Idle"}}]}}"
at apiserver/src/api/node.rs:56
in apiserver::telemetry::HTTP request with http.method: POST, http.route: /crdconvert, http.flavor: 1.1, http.scheme: https, http.host: brupop-apiserver.brupop-bottlerocket-aws.svc:443, http.client_ip: [2600:1f13:5ba:2303:7fca:952:3ffc:8c55]:46064, http.user_agent: kube-apiserver-admission, http.target: /crdconvert?timeout=30s, otel.kind: "server", request_id: fe08d99c-6d91-49f1-8600-32876fe6ae64

2022-10-21T17:50:50.180781Z INFO apiserver::api::node: Original conversion request, conversion_req: Json(ConversionRequest { kind: "ConversionReview", api_version: "apiextensions.k8s.io/v1", request: Request { uid: "0f4ac969-4630-4031-9391-a0c209857157", desired_api_version: "brupop.bottlerocket.aws/v1", objects: [Object {"apiVersion": String("brupop.bottlerocket.aws/v2"), "kind": String("BottlerocketShadow"), "metadata": Object {"creationTimestamp": String("2022-10-21T17:34:26Z"), "generation": Number(5), "managedFields": Array [Object {"apiVersion": String("brupop.bottlerocket.aws/v2"), "fieldsType": String("FieldsV1"), "fieldsV1": Object {"f:metadata": Object {"f:ownerReferences": Object {".": Object {}, "k:{"uid":"23bd80ae-aeff-42d3-a9fd-6bce8d518149"}": Object {".": Object {}, "f:apiVersion": Object {}, "f:kind": Object {}, "f:name": Object {}, "f:uid": Object {}}}}, "f:spec": Object {".": Object {}, "f:state": Object {}, "f:state_transition_timestamp": Object {}, "f:version": Object {}}, "f:status": Object {".": Object {}, "f:crash_count": Object {}, "f:current_state": Object {}, "f:current_version": Object {}, "f:target_version": Object {}}}, "manager": String("unknown"), "operation": String("Update"), "time": String("2022-10-21T17:34:46Z")}], "name": String("brs-ip-192-168-134-119.us-west-2.compute.internal"), "namespace": String("brupop-bottlerocket-aws"), "ownerReferences": Array [Object {"apiVersion": String("v1"), "kind": String("Node"), "name": String("ip-192-168-134-119.us-west-2.compute.internal"), "uid": String("23bd80ae-aeff-42d3-a9fd-6bce8d518149")}], "uid": String("261bcda8-ff86-451c-b098-902e53ff83ed")}, "spec": Object {"state": String("Idle"), "state_transition_timestamp": String("2022-10-21T17:50:26.775740057+00:00"), "version": String("1.10.1")}, "status": Object {"crash_count": Number(0), "current_state": String("Idle"), "current_version": String("1.10.1"), "target_version": String("1.10.1")}}] } })
at apiserver/src/api/node.rs:53
in apiserver::telemetry::HTTP request with http.method: POST, http.route: /crdconvert, http.flavor: 1.1, http.scheme: https, http.host: brupop-apiserver.brupop-bottlerocket-aws.svc:443, http.client_ip: [2600:1f13:5ba:2305:a3a3:d7ff:9878:847d]:39894, http.user_agent: kube-apiserver-admission, http.target: /crdconvert?timeout=30s, otel.kind: "server", request_id: 5dd4e4cf-1bca-4e78-b98f-d9524386a78e

2022-10-21T17:50:50.180989Z INFO apiserver::api::node: Converted response:, response_string: "{"kind":"ConversionReview","apiVersion":"apiextensions.k8s.io/v1","response":{"uid":"0f4ac969-4630-4031-9391-a0c209857157","result":{"status":"Success","message":null},"convertedObjects":[{"apiVersion":"brupop.bottlerocket.aws/v1","kind":"BottlerocketShadow","metadata":{"name":"brs-ip-192-168-134-119.us-west-2.compute.internal","namespace":"brupop-bottlerocket-aws","ownerReferences":[{"apiVersion":"v1","kind":"Node","name":"ip-192-168-134-119.us-west-2.compute.internal","uid":"23bd80ae-aeff-42d3-a9fd-6bce8d518149"}],"uid":"261bcda8-ff86-451c-b098-902e53ff83ed"},"spec":{"state":"Idle","state_transition_timestamp":"2022-10-21T17:50:26.775740057+00:00","version":"1.10.1"},"status":{"current_version":"1.10.1","target_version":"1.10.1","current_state":"Idle"}}]}}"
at apiserver/src/api/node.rs:56
in apiserver::telemetry::HTTP request with http.method: POST, http.route: /crdconvert, http.flavor: 1.1, http.scheme: https, http.host: brupop-apiserver.brupop-bottlerocket-aws.svc:443, http.client_ip: [2600:1f13:5ba:2305:a3a3:d7ff:9878:847d]:39894, http.user_agent: kube-apiserver-admission, http.target: /crdconvert?timeout=30s, otel.kind: "server", request_id: 5dd4e4cf-1bca-4e78-b98f-d9524386a78e

2022-10-21T17:50:50.827843Z INFO apiserver::api::node: Original conversion request, conversion_req: Json(ConversionRequest { kind: "ConversionReview", api_version: "apiextensions.k8s.io/v1", request: Request { uid: "a6a58792-7248-4395-97d0-0f01e0a29ea5", desired_api_version: "brupop.bottlerocket.aws/v1", objects: [Object {"apiVersion": String("brupop.bottlerocket.aws/v2"), "kind": String("BottlerocketShadow"), "metadata": Object {"creationTimestamp": String("2022-10-21T17:34:26Z"), "generation": Number(6), "managedFields": Array [Object {"apiVersion": String("brupop.bottlerocket.aws/v2"), "fieldsType": String("FieldsV1"), "fieldsV1": Object {"f:metadata": Object {"f:ownerReferences": Object {".": Object {}, "k:{"uid":"23bd80ae-aeff-42d3-a9fd-6bce8d518149"}": Object {".": Object {}, "f:apiVersion": Object {}, "f:kind": Object {}, "f:name": Object {}, "f:uid": Object {}}}}, "f:spec": Object {".": Object {}, "f:state": Object {}}, "f:status": Object {".": Object {}, "f:crash_count": Object {}, "f:current_state": Object {}, "f:current_version": Object {}, "f:target_version": Object {}}}, "manager": String("unknown"), "operation": String("Update"), "time": String("2022-10-21T17:34:46Z")}], "name": String("brs-ip-192-168-134-119.us-west-2.compute.internal"), "namespace": String("brupop-bottlerocket-aws"), "ownerReferences": Array [Object {"apiVersion": String("v1"), "kind": String("Node"), "name": String("ip-192-168-134-119.us-west-2.compute.internal"), "uid": String("23bd80ae-aeff-42d3-a9fd-6bce8d518149")}], "uid": String("261bcda8-ff86-451c-b098-902e53ff83ed")}, "spec": Object {"state": String("Idle")}, "status": Object {"crash_count": Number(0), "current_state": String("Idle"), "current_version": String("1.10.1"), "target_version": String("1.10.1")}}] } })
at apiserver/src/api/node.rs:53
in apiserver::telemetry::HTTP request with http.method: POST, http.route: /crdconvert, http.flavor: 1.1, http.scheme: https, http.host: brupop-apiserver.brupop-bottlerocket-aws.svc:443, http.client_ip: [2600:1f13:5ba:2303:7fca:952:3ffc:8c55]:46064, http.user_agent: kube-apiserver-admission, http.target: /crdconvert?timeout=30s, otel.kind: "server", request_id: 97bab766-898b-4d0b-99f3-23d4dce819bd

2022-10-21T17:50:50.827843Z INFO apiserver::api::node: Original conversion request, conversion_req: Json(ConversionRequest { kind: "ConversionReview", api_version: "apiextensions.k8s.io/v1", request: Request { uid: "63d47e02-0426-416d-ab6f-30bf7eaf39fe", desired_api_version: "brupop.bottlerocket.aws/v1", objects: [Object {"apiVersion": String("brupop.bottlerocket.aws/v2"), "kind": String("BottlerocketShadow"), "metadata": Object {"creationTimestamp": String("2022-10-21T17:34:26Z"), "generation": Number(6), "managedFields": Array [Object {"apiVersion": String("brupop.bottlerocket.aws/v2"), "fieldsType": String("FieldsV1"), "fieldsV1": Object {"f:metadata": Object {"f:ownerReferences": Object {".": Object {}, "k:{"uid":"23bd80ae-aeff-42d3-a9fd-6bce8d518149"}": Object {".": Object {}, "f:apiVersion": Object {}, "f:kind": Object {}, "f:name": Object {}, "f:uid": Object {}}}}, "f:spec": Object {".": Object {}, "f:state": Object {}}, "f:status": Object {".": Object {}, "f:crash_count": Object {}, "f:current_state": Object {}, "f:current_version": Object {}, "f:target_version": Object {}}}, "manager": String("unknown"), "operation": String("Update"), "time": String("2022-10-21T17:34:46Z")}], "name": String("brs-ip-192-168-134-119.us-west-2.compute.internal"), "namespace": String("brupop-bottlerocket-aws"), "ownerReferences": Array [Object {"apiVersion": String("v1"), "kind": String("Node"), "name": String("ip-192-168-134-119.us-west-2.compute.internal"), "uid": String("23bd80ae-aeff-42d3-a9fd-6bce8d518149")}], "uid": String("261bcda8-ff86-451c-b098-902e53ff83ed")}, "spec": Object {"state": String("Idle")}, "status": Object {"crash_count": Number(0), "current_state": String("Idle"), "current_version": String("1.10.1"), "target_version": String("1.10.1")}}] } })
at apiserver/src/api/node.rs:53
in apiserver::telemetry::HTTP request with http.method: POST, http.route: /crdconvert, http.flavor: 1.1, http.scheme: https, http.host: brupop-apiserver.brupop-bottlerocket-aws.svc:443, http.client_ip: [2600:1f13:5ba:2305:a3a3:d7ff:9878:847d]:39894, http.user_agent: kube-apiserver-admission, http.target: /crdconvert?timeout=30s, otel.kind: "server", request_id: e9ee0180-84d4-451e-ad62-34de529f9c2b

2022-10-21T17:50:50.828118Z INFO apiserver::api::node: Converted response:, response_string: "{"kind":"ConversionReview","apiVersion":"apiextensions.k8s.io/v1","response":{"uid":"a6a58792-7248-4395-97d0-0f01e0a29ea5","result":{"status":"Success","message":null},"convertedObjects":[{"apiVersion":"brupop.bottlerocket.aws/v1","kind":"BottlerocketShadow","metadata":{"name":"brs-ip-192-168-134-119.us-west-2.compute.internal","namespace":"brupop-bottlerocket-aws","ownerReferences":[{"apiVersion":"v1","kind":"Node","name":"ip-192-168-134-119.us-west-2.compute.internal","uid":"23bd80ae-aeff-42d3-a9fd-6bce8d518149"}],"uid":"261bcda8-ff86-451c-b098-902e53ff83ed"},"spec":{"state":"Idle","state_transition_timestamp":null,"version":null},"status":{"current_version":"1.10.1","target_version":"1.10.1","current_state":"Idle"}}]}}"
at apiserver/src/api/node.rs:56
in apiserver::telemetry::HTTP request with http.method: POST, http.route: /crdconvert, http.flavor: 1.1, http.scheme: https, http.host: brupop-apiserver.brupop-bottlerocket-aws.svc:443, http.client_ip: [2600:1f13:5ba:2303:7fca:952:3ffc:8c55]:46064, http.user_agent: kube-apiserver-admission, http.target: /crdconvert?timeout=30s, otel.kind: "server", request_id: 97bab766-898b-4d0b-99f3-23d4dce819bd

2022-10-21T17:50:50.828118Z INFO apiserver::api::node: Converted response:, response_string: "{"kind":"ConversionReview","apiVersion":"apiextensions.k8s.io/v1","response":{"uid":"63d47e02-0426-416d-ab6f-30bf7eaf39fe","result":{"status":"Success","message":null},"convertedObjects":[{"apiVersion":"brupop.bottlerocket.aws/v1","kind":"BottlerocketShadow","metadata":{"name":"brs-ip-192-168-134-119.us-west-2.compute.internal","namespace":"brupop-bottlerocket-aws","ownerReferences":[{"apiVersion":"v1","kind":"Node","name":"ip-192-168-134-119.us-west-2.compute.internal","uid":"23bd80ae-aeff-42d3-a9fd-6bce8d518149"}],"uid":"261bcda8-ff86-451c-b098-902e53ff83ed"},"spec":{"state":"Idle","state_transition_timestamp":null,"version":null},"status":{"current_version":"1.10.1","target_version":"1.10.1","current_state":"Idle"}}]}}"
at apiserver/src/api/node.rs:56
in apiserver::telemetry::HTTP request with http.method: POST, http.route: /crdconvert, http.flavor: 1.1, http.scheme: https, http.host: brupop-apiserver.brupop-bottlerocket-aws.svc:443, http.client_ip: [2600:1f13:5ba:2305:a3a3:d7ff:9878:847d]:39894, http.user_agent: kube-apiserver-admission, http.target: /crdconvert?timeout=30s, otel.kind: "server", request_id: e9ee0180-84d4-451e-ad62-34de529f9c2b

It's worth noting that there is a short period where the "conversion" logs from v2 to v1 show up and they they don't appear again. This happens when the controller / servers are first coming up and I believe when the watchers are being initilized on the CRDs

[jpmcb]

Another good data point here is the example / framework for these conversion webhooks also has a a downgrade path: https://github.com/madorn/crd-conversion-webhook/blob/4a8b05246a251f0f68e292e5f9c81a734ff56036/converter/example_converter.go#L39-L77

[stmcginnis]

Definitely need the ability to downgrade versions. Everything looks great to me and tests now cover both scenarios. 👍

[gthao313]

LGTM! Nice work!

[gthao313]

Deployed on my side and very confused why the desired_api_version is v1

INFO apiserver::api::node: Original conversion request, conversion_req: Json(ConversionRequest { kind: "ConversionReview", api_version: "apiextensions.k8s.io/v1", request: Request { uid: "182ee520-7fed-492c-950b-bbc3b066eda7", desired_api_version: "brupop.bottlerocket.aws/v1", objects: [Object {"apiVersion": String("brupop.bottlerocket.aws/v2"), ...] } })

[jpmcb]

desired_api_version: "brupop.bottlerocket.aws/v1"

I believe since the Stored Version" is v2`, when the watcher is created for both versions of the CRD (including v1), the kubernetes API queries the webhook for a conversion from v2 to v1. This isn't an actual object conversion, it's just requesting an in place "translation" from one to the other.

It is confusing why these pop up, but I believe it's a paradigm built into kubernetes and we run into since we are supporting multiple versions of our CRD and isn't something from our controller or our bits.

If you watch the logs on the brupop API servers for a bit, you'll notice that these conversion requests eventually stop. Since there are no v1 resources the watchers don't need to make any more conversions.

I'm good to merge this if you are or we can leave it for others to comment on?

[gthao313]

Yeah. I'm good with your changes! Actually I noticed that even the desired version was v1, but the request never be progressed. thanks! : )