Add script for migrating from other CNI to Antrea #5677

hjiajing · 2023-11-08T02:38:24Z

A new image "antrea-migrator" is responsible for restarting all Pods on each Nodes. It is a DaemonSet that tries to kill sandboxes to restart the Pods in-place. In this way, the cluster could be migrated from old CNIs to Antrea.

luolanzone · 2023-11-08T09:40:22Z

Please update the title and comment: s/scrtip/script/
Link this PR to the issue. You should add more descriptions for this PR to clarify what does this tool do.

hjiajing · 2023-11-08T09:46:21Z

Please update the title and comment: s/scrtip/script/ Link this PR to the issue. You should add more descriptions for this PR to clarify what does this tool do.

Sure. I will add more descriptions for the antctl migrate tool.

build/images/scripts/restart_sandbox

build/yamls/antrea-migrator.yml

docs/migrate-to-antrea.md

pkg/antctl/raw/migrate/convertnetworkpolicy/command.go

luolanzone · 2023-11-13T07:41:44Z

pkg/antctl/raw/migrate/convertnetworkpolicy/command.go

+
+	if len(rule) != 0 {
+		antreaRules = append(antreaRules,
+			v1beta1.Rule{Action: migrate.ToPtr(v1beta1.RuleActionReject)})


I don't get this, why Reject rule is added?

In Calico NP, if one NP is applied to some Pods, then these Pods' ingress and egress traffic will be rejected except Allow fields in that NP, so I added a default Reject rule.

pkg/antctl/raw/migrate/convertnetworkpolicy/command.go

luolanzone · 2023-11-13T08:02:48Z

pkg/client/clientset/versioned/typed/crd/v1alpha1/clusternetworkpolicy.go

@@ -1,4 +1,4 @@
-// Copyright 2021 Antrea Authors
+// Copyright 2023 Antrea Authors


Why is this changed?

I reformated this file accidentally, resulting in the import files being sorted. Then make code-gen command will edit this file every time even if I undo my format operation. Maybe I could edit the year manually.

Not sure what's kind of reformat are you referring to, but I feel it's better to avoid this change if there is no code change on the CRD.

pkg/antctl/raw/migrate/convertnetworkpolicy/command.go

build/images/scripts/migrate_cni

docs/migrate-to-antrea.md

jianjuns · 2024-01-03T22:29:43Z

docs/migrate-to-antrea.md

+
+The migration process is divided into three steps:
+
+1. Clean up old CNI.


the old CNI

"The" Added.

You probably missed this one.

Sorry, thanks for the reminder. Addressed.

docs/migrate-to-antrea.md

tnqn · 2024-01-08T03:48:12Z

build/yamls/antrea-migrator.yml

+  - apiGroups:
+      - ""
+    resources:
+      - pods
+    verbs:
+      - create
+  - apiGroups:
+      - ""
+    resources:
+      - pods/exec
+    verbs:
+      - create
+  - apiGroups:
+      - "apps/v1"
+    resources:
+      - daemonsets
+    verbs:
+      - create


why does it require these permissions?

Removed these rules.

build/images/Dockerfile.build.migrator

build/images/scripts/migrate_cni

tnqn · 2024-01-08T04:24:16Z

docs/migrate-to-antrea.md

+After Antrea is installed in the cluster, the next step is to restart all
+Pods in the cluster in-place by the following command. This step will create
+a DaemonSet in the cluster, which will restart all Pods in the cluster in-place.


After Antrea is up and running, you can now deploy Antrea migrator by the following command. The migrator runs as a DaemonSet, antrea-migrator, in the cluster, which will restart all non hostNetwork Pods in the cluster in-place and perform necessary network resource cleanup.

tnqn · 2024-01-08T04:31:54Z

docs/migrate-to-antrea.md

+network management and IPAM from the old CNI. In order to avoid the Pods
+being rescheduled, we restart all Pods in-place by deploying a DaemonSet
+named `antrea-migrator`, which will run a Pod on each Node. The
+`antrea-migrator` Pod will stop all containerd tasks of Pods with CNI network on
+each Node, and the containerd tasks will be restarted by the containerd
+service. In this way, all Pods in the cluster will not be rescheduled but will
+be restarted in-place.


Suggested change

network management and IPAM from the old CNI. In order to avoid the Pods

being rescheduled, we restart all Pods in-place by deploying a DaemonSet

named `antrea-migrator`, which will run a Pod on each Node. The

`antrea-migrator` Pod will stop all containerd tasks of Pods with CNI network on

each Node, and the containerd tasks will be restarted by the containerd

service. In this way, all Pods in the cluster will not be rescheduled but will

be restarted in-place.

network management and IPAM from the old CNI. In order to avoid the Pods

being rescheduled and minimize service downtime, the migrator restarts

all non-hostNetwork Pods in-place by restarting their sandbox container.

Therefore, it's expected to see these Pods' `RESTARTS` being increased

by 1 like below:

docs/migrate-to-antrea.md

tnqn · 2024-01-08T04:37:56Z

hack/release/prepare-assets.sh

@@ -107,6 +107,8 @@ export IMG_TAG=$VERSION
 export IMG_NAME=projects.registry.vmware.com/antrea/antrea-ubuntu
 ./hack/generate-standard-manifests.sh --mode release --out "$OUTPUT_DIR"

+sed "s|antrea\/antrea-migrator:latest|$(echo $IMG_NAME | sed "s/ubuntu/migrator/"):$VERSION|g" ./build/yamls/antrea-migrator.yml > "$OUTPUT_DIR"/antrea-migrator.yml


Removed this line since we do not need to add it to release asserts.

tnqn · 2024-01-08T04:38:44Z

docs/migrate-to-antrea.md

+a DaemonSet in the cluster, which will restart all Pods in the cluster in-place.
+
+```bash
+$ kubectl apply -f https://github.com/antrea-io/antrea/releases/download/v1.15.0/antrea-migrator.yml


Suggested change

$ kubectl apply -f https://github.com/antrea-io/antrea/releases/download/v1.15.0/antrea-migrator.yml

$ kubectl apply -f https://github.com/antrea-io/antrea/main/build/yamls/antrea-migrator.yml

Like antrea-aks-node-init.yml, the resource is version-generic and case-specific, I think we don't need to add it to release assets.

Changed to main/build/yamls

luolanzone

A few nits

luolanzone · 2024-01-08T03:56:39Z

build/images/Dockerfile.build.migrator

@@ -0,0 +1,33 @@
+# Copyright 2023 Antrea Authors


Changed to 2024.

build/images/scripts/migrate_cni

luolanzone · 2024-01-08T04:00:39Z

docs/migrate-to-antrea.md

+
+After Antrea is installed in the cluster, the next step is to restart all
+Pods in the cluster in-place by the following command. This step will create
+a DaemonSet in the cluster, which will restart all Pods in the cluster in-place.


Maybe highlight antrea-migrator is supported since v1.15.0.

Done. A description of the version is added in front of this doc.

luolanzone · 2024-01-08T06:05:49Z

hack/release/prepare-assets.sh

@@ -107,6 +107,8 @@ export IMG_TAG=$VERSION
 export IMG_NAME=projects.registry.vmware.com/antrea/antrea-ubuntu
 ./hack/generate-standard-manifests.sh --mode release --out "$OUTPUT_DIR"

+sed "s|antrea\/antrea-migrator:latest|$(echo $IMG_NAME | sed "s/ubuntu/migrator/"):$VERSION|g" ./build/yamls/antrea-migrator.yml > "$OUTPUT_DIR"/antrea-migrator.yml


Could you use "projects.registry.vmware.com/antrea/antrea-migrator:$VERSION" directly to replace antrea-migrator:latest? There is an on-going change #5794 to split agent and controller images. Maybe better to have your own image string in case there is conflict.

Removed this line since we do not need to add it to release asserts.

tnqn

Overall LGTM

build/yamls/antrea-migrator.yml

docs/migrate-to-antrea.md

jianjuns · 2024-01-09T18:24:54Z

docs/migrate-to-antrea.md

+migrate-example-6d6b97f96b-jpflg   1/1     Running   1 (23s ago)   2m5s   10.10.1.5   test-worker   <none>           <none>
+```
+
+When we meet the condition that all Pods of antrea-migrator are in `Running`


antrea-migrator -> antrea-migrator

Please change all occasions.

Fixed, as well as the the other occasions.

jianjuns · 2024-01-09T18:25:40Z

docs/migrate-to-antrea.md

+The reason for restarting all Pods is that Antrea needs to take over the
+network management and IPAM from the old CNI. In order to avoid the Pods
+being rescheduled and minimize service downtime, the migrator restarts
+all non-hostNetwork Pods in-place by restarting their sandbox container.


container -> containers

jianjuns · 2024-01-09T18:27:20Z

docs/migrate-to-antrea.md

+migrate-example-6d6b97f96b-jpflg   1/1     Running   1 (23s ago)   2m5s   10.10.1.5   test-worker   <none>           <none>
+```
+
+When we meet the condition that all Pods of antrea-migrator are in `Running`


"When the antrea-migrator Pods on all Nodes are"?

jianjuns · 2024-01-09T18:27:31Z

docs/migrate-to-antrea.md

+
+When we meet the condition that all Pods of antrea-migrator are in `Running`
+state, the migration process is completed. You can then remove the antrea-migrator
+DaemonSet safely by the following command:


luolanzone

two nits.
Btw, your DCO check failed, please check if your sign-off info is missing.

docs/migrate-to-antrea.md

tnqn

LGTM

luolanzone

LGTM, one nit.

docs/migrate-to-antrea.md

tnqn · 2024-01-12T04:16:49Z

@jianjuns @antoninbas let me know if you will take another look, thanks.

jianjuns · 2024-01-12T17:23:12Z

I have no extra comment.

antoninbas

couple nits, otherwise lgtm

antoninbas · 2024-01-12T17:58:43Z

build/yamls/antrea-migrator.yml

+          command:
+            - "sleep"
+            - "infinity"


nit: it's nicer to default to the pause command in your container image
the pause command handles signals well, unlike sleep. Deleting Pods where a container uses sleep infinity can take 30s (default grace period IIRC).
see https://github.com/antrea-io/image-utils/blob/25f87067c80ab22c0b8574e1bb13cd859f669707/images/toolbox/Dockerfile#L60-L62

/pause is added to the image and the command of the YAML file is removed.

antoninbas · 2024-01-12T17:59:25Z

docs/migrate-to-antrea.md

+NOTE: The following is a reference list of CNIs and versions we have already
+verified the migration process. CNIs and versions that are not listed here


The following is a reference list of CNIs and versions for which we have verified the migration process.

antoninbas · 2024-01-12T18:00:48Z

docs/migrate-to-antrea.md

+network management and IPAM from the old CNI. In order to avoid the Pods
+being rescheduled and minimize service downtime, the migrator restarts
+all non-hostNetwork Pods in-place by restarting their sandbox containers.
+Therefore, it's expected to see these Pods' `RESTARTS` being increased


it's expected to see the RESTARTS count for these Pods being increased

Add a YAML file antrea-migrator.yml to migrate clusters with other CNIs to Antrea. It will restart all Pods in-place. A new image "antrea-migrator" is responsible for restarting all Pods on each Nodes. It is a DaemonSet that tries to kill sandboxes to restart the Pods in-place. Signed-off-by: hjiajing <hjiajing@vmware.com>

antoninbas · 2024-01-16T17:44:04Z

/skip-all

hjiajing force-pushed the antrea-migration branch 4 times, most recently from 294746a to 18d73e3 Compare November 8, 2023 06:29

luolanzone added the action/release-note Indicates a PR that should be included in release notes. label Nov 8, 2023

luolanzone added this to the Antrea v1.15 release milestone Nov 8, 2023

hjiajing force-pushed the antrea-migration branch from 18d73e3 to 97cb804 Compare November 8, 2023 09:17

hjiajing force-pushed the antrea-migration branch from 97cb804 to 3f8c169 Compare November 8, 2023 09:43

hjiajing linked an issue Nov 8, 2023 that may be closed by this pull request

A proposal for CNI migration from Calico to Antrea #5578

Closed

hjiajing changed the title ~~[WIP] Add scritp for migrating from other CNI to Antrea~~ [WIP] Add script for migrating from other CNI to Antrea Nov 8, 2023

hjiajing force-pushed the antrea-migration branch 9 times, most recently from fbc3a98 to 6356dbe Compare November 9, 2023 08:30

hjiajing changed the title ~~[WIP] Add script for migrating from other CNI to Antrea~~ Add script for migrating from other CNI to Antrea Nov 10, 2023

hjiajing force-pushed the antrea-migration branch 2 times, most recently from ccd7f28 to b288390 Compare November 12, 2023 04:43

luolanzone reviewed Nov 13, 2023

View reviewed changes

hjiajing force-pushed the antrea-migration branch 4 times, most recently from 904508a to 370b89c Compare November 15, 2023 06:27

hjiajing force-pushed the antrea-migration branch from 21343f2 to 706c955 Compare January 3, 2024 03:12

jianjuns reviewed Jan 3, 2024

View reviewed changes

hjiajing force-pushed the antrea-migration branch 2 times, most recently from 7d5bc4e to 266531a Compare January 5, 2024 02:15

tnqn reviewed Jan 8, 2024

View reviewed changes

luolanzone reviewed Jan 8, 2024

View reviewed changes

hjiajing force-pushed the antrea-migration branch from 266531a to 0fcc98b Compare January 9, 2024 04:37

tnqn reviewed Jan 9, 2024

View reviewed changes

build/yamls/antrea-migrator.yml Outdated Show resolved Hide resolved

docs/migrate-to-antrea.md Outdated Show resolved Hide resolved

docs/migrate-to-antrea.md Outdated Show resolved Hide resolved

hjiajing force-pushed the antrea-migration branch from 0fcc98b to 0561773 Compare January 9, 2024 15:47

jianjuns reviewed Jan 9, 2024

View reviewed changes

luolanzone reviewed Jan 10, 2024

View reviewed changes

docs/migrate-to-antrea.md Outdated Show resolved Hide resolved

docs/migrate-to-antrea.md Show resolved Hide resolved

hjiajing force-pushed the antrea-migration branch 2 times, most recently from 9c8fd50 to 19c95cd Compare January 10, 2024 03:21

tnqn reviewed Jan 10, 2024

View reviewed changes

docs/migrate-to-antrea.md Outdated Show resolved Hide resolved

hjiajing force-pushed the antrea-migration branch from 19c95cd to f5bcc9c Compare January 10, 2024 06:57

tnqn previously approved these changes Jan 11, 2024

View reviewed changes

luolanzone previously approved these changes Jan 11, 2024

View reviewed changes

docs/migrate-to-antrea.md Outdated Show resolved Hide resolved

hjiajing dismissed stale reviews from luolanzone and tnqn via 4f106a5 January 11, 2024 08:30

hjiajing force-pushed the antrea-migration branch from f5bcc9c to 4f106a5 Compare January 11, 2024 08:30

luolanzone previously approved these changes Jan 12, 2024

View reviewed changes

antoninbas reviewed Jan 12, 2024

View reviewed changes

hjiajing dismissed luolanzone’s stale review via 36689d9 January 15, 2024 03:02

hjiajing force-pushed the antrea-migration branch from 4f106a5 to 36689d9 Compare January 15, 2024 03:02

antoninbas approved these changes Jan 16, 2024

View reviewed changes

antoninbas merged commit e469bd9 into antrea-io:main Jan 16, 2024
48 of 54 checks passed

		@@ -1,4 +1,4 @@
		// Copyright 2021 Antrea Authors
		// Copyright 2023 Antrea Authors


		The migration process is divided into three steps:

		1. Clean up old CNI.

	$ kubectl apply -f https://github.com/antrea-io/antrea/releases/download/v1.15.0/antrea-migrator.yml
	$ kubectl apply -f https://github.com/antrea-io/antrea/main/build/yamls/antrea-migrator.yml

		NOTE: The following is a reference list of CNIs and versions we have already
		verified the migration process. CNIs and versions that are not listed here

Add script for migrating from other CNI to Antrea #5677

Add script for migrating from other CNI to Antrea #5677

Conversation

hjiajing commented Nov 8, 2023 • edited Loading

luolanzone commented Nov 8, 2023

hjiajing commented Nov 8, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tnqn Jan 8, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

luolanzone left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tnqn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

luolanzone left a comment

Choose a reason for hiding this comment

tnqn left a comment

Choose a reason for hiding this comment

luolanzone left a comment

Choose a reason for hiding this comment

tnqn commented Jan 12, 2024

jianjuns commented Jan 12, 2024

antoninbas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

antoninbas commented Jan 16, 2024

hjiajing commented Nov 8, 2023 •

edited

Loading

tnqn Jan 8, 2024 •

edited

Loading