Support deploying one FRR container in Kind network #6488

hongliangl · 2024-06-26T08:13:14Z

This commit introduces a new parameter --deploy-external-frr
in ci/kind/kind-setup.sh, enabling to deploy one FRR container in
the Kind cluster network. This serves as the foundation for running
BGPPolicy e2e tests.

ci/kind/test-e2e-kind.sh

ci/kind/kind-setup.sh

build/charts/antrea/values.yaml

ci/kind/test-e2e-kind.sh

antoninbas

How will we ensure that normal Pods used for testing (toolbox, agnhost, nginx, ...) are not scheduled on this extra Node? I guess it will never be Ready and so Pods won't be scheduled there unless they have the appropriate toleration?

Unless I am mistaken, with this change we effectively go from a 3-Node K8s cluster for testing to a 4-Node K8s cluster? This is not an insignificant change as the Github runners have limited resources and an extra Node may put a strain on these resources. Would like @tnqn to review this PR as well.

ci/kind/test-e2e-kind.sh

hongliangl · 2024-06-28T00:34:49Z

How will we ensure that normal Pods used for testing (toolbox, agnhost, nginx, ...) are not scheduled on this extra Node? I guess it will never be Ready and so Pods won't be scheduled there unless they have the appropriate toleration?

Unless I am mistaken, with this change we effectively go from a 3-Node K8s cluster for testing to a 4-Node K8s cluster? This is not an insignificant change as the Github runners have limited resources and an extra Node may put a strain on these resources. Would like @tnqn to review this PR as well.

I think we should only enable this option in the test that enables all features. I am planning to use it to test BGP only (install a host network FRR Pod on the Node without Antrea to mock remote BGP router).

tnqn · 2024-06-28T04:44:14Z

How will we ensure that normal Pods used for testing (toolbox, agnhost, nginx, ...) are not scheduled on this extra Node? I guess it will never be Ready and so Pods won't be scheduled there unless they have the appropriate toleration?
Unless I am mistaken, with this change we effectively go from a 3-Node K8s cluster for testing to a 4-Node K8s cluster? This is not an insignificant change as the Github runners have limited resources and an extra Node may put a strain on these resources. Would like @tnqn to review this PR as well.

I think we should only enable this option in the test that enables all features. I am planning to use it to test BGP only (install a host network FRR Pod on the Node without Antrea to mock remote BGP router).

I have the same concern as @antoninbas. In my understanding the test just needs a BGP router, right? Why it needs a NotReady worker Node to run it, instead of just running the router directly via docker run like the existing "external-server"? we could even add the functionaliy to the existing container. The current approach that sets up another node introduces more code to deal with Node configuration, Pod affinity, overhead to run kubelet, containerd on that node, and the topology may look confusing as NotReady Node normally means something wrong.

hongliangl · 2024-06-28T05:08:48Z

How will we ensure that normal Pods used for testing (toolbox, agnhost, nginx, ...) are not scheduled on this extra Node? I guess it will never be Ready and so Pods won't be scheduled there unless they have the appropriate toleration?
Unless I am mistaken, with this change we effectively go from a 3-Node K8s cluster for testing to a 4-Node K8s cluster? This is not an insignificant change as the Github runners have limited resources and an extra Node may put a strain on these resources. Would like @tnqn to review this PR as well.

I think we should only enable this option in the test that enables all features. I am planning to use it to test BGP only (install a host network FRR Pod on the Node without Antrea to mock remote BGP router).

I have the same concern as @antoninbas. In my understanding the test just needs a BGP router, right? Why it needs a NotReady worker Node to run it, instead of just running the router directly via docker run like the existing "external-server"? we could even add the functionaliy to the existing container. The current approach that sets up another node introduces more code to deal with Node configuration, Pod affinity, overhead to run kubelet, containerd on that node, and the topology may look confusing as NotReady Node normally means something wrong.

Got that. If so, I will try to enhance the current external server to mock BGP server. I think we need to add some framework code to operate the external server, for example:

Install FRR
start FRR BGP
Add BGP peer, and the peer is the k8s Nodes selected by the test BGPPolicy.
Run wget/curl to verify the advertised routes from k8s cluster.

Maybe more details should be considered. How do you think about this proposal? @tnqn @antoninbas

tnqn · 2024-06-28T06:59:46Z

Got that. If so, I will try to enhance the current external server to mock BGP server. I think we need to add some framework code to operate the external server, for example:

Install FRR

start FRR BGP

Add BGP peer, and the peer is the k8s Nodes selected by the test BGPPolicy.

Run wget/curl to verify the advertised routes from k8s cluster.

Maybe more details should be considered. How do you think about this proposal? @tnqn @antoninbas

Not sure what you mean by framework code. In my mind you just need an image for 1, customize docker run command to start BGP for 2 and 3, figure out a way to validate 4 in tests.

hongliangl · 2024-06-28T07:18:26Z

Got that. If so, I will try to enhance the current external server to mock BGP server. I think we need to add some framework code to operate the external server, for example:

Install FRR

start FRR BGP

Add BGP peer, and the peer is the k8s Nodes selected by the test BGPPolicy.

Run wget/curl to verify the advertised routes from k8s cluster.

Maybe more details should be considered. How do you think about this proposal? @tnqn @antoninbas

Not sure what you mean by framework code. In my mind you just need an image for 1, customize docker run command to start BGP for 2 and 3, figure out a way to validate 4 in tests.

I didn't make it clear. I just meant that I may need to add some fundamental functions because I'm not sure if we have some existing convenient functions to configure the external server you added. @tnqn

tnqn · 2024-06-28T07:25:18Z

I didn't make it clear. I just meant that I may need to add some fundamental functions because I'm not sure if we have some existing convenient functions to configure the external server you added. @tnqn

I think there is no problem to have another parameter to deploy a new bgp router (not based on the external server), given they don't have many things in common, and there isn't much code to bring up the external server.

luolanzone

LGTM overall

ci/kind/kind-setup.sh

test/e2e/framework.go

XinShuYang

LGTM

ci/kind/kind-setup.sh

tnqn · 2024-07-05T10:07:13Z

ci/kind/test-e2e-kind.sh

+  if $bgp_policy; then
+    external_frr_cid=$(docker ps -f name="^antrea-external-frr" --format '{{.ID}}')
+    external_frr_ips=$(docker inspect $external_frr_cid -f '{{.NetworkSettings.Networks.kind.IPAddress}},{{.NetworkSettings.Networks.kind.GlobalIPv6Address}}')
+    EXTRA_ARGS="$EXTRA_ARGS --external-frr-cid $external_frr_cid --external-frr-ips $external_frr_ips"


why we need to pass external_frr_cid to test code? I think this would cause the test can only run in kind cluster.

I thought we could run some commands in the FRR container by passing the cid to test code.

I think this would cause the test can only run in kind cluster.

A question: is the external agnhost container only supported in Kind cluster? I thought that it was the same for the external frr container.

I thought we could run some commands in the FRR container by passing the cid to test code.

Is there a public API to access the FRR application, like "/clientip" of agnhost container?

A question: is the external agnhost container only supported in Kind cluster? I thought that it was the same for the external frr container.

The test code doesn't make any assumption on how the external agnhost server is deployed so technically you can specify a VM IP or a public IP. It only needs the server's IP.
By passing the container ID to test code, I assume you would then use it to run some validation, then the code can be only ran in a kind cluster, and can't work with real BGP router.

Is there a public API to access the FRR application, like "/clientip" of agnhost container?

I'm afraid we cannot configure it through network. I found that FRR application doesn't provide any API based TCP/IP. It can be configured through Unix Socket API locally.

By passing the container ID to test code, I assume you would then use it to run some validation, then the code can be only ran in a kind cluster, and can't work with real BGP router.

I use the container ID to configure the FRR application using the command like docker exec -it <container ID> <command>.

luolanzone

LGTM overall, please replace an FRR to 'a FRR' in your PR title and description.

test/e2e/framework.go

This commit introduces a new parameter `--deploy-external-frr` in `ci/kind/kind-setup.sh`, enabling to deploy one FRR container in the Kind cluster network. This serves as the foundation for running BGPPolicy e2e tests. Signed-off-by: Hongliang Liu <lhongliang@vmware.com>

hongliangl · 2024-07-17T03:04:50Z

@tnqn @luolanzone Do we have any more comments about this? If not, could we merge this PR? Thanks

luolanzone

Thanks for the efforts, no more comments from my side, @tnqn could you take another look?

tnqn

LGTM

tnqn · 2024-07-22T12:50:35Z

/skip-all

tnqn · 2024-07-22T12:52:03Z

/skip-all

hongliangl requested review from tnqn, XinShuYang and luolanzone June 26, 2024 08:13

XinShuYang reviewed Jun 26, 2024

View reviewed changes

ci/kind/test-e2e-kind.sh Outdated Show resolved Hide resolved

XinShuYang reviewed Jun 26, 2024

View reviewed changes

ci/kind/kind-setup.sh Outdated Show resolved Hide resolved

hongliangl force-pushed the 20240625-exclude-node-in-ci branch from b776865 to 3e2d7e4 Compare June 26, 2024 09:27

hongliangl requested a review from antoninbas June 27, 2024 00:05

antoninbas reviewed Jun 27, 2024

View reviewed changes

build/charts/antrea/values.yaml Outdated Show resolved Hide resolved

hongliangl force-pushed the 20240625-exclude-node-in-ci branch from 3e2d7e4 to 023b415 Compare June 27, 2024 11:53

hongliangl requested review from antoninbas and XinShuYang June 27, 2024 11:59

hongliangl commented Jun 27, 2024

View reviewed changes

ci/kind/test-e2e-kind.sh Outdated Show resolved Hide resolved

hongliangl commented Jun 27, 2024

View reviewed changes

ci/kind/test-e2e-kind.sh Outdated Show resolved Hide resolved

antoninbas reviewed Jun 27, 2024

View reviewed changes

ci/kind/test-e2e-kind.sh Outdated Show resolved Hide resolved

hongliangl force-pushed the 20240625-exclude-node-in-ci branch from 3c275ca to 0ad4c1d Compare July 1, 2024 01:03

hongliangl changed the title ~~Support deploying an additional Node in Kind without Antrea~~ Support deploying an FRR container in Kind network Jul 1, 2024

hongliangl requested a review from antoninbas July 1, 2024 01:17

hongliangl force-pushed the 20240625-exclude-node-in-ci branch 2 times, most recently from 0fb4110 to 9557b49 Compare July 1, 2024 06:51

luolanzone reviewed Jul 2, 2024

View reviewed changes

ci/kind/kind-setup.sh Show resolved Hide resolved

hongliangl force-pushed the 20240625-exclude-node-in-ci branch from 9557b49 to 059eec6 Compare July 2, 2024 08:27

hongliangl requested a review from luolanzone July 2, 2024 08:28

hongliangl force-pushed the 20240625-exclude-node-in-ci branch from 059eec6 to 7526b58 Compare July 2, 2024 13:55

XinShuYang reviewed Jul 3, 2024

View reviewed changes

test/e2e/framework.go Show resolved Hide resolved

XinShuYang previously approved these changes Jul 3, 2024

View reviewed changes

tnqn reviewed Jul 5, 2024

View reviewed changes

hongliangl dismissed XinShuYang’s stale review via 0efa32d July 8, 2024 06:24

hongliangl force-pushed the 20240625-exclude-node-in-ci branch 3 times, most recently from 12917bb to 7a7a8bc Compare July 12, 2024 03:59

hongliangl requested review from tnqn and XinShuYang July 12, 2024 04:00

hongliangl force-pushed the 20240625-exclude-node-in-ci branch from 7a7a8bc to cbe8956 Compare July 12, 2024 05:46

hongliangl mentioned this pull request Jul 12, 2024

Add e2e tests for BGPPolicy #6523

Merged

hongliangl added this to the Antrea v2.1 release milestone Jul 15, 2024

luolanzone reviewed Jul 16, 2024

View reviewed changes

test/e2e/framework.go Outdated Show resolved Hide resolved

hongliangl force-pushed the 20240625-exclude-node-in-ci branch from 03f7adf to 3e83040 Compare July 16, 2024 03:21

hongliangl changed the title ~~Support deploying an FRR container in Kind network~~ Support deploying one FRR container in Kind network Jul 16, 2024

hongliangl requested a review from luolanzone July 16, 2024 03:21

hongliangl mentioned this pull request Jul 16, 2024

Implement the controller for API BGPPolicy #6203

Merged

7 tasks

luolanzone approved these changes Jul 19, 2024

View reviewed changes

tnqn approved these changes Jul 22, 2024

View reviewed changes

tnqn merged commit 943d9b3 into antrea-io:main Jul 22, 2024
52 of 58 checks passed

hongliangl deleted the 20240625-exclude-node-in-ci branch July 22, 2024 13:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support deploying one FRR container in Kind network #6488

Support deploying one FRR container in Kind network #6488

hongliangl commented Jun 26, 2024 •

edited

Loading

antoninbas left a comment

hongliangl commented Jun 28, 2024 •

edited

Loading

tnqn commented Jun 28, 2024

hongliangl commented Jun 28, 2024 •

edited

Loading

tnqn commented Jun 28, 2024

hongliangl commented Jun 28, 2024

tnqn commented Jun 28, 2024

luolanzone left a comment

XinShuYang left a comment

tnqn Jul 5, 2024

hongliangl Jul 8, 2024

tnqn Jul 8, 2024

hongliangl Jul 8, 2024

luolanzone left a comment

hongliangl commented Jul 17, 2024

luolanzone left a comment

tnqn left a comment

tnqn commented Jul 22, 2024

tnqn commented Jul 22, 2024

Support deploying one FRR container in Kind network #6488

Support deploying one FRR container in Kind network #6488

Conversation

hongliangl commented Jun 26, 2024 • edited Loading

antoninbas left a comment

Choose a reason for hiding this comment

hongliangl commented Jun 28, 2024 • edited Loading

tnqn commented Jun 28, 2024

hongliangl commented Jun 28, 2024 • edited Loading

tnqn commented Jun 28, 2024

hongliangl commented Jun 28, 2024

tnqn commented Jun 28, 2024

luolanzone left a comment

Choose a reason for hiding this comment

XinShuYang left a comment

Choose a reason for hiding this comment

tnqn Jul 5, 2024

Choose a reason for hiding this comment

hongliangl Jul 8, 2024

Choose a reason for hiding this comment

tnqn Jul 8, 2024

Choose a reason for hiding this comment

hongliangl Jul 8, 2024

Choose a reason for hiding this comment

luolanzone left a comment

Choose a reason for hiding this comment

hongliangl commented Jul 17, 2024

luolanzone left a comment

Choose a reason for hiding this comment

tnqn left a comment

Choose a reason for hiding this comment

tnqn commented Jul 22, 2024

tnqn commented Jul 22, 2024

hongliangl commented Jun 26, 2024 •

edited

Loading

hongliangl commented Jun 28, 2024 •

edited

Loading

hongliangl commented Jun 28, 2024 •

edited

Loading