Fix incorrect MTU configurations #5880

hjiajing · 2024-01-15T14:36:20Z

The commit fixes 3 incorrect MTU configurations:

When using the WireGuard encryption mode, the Pod eth0's MTU was not correct. The MTU deducted Geneve overhead because the default tunnel type is Geneve while it should deduct the WireGuard overhead as traffic will be encrypted instead of encapsulated.
When using the GRE tunnel type, the Pod eth0's MTU was not correct. The actual overhead is 14 outer MAC, 20 outer IP, and 8 GRE header (4 standard header + 4 key field), summing up to 42 bytes.
When enabling Wireguard for Multicluster, the MTU of all Pod interfaces and wireguard interface were reduced 130 bytes (50 for geneve + 80 for wireguard), however, cross-cluster traffic sent from Pods were not forwarded by wireguard interface. This is because traffic originated from Pods will be encapsulated on gateway Node, and it's the encapsulated packet which will be encrypted. If the wireguard interface is set with the same MTU as the Pod interface, the encapsulated packet will exceed wireguard interface's MTU.

Fixes #5868
Fixes #5913
Fixes #5914

pkg/agent/config/node_config.go

luolanzone · 2024-01-16T05:46:08Z

@hjiajing the sign-off info is missing. DCO check failed.

tnqn

Please check the usage of the result of the calculation globally and not cause new problems.
If possible, we could add a validation to existing wireguard e2e test to check packet of MTU size not get dropped.
Link to the original issue

tnqn · 2024-01-17T14:30:32Z

pkg/agent/config/node_config.go

+		if nc.TrafficEncryptionMode == TrafficEncryptionModeWireGuard {
+			mtuDeduction = WireGuardOverhead
+		} else if nc.TrafficEncryptionMode == TrafficEncryptionModeIPSec {
+			mtuDeduction = IPSecESPOverhead


The handling will cause some other issues:

The comment of NetworkConfig.MTUDeduction says it doesn't count IPsec and WireGuard overhead.

MTUDeduction only counts IPv4 tunnel overhead, no IPsec and WireGuard overhead.

mc_route_controller.go will deduct the overhead another time:

MTU: controller.nodeConfig.NodeTransportInterfaceMTU - controller.networkConfig.MTUDeduction - config.WireGuardOverhead,

getInterfaceMTU will deduct the overhead another time:

if i.networkConfig.TrafficEncryptionMode == config.TrafficEncryptionModeIPSec { mtu -= config.IPSecESPOverhead }

Thanks for the comment, I removed the return value of CalculateMTUDeduction, when we need to calculate MTUs, the caller just uses networkConfig.MTUDeduction directly. And when the TrafficEncryptionMode is WireGuard, it will deduct the WG overhead.

tnqn

If possible, we could add a validation to existing wireguard e2e test to check packet of MTU size not get dropped.

Link to the original issue

We need both unit tests and e2e tests to prove the fix works as expected and avoid another regression in the future.

tnqn · 2024-01-18T09:44:54Z

pkg/agent/agent.go

+	if i.networkConfig.TrafficEncryptionMode == config.TrafficEncryptionModeWireGuard {
+		mtu -= config.WireGuardOverhead
+	} else {
+		mtu -= i.networkConfig.MTUDeduction
+		if i.networkConfig.TrafficEncryptionMode == config.TrafficEncryptionModeIPSec {
+			mtu -= config.IPSecESPOverhead
+		}


I feel it's still a bit unconsolidated to spread the deduction logic in several places, which is error prone and caused the previous bug. Could we take all cases into consideration in CalculateMTUDeduction and use the result everywhere?
By all cases, I mean wireguard only, wireguard + encap (multi-cluster case), ipsec, overlay.

Sure, I will refactor this part to one MTU calculator for all scenarios.

I added a new variable "EncryptionDeduction" in NetworkConfig, and the MTU will deducts TunnelDeduction and EncryptionDeduction.

Why don't we just calculate a single deduction value?

The TunnelDeduction is also used in calculating multi-cluster Pod MTU, so I divided the deduction into two parts.

Discussed with @hjiajing, he will change to a single deduction value.

hjiajing · 2024-01-21T10:37:49Z

/test-e2e

hjiajing · 2024-01-21T13:04:42Z

/test-e2e

hjiajing · 2024-01-21T16:18:22Z

/test-e2e

luolanzone · 2024-01-22T02:48:20Z

pkg/agent/config/node_config.go

 	// When Multi-cluster Gateway is enabled, we need to reduce MTU for potential cross-cluster traffic.
 	if nc.TrafficEncapMode.SupportsEncap() || nc.EnableMulticlusterGW {
 		if nc.TunnelType == ovsconfig.VXLANTunnel {
-			mtuDeduction = vxlanOverhead
+			nc.MTUDeduction = vxlanOverhead


IPsec work together with those encap tunnel modes, I think this should be "nc.MTUDeduction += vxlanOverhead". cc @xliuxu

I have a question regarding the IPSec with Encap mode and Multicluster enabled, do we verified this kind of traffic before for MC?

+1 to @luolanzone for the IPsec case.

Fixed. changed = to +=.

luolanzone · 2024-01-22T03:16:44Z

test/e2e/framework.go

+	// Example stdout:
+	// 22: vxlan-29acf3@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master ovs-system state UP mode DEFAULT group default
+	// link/ether be:41:93:69:87:02 brd ff:ff:ff:ff:ff:ff link-netns cni-320ae61f-5e51-d123-0169-9f1807390500
+	fields := strings.Fields(strings.Split(strings.TrimSpace(stdout), "\n")[0])


It might be simper to get the MTU from the system file directly: cat /sys/class/net/<interface_name>/mtu

Agreed, done.

hjiajing · 2024-01-22T07:16:04Z

/test-e2e

pkg/agent/config/node_config_test.go

tnqn · 2024-01-22T08:41:25Z

pkg/agent/multicluster/mc_route_controller.go

@@ -129,7 +129,7 @@ func NewMCDefaultRouteController(
 		controller.wireGuardConfig = &config.WireGuardConfig{
 			Port: multiclusterConfig.WireGuard.Port,
 			Name: multiclusterWireGuardInterface,
-			MTU:  controller.nodeConfig.NodeTransportInterfaceMTU - controller.networkConfig.MTUDeduction - config.WireGuardOverhead,
+			MTU:  controller.nodeConfig.NodeTransportInterfaceMTU - controller.networkConfig.MTUDeduction,


Isn't it just controller.networkConfig.InterfaceMTU?

tnqn · 2024-01-22T08:47:08Z

test/e2e/framework.go

@@ -2481,6 +2481,47 @@ func (data *TestData) GetTransportInterface() (string, error) {
 	return "", fmt.Errorf("no interface was assigned with Node IP %s", nodeIP)
 }

+func (data *TestData) GetPodInterfaceName(nodeName string, podName string) (string, error) {


I don't mean checking MTU in e2e. The calculation has already been validated in unit tests, the mistake of calculation can be caught by unit test. What we really is to ensure the calculation can work by testing real traffic, which was not there before.

My suggestion of how to write the e2e:
Just change the existing connectivity test to test the MTU size packet can be forwarded. The test should fail without the fix and succeed with the fix.
For example, in runPingMesh, change the packet size to the MTU-28 and specify -M do to set Don't Frangment bit.

Sure, I will add this part.

E2E test added.

tnqn · 2024-01-22T09:24:59Z

cmd/antrea-agent/agent.go

@@ -205,7 +205,7 @@ func run(o *Options) error {
 		IPsecConfig: config.IPsecConfig{
 			AuthenticationMode: ipsecAuthenticationMode,
 		},
-		EnableMulticlusterGW: enableMulticlusterGW,
+		MulticlusterConfig: o.config.Multicluster,


Do not just replace it without removing the original attribute.

It's inconsistent to use o.config.Multicluster as the source of whether multicluster gateway is enabled. If you take a look at how enableMulticlusterGW is calculated, there is other condition.

I believe we should keep passing enableMulticlusterGW, and add a parsed nc.MulticlusterConfig.TrafficEncryptionMode.

Sorry I removed this line by mistake, added back.

hjiajing · 2024-01-22T17:40:37Z

/test-e2e

luolanzone

Please also check golang-lint failures.

pkg/agent/config/node_config.go

test/e2e/connectivity_test.go

test/e2e/mtu_test.go

hjiajing · 2024-01-23T03:37:43Z

/test-e2e

tnqn · 2024-01-24T17:15:04Z

@antoninbas could you take a look at the PR?

antoninbas

LGTM

antoninbas · 2024-01-24T18:04:48Z

pkg/agent/config/node_config.go

+	}
+	if nc.TrafficEncryptionMode == TrafficEncryptionModeWireGuard {
+		// When WireGuard is enabled, cross-node traffic will only be encrypted, just reduce MTU for encryption.
+		nc.MTUDeduction = WireGuardOverhead


do you want to do 60B when IPv6 is disabled and 80B when IPv6 is enabled?

see https://lists.zx2c4.com/pipermail/wireguard/2017-December/002201.html

thanks for the info, will add it.

tnqn · 2024-01-25T04:13:42Z

/test-e2e

tnqn · 2024-01-25T05:29:17Z

/test-e2e

ci/jenkins/test-vmc.sh

tnqn · 2024-01-25T05:30:49Z

test/e2e/connectivity_test.go

+	if t.Failed() {
+		time.Sleep(15 * time.Minute)
+	}


for testing only

tnqn · 2024-01-25T08:16:24Z

/test-e2e

Signed-off-by: Quan Tian <qtian@vmware.com>

tnqn · 2024-01-25T11:53:11Z

/test-e2e
/test-ipv6-e2e
/test-ipv6-only-e2e
/test-windows-e2e
/test-flexible-ipam-e2e
/test-multicluster-e2e
/test-multicast-e2e

tnqn · 2024-01-25T14:07:24Z

ipv4 e2e failed, but ipsec's MTU may be wrong from the beginning, need more investigation.

It may be related by MTU calculation bug in specific versions, created #5922

tnqn · 2024-01-25T14:20:16Z

/test-conformance
/test-networkpolicy

tnqn · 2024-01-25T15:33:02Z

/test-windows-e2e
/test-multicluster-e2e

tnqn · 2024-01-26T02:30:24Z

/test-multicluster-e2e
/test-windows-containerd-e2e

hjiajing · 2024-01-26T02:46:46Z

/test-multicluster-e2e

The commit fixes 3 incorrect MTU configurations: 1. When using the WireGuard encryption mode, the Pod eth0's MTU was not correct. The MTU deducted Geneve overhead because the default tunnel type is Geneve while it should deduct the WireGuard overhead as traffic will be encrypted instead of encapsulated. 2. When using the GRE tunnel type, the Pod eth0's MTU was not correct. The actual overhead is 14 outer MAC, 20 outer IP, and 8 GRE header (4 standard header + 4 key field), summing up to 42 bytes. 3. When enabling Wireguard for Multicluster, the MTU of all Pod interfaces and wireguard interface were reduced 130 bytes (50 for geneve + 80 for wireguard), however, cross-cluster traffic sent from Pods were not forwarded by wireguard interface. This is because traffic originated from Pods will be encapsulated on gateway Node, and it's the encapsulated packet which will be encrypted. If the wireguard interface is set with the same MTU as the Pod interface, the encapsulated packet will exceed wireguard interface's MTU. Signed-off-by: Jiajing Hu <hjiajing@vmware.com> Signed-off-by: Quan Tian <qtian@vmware.com> Co-authored-by: Quan Tian <qtian@vmware.com>

hjiajing requested a review from antoninbas January 15, 2024 14:36

luolanzone added this to the Antrea v1.15 release milestone Jan 16, 2024

hjiajing linked an issue Jan 16, 2024 that may be closed by this pull request

MTU is wrong for Pod's eth0 interface when using Wireguard #5868

Closed

luolanzone reviewed Jan 16, 2024

View reviewed changes

pkg/agent/config/node_config.go Outdated Show resolved Hide resolved

luolanzone added the action/backport Indicates a PR that requires backports. label Jan 17, 2024

tnqn reviewed Jan 17, 2024

View reviewed changes

hjiajing force-pushed the fix-wg-mtu branch from c928995 to f800105 Compare January 18, 2024 07:11

tnqn reviewed Jan 18, 2024

View reviewed changes

hjiajing force-pushed the fix-wg-mtu branch 3 times, most recently from 9aff2f1 to b2c145b Compare January 21, 2024 10:05

hjiajing force-pushed the fix-wg-mtu branch from b2c145b to 126b14d Compare January 21, 2024 16:18

luolanzone reviewed Jan 22, 2024

View reviewed changes

hjiajing force-pushed the fix-wg-mtu branch 2 times, most recently from c2c3d32 to 0cbb17f Compare January 22, 2024 07:15

tnqn reviewed Jan 22, 2024

View reviewed changes

hjiajing force-pushed the fix-wg-mtu branch 2 times, most recently from d8ce863 to c8bc079 Compare January 22, 2024 17:38

luolanzone reviewed Jan 23, 2024

View reviewed changes

hjiajing force-pushed the fix-wg-mtu branch from c8bc079 to 67eb290 Compare January 23, 2024 03:35

hjiajing force-pushed the fix-wg-mtu branch 2 times, most recently from e751c22 to d8dbf0b Compare January 23, 2024 07:05

antoninbas previously approved these changes Jan 24, 2024

View reviewed changes

tnqn dismissed antoninbas’s stale review via da3c263 January 25, 2024 05:29

tnqn force-pushed the fix-wg-mtu branch from 2631edc to da3c263 Compare January 25, 2024 05:29

tnqn reviewed Jan 25, 2024

View reviewed changes

ci/jenkins/test-vmc.sh Outdated Show resolved Hide resolved

tnqn reviewed Jan 25, 2024

View reviewed changes

test/e2e/connectivity_test.go Outdated

Comment on lines 120 to 122

if t.Failed() {

time.Sleep(15 * time.Minute)

}

Copy link

Member

tnqn Jan 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for testing only

tnqn force-pushed the fix-wg-mtu branch from da3c263 to db88528 Compare January 25, 2024 08:16

tnqn mentioned this pull request Jan 25, 2024

PMTU is wrong when using GRE+IPsec with some Linux kernel versions #5922

Closed

Fix code and test related to MTU validation

00b12d8

Signed-off-by: Quan Tian <qtian@vmware.com>

tnqn force-pushed the fix-wg-mtu branch from db88528 to 00b12d8 Compare January 25, 2024 11:49

tnqn mentioned this pull request Jan 25, 2024

Release 1.14.2 #5892

Merged

tnqn mentioned this pull request Jan 25, 2024

Release 1.15.0 #5921

Merged

antoninbas approved these changes Jan 25, 2024

View reviewed changes

tnqn merged commit ce46eb1 into antrea-io:main Jan 26, 2024
53 of 58 checks passed

tnqn mentioned this pull request Jan 26, 2024

Automated cherry pick of #5880: Fix incorrect MTU configurations (#5880) #5926: Ensure MTU is set correctly when WireGuard interface already #5927

Merged

tnqn mentioned this pull request Jan 26, 2024

e2e: Windows e2e tests failed because toolbox has no windows versions antrea-io/image-utils#26

Closed

tnqn mentioned this pull request Mar 25, 2024

Automated cherry pick of #5880: Fix incorrect MTU configurations (#5880) #5926: Ensure MTU is set correctly when WireGuard interface already #5997: Use 65000 MTU upper bound for interfaces in encap mode #6145

Merged

Fix incorrect MTU configurations #5880

Fix incorrect MTU configurations #5880

Conversation

hjiajing commented Jan 15, 2024 • edited by tnqn Loading

luolanzone commented Jan 16, 2024

tnqn left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tnqn left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hjiajing commented Jan 21, 2024

hjiajing commented Jan 21, 2024

hjiajing commented Jan 21, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hjiajing commented Jan 22, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hjiajing commented Jan 22, 2024

luolanzone left a comment

Choose a reason for hiding this comment

hjiajing commented Jan 23, 2024

tnqn commented Jan 24, 2024

antoninbas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tnqn commented Jan 25, 2024

tnqn commented Jan 25, 2024

Choose a reason for hiding this comment

tnqn commented Jan 25, 2024

tnqn commented Jan 25, 2024

tnqn commented Jan 25, 2024

tnqn commented Jan 25, 2024

tnqn commented Jan 25, 2024

tnqn commented Jan 26, 2024

hjiajing commented Jan 26, 2024

hjiajing commented Jan 15, 2024 •

edited by tnqn

Loading

tnqn left a comment •

edited

Loading

tnqn left a comment •

edited

Loading