Use a Node's primary NIC as the secondary OVS bridge physical interface #6108

aroradaman · 2024-03-14T13:28:39Z

Fixes: #5735

aroradaman · 2024-03-16T06:13:50Z

/cc @jianjuns

pkg/agent/secondarynetwork/init.go

cmd/antrea-agent/agent.go

pkg/agent/secondarynetwork/init.go

jianjuns · 2024-03-26T22:14:56Z

pkg/agent/secondarynetwork/init.go

 	if err != nil {
 		return err
 	}

+	phyInterfaces := make([]string, len(secNetconfig.OVSBridges[0].PhysicalInterfaces))


We can move these two lines to be inside the if len(secNetconfig.OVSBridges[0].PhysicalInterfaces) == 1 block? Here we can do phyInterfaces := secNetconfig.OVSBridges[0].PhysicalInterfaces

So secNetConfig is passed as a pointer, and I'm updating the name of the interface eth0 -> eth0~ just before calling connectPhyInterfacesToOVSBridge(phyInterfaces).
phyInterfaces := secNetconfig.OVSBridges[0].PhysicalInterfaces this would create a shallow copy and renaming interface name in phyInterfaces would mutate the original config struct, right?

cmd/antrea-agent/agent.go

pkg/agent/util/net_linux.go

jianjuns · 2024-03-26T23:08:47Z

pkg/agent/agent_linux.go

-			klog.ErrorS(err, "Configure route to uplink interface failed", "uplink", uplinkName)
-		}
-	}
+	util.RestorePhyInterfaceConfiguration(i.ovsBridge, i.nodeConfig.UplinkNetConfig)


I feel it may be safer to follow your new way of getting the interface config again from the system, so even the config (IP, routes, etc.) have been changed after the interface is moved the bridge, we can restore the changes. What you think? @gran-vmv : thoughts?
It is good to check if some fields in nodeConfig.UplinkNetConfig can be removed, if we just re-fetch them from the system and they are not used anywhere else.

pkg/agent/secondarynetwork/init.go

jianjuns · 2024-03-27T18:54:18Z

pkg/agent/secondarynetwork/init.go

+	copy(phyInterfaces, secNetconfig.OVSBridges[0].PhysicalInterfaces)
+	if len(secNetconfig.OVSBridges[0].PhysicalInterfaces) == 1 {
+		externalIDs := map[string]interface{}{
+			interfacestore.AntreaInterfaceTypeKey: interfacestore.AntreaUplink,


Ok. Fine to leave it in the caller then.

jianjuns · 2024-03-27T18:59:20Z

pkg/agent/agent_linux.go

-			klog.ErrorS(err, "Configure route to uplink interface failed", "uplink", uplinkName)
-		}
-	}
+	util.RestoreHostInterfaceConfiguration(i.ovsBridge, i.nodeConfig.UplinkNetConfig)


Copy back my earlier comments that got resolved after code update:

I feel it may be safer to follow your new way of getting the interface config again from the system, so even the config (IP, routes, etc.) have been changed after the interface is moved the bridge, we can restore the changes. What you think? @gran-vmv : thoughts?
It is good to check if some fields in nodeConfig.UplinkNetConfig can be removed, if we just re-fetch them from the system and they are not used anywhere else.

Seems @gran-vmv is taking vacation now. @wenyingd : do you have any comment?

I f my understanding is correct, your point is when trying to restore the configurations, the steps is like this, 1) read configurations (Name, MAC, IP, Route) from the OVS internal port (host Interface), 2) remove OVS ports including both host Interface and uplink, 3) move the configurations read in step 1) to the uplink ?

This solution sounds good to me, the advantage is it can recover the runtime configurations which are updated in the period since the uplink was moved to OVS, which may not exist in the memory. An precondition is this restore method must be called after the uplink was successfully moved to OVS, but not be used a rollback during the period when moving the uplink to OVS.

Yes, I think IP and route are more important.

An precondition is this restore method must be called after the uplink was successfully moved to OVS, but not be used a rollback during the period when moving the uplink to OVS.

This is a good point. @aroradaman : could you check we can run into this case - restoration after host interface connection failure, that all host interface configurations are moved to the bridge port?

wenyingd · 2024-03-29T10:31:09Z

pkg/agent/util/net_linux.go

+	if err := RenameInterface(linkConfig.Name, bridgedName); err != nil {
+		return "", false, err
+	}
+	if _, err := bridge.CreateInternalPort(linkConfig.Name, int32(linkConfig.OFPort), linkConfig.MAC.String(), externalIDs); err != nil {


Here linkConfig.OFPort is supposed to be prepared for the uplink not the host interface (internal port), it seems a misuse?

Good catch.

I'm passing the following config to PrepareHostInterfaceConnection from ConnectUplinkToOVSBridge now.

&config.AdapterNetConfig{ Name: uplinkNetConfig.Name, OFPort: i.nodeConfig.HostInterfaceOFPort, Index: uplinkNetConfig.Index, MAC: uplinkNetConfig.MAC, IPs: uplinkNetConfig.IPs, Routes: uplinkNetConfig.Routes, }

wenyingd · 2024-03-29T10:56:49Z

pkg/agent/agent_linux.go

-			klog.ErrorS(err, "Configure route to uplink interface failed", "uplink", uplinkName)
-		}
-	}
+	util.RestoreHostInterfaceConfiguration(i.ovsBridge, i.nodeConfig.UplinkNetConfig)


I f my understanding is correct, your point is when trying to restore the configurations, the steps is like this, 1) read configurations (Name, MAC, IP, Route) from the OVS internal port (host Interface), 2) remove OVS ports including both host Interface and uplink, 3) move the configurations read in step 1) to the uplink ?

This solution sounds good to me, the advantage is it can recover the runtime configurations which are updated in the period since the uplink was moved to OVS, which may not exist in the memory. An precondition is this restore method must be called after the uplink was successfully moved to OVS, but not be used a rollback during the period when moving the uplink to OVS.

pkg/agent/agent_linux.go

jianjuns · 2024-04-01T18:04:45Z

pkg/agent/util/net_linux.go

+		if err := DeleteOVSPort(brName, linkConfig.Name); err != nil {
+			klog.ErrorS(err, "Delete OVS port failed", "port", linkConfig.Name)
+		}
+		if err := DeleteOVSPort(brName, bridgedName); err != nil {


To Wenying's point, if the uplink port is not created, which means the physical interface has not been moved to the bridge, we should not restore the IPs & routes from the OVS internal port to the interface?

And for all these operations, we should just ignore the not-exist error, as the func can be called after a partial failure happened at interface moving, and so possible the interface is not moved and the OVS port is not created.

Added a check to exit early, if the bridged port doesn't exists (eth0~).

And for all these operations, we should just ignore the not-exist error

DeleteOVSPort runs the del command with --if-exists flag, so those errors will be ignored. And also the first condition would now prevent this.

pkg/agent/secondarynetwork/init.go

jianjuns · 2024-04-03T18:23:11Z

pkg/agent/secondarynetwork/init.go

+	if len(secNetConfig.OVSBridges[0].PhysicalInterfaces) == 1 {
+		iface, ifaceIPs, ifaceRoutes, err := util.GetInterfaceConfig(secNetConfig.OVSBridges[0].PhysicalInterfaces[0])
+		if err != nil {
+			klog.ErrorS(err, "failed to restore ovs bridge")


I think we should log the exact error: klog.ErrorS(err, "Failed to get interface config", "interface", secNetConfig.OVSBridges[0].PhysicalInterfaces[0]).

Can we know If the interface does not exist, and not log that? This is not very important.

pkg/agent/util/net_linux.go

pkg/agent/agent_linux.go

pkg/agent/util/net_linux.go

jianjuns · 2024-04-04T18:34:18Z

pkg/agent/util/net_linux.go

+
+// RestoreHostInterfaceConfiguration restore the configuration from bridge back to host interface, revering the
+// actions taken in PrepareHostInterfaceConnection.
+func RestoreHostInterfaceConfiguration(


I re-checked the interface move and restore code paths, and I see the following steps:

Move host interface bridge: a1. rename host interface a2. create internal port a3. move config to internal port a4. move host interface to bridge

Restore host interface: b1. get internal port config b2. delete internal port b3. delete host interface bridge port b4. rename host interface b5. restore host interface config

If we failed at a2 earlier, then the code will not go to b3.
So, I am thinking about the following changes:

even b1 fails and b2 fails because of port does not exist, we should still do b3 and b4.

we should also ignore port not exist for b3

probably move b1 into RestoreHostInterfaceConfiguration; but we can pass the saved IPs&routes to the func if you want to fall back the saved state if GetInterfaceConfig fails (in my mind this is not very necessary and we can just rely on GetInterfaceConfig).

if IPs or routes are nil, we skip restoring them in b5.

Let me know what you think.

How about we execute the steps of RestoreHostInterfaceConfiguration in reverse of the order in which they were executed in PrepareHostInterfaceConnection? And we only execute steps when required:

we can call delete-port (both internal and host) with --if-exists

move ips if required (nil config means ips are already present with the interface, right?)

rename interface if both eth0~ exists and eth0 doesn't exist.

(this is similar to what you suggested above, but following the reverse order of execution)

Init()

PrepareHostInterfaceConnection (eth0 idx:3)

a1. Rename (eth0 idx:3) -> (eth0~ idx:3)

a2. Create internal (eth0 idx:50)

a3. MoveConfig (eth0 idx:50)

a4. CreateUplinkPort(eth0~ idx:3)

Restore()

RestoreHostInterfaceConfiguration (eth0 idx:50)

b1. Get config from (eth0 idx:50)
pre req: to get the config

b2. Delete (eth0~ idx:3)
(reverse of a4, if port doesn't exist)

b3. MoveConfig (eth0~ idx:3)
(reverse of a3, if config is non-empty)

b4. Delete Internal Port (eth0 idx:50)
(reverse of a2, if port doesn't exist)

b5. Rename (eth0~ idx:3) -> (eth0 idx:3)
(reverse of a1)

Souds good to me.

I have used the original order only, somehow reverse order was not working.
(Have added the checks before performing any action.)

jianjuns · 2024-04-04T18:47:05Z

pkg/agent/agent_linux.go

@@ -166,30 +166,6 @@ func (i *Initializer) saveHostRoutes() error {
 	return nil


I feel we can remove saveHostRoutes and change prepareOVSBridgeForK8sNode to use GetInterfaceConfig. Could you check?

@wenyingd : I saw there is a difference between GetAllIPNetsByName and GetIPNetsByLink called by GetInterfaceConfig that the former ignores link local addresses. Should we always ignore link local addresses? Could/should we change GetIPNetsByLink to do that too?

I saw GetInterfaceConfig is used mainly by Egress. @tnqn @xliuxu : do you think we can unify these two funcs: either both ignore or both include link local addresses.

I feel we can remove saveHostRoutes and change prepareOVSBridgeForK8sNode to use GetInterfaceConfig. Could you check?

It may not work for some corner cases if the uplink is configured by DHCP, and static routes exist on the host before attaching it to OVS. If the NIC is configured with DHCP, the routes on the host interface are configured from DHCP responses, then the static routes are possibly lost if we don't do it in the program.

Should we always ignore link local addresses? Could/should we change GetIPNetsByLink to do that too?

Having checked the code, it sounds good to me that ignoring the link-local addresses in GetIPNetsByLink.

I went thought relevant functions today and agree they are kind of messy. I'm working on a patch trying to sort out these functions and moving business specific code out of util package.

Yes, that is true. For this PR, do you agree to change GetIPNetsByLink to ignore link local addresses? We can leave refactoring (like removing GetAllIPNetsByName or not) to you.

GetInterfaceConfig is used by ExternalNode feature, not related to Egress.
Actually both GetIPNetsByLink and GetAllIPNetsByName are used to get the IPs that will be moved to OVS bridge port, one for K8s bridging mode, one for ExternalNode. Since they are essentially the same, I don't think there is a problem to make both ignore link local addresses.

Ok. then let us change GetIPNetsByLink to ignore link local addresses. @aroradaman

jianjuns · 2024-04-08T20:41:14Z

pkg/agent/agent_linux.go

-			klog.ErrorS(err, "Configure route to uplink interface failed", "uplink", uplinkName)
-		}
-	}
+	util.RestoreHostInterfaceConfiguration(i.ovsBridge, i.nodeConfig.UplinkNetConfig.Name)


I guess we can skip setting link index, IP, routes in prepareOVSBridgeForK8sNode.

pkg/agent/util/net_linux.go

jianjuns · 2024-04-08T21:26:34Z

pkg/agent/agent_linux.go

@@ -166,30 +166,6 @@ func (i *Initializer) saveHostRoutes() error {
 	return nil


I saw GetInterfaceConfig is used mainly by Egress. @tnqn @xliuxu : do you think we can unify these two funcs: either both ignore or both include link local addresses.

aroradaman · 2024-04-15T18:37:24Z

@jianjuns I'm facing a weird problem.
EnsureIPv6EnabledOnInterface succeeds but I end up with IPv6 disabled on the internal port and I ultimately end up losing the IPv6 address. I tired to log the value of sysctl periodically in a go-routine and noticed that the path /proc/sys/netipv6/conf/eth0/disable_ipv6 disappears for a few seconds and then magically appears back with the default value of 1.

I0415 18:17:43.764311       1 server.go:524] "CmdAdd for container succeeded" container="6ed33ecc46e5ba305c51d99d5211ac00f9e58d25e497312b0d0be95c8247ea51"

--->
I0415 18:17:43.829266       1 net_linux.go:477] "sysctl" path="ipv6/conf/eth0/disable_ipv6" value=0
I0415 18:17:43.829292       1 net_linux.go:482] "sysctl" path="ipv6/conf/eth0~/disable_ipv6" value=0
<---

I0415 18:17:44.506174       1 allocator.go:154] "IP Pool update succeeded" pool="secnet-ipv4-1" allocation={"ipAddresses":[{"ipAddress":"148.14.24.2","phase":"Allocated","owner":{"pod":{"name":"vlan-pod2","namespace":"testvlannetwork-qprush5v","containerID":"6ed33ecc46e5ba305c51d99d5211ac00f9e58d25e497312b0d0be95c8247ea51","ifName":"eth1"}}}],"usage":{"total":253,"used":1}}
I0415 18:17:44.512202       1 pod_configuration.go:260] "Configured container interface" Pod="testvlannetwork-qprush5v/vlan-pod2" container="6ed33ecc46e5ba305c51d99d5211ac00f9e58d25e497312b0d0be95c8247ea51" interface="eth1" hostInterface="vlan-pod-725b29"
I0415 18:17:44.519200       1 allocator.go:154] "IP Pool update succeeded" pool="secnet-ipv4-1" allocation={"ipAddresses":[{"ipAddress":"148.14.24.2","phase":"Allocated","owner":{"pod":{"name":"vlan-pod2","namespace":"testvlannetwork-qprush5v","containerID":"6ed33ecc46e5ba305c51d99d5211ac00f9e58d25e497312b0d0be95c8247ea51","ifName":"eth1"}}},{"ipAddress":"148.14.24.3","phase":"Allocated","owner":{"pod":{"name":"vlan-pod1","namespace":"testvlannetwork-qprush5v","containerID":"7c79866797a73d1e98b9018e03524c348bae65c82e4b7bb23ffdb7d56659c5d8","ifName":"eth1"}}}],"usage":{"total":253,"used":2}}
I0415 18:17:44.524549       1 pod_configuration.go:260] "Configured container interface" Pod="testvlannetwork-qprush5v/vlan-pod1" container="7c79866797a73d1e98b9018e03524c348bae65c82e4b7bb23ffdb7d56659c5d8" interface="eth1" hostInterface="vlan-pod-39ea04"
I0415 18:17:44.530864       1 allocator.go:154] "IP Pool update succeeded" pool="secnet-ipv4-3" allocation={"ipAddresses":[{"ipAddress":"148.14.26.2","phase":"Allocated","owner":{"pod":{"name":"vlan-pod2","namespace":"testvlannetwork-qprush5v","containerID":"6ed33ecc46e5ba305c51d99d5211ac00f9e58d25e497312b0d0be95c8247ea51","ifName":"eth2"}}}],"usage":{"total":253,"used":1}}
I0415 18:17:44.539065       1 allocator.go:154] "IP Pool update succeeded" pool="secnet-ipv4-2" allocation={"ipAddresses":[{"ipAddress":"148.14.25.111","phase":"Allocated","owner":{"pod":{"name":"vlan-pod3","namespace":"testvlannetwork-qprush5v","containerID":"15d503003f071d1805251ad02118496fd13aeb3c01c31b9c1be4fa06b2a6200e","ifName":"eth1"}}},{"ipAddress":"148.14.25.112","phase":"Allocated","owner":{"pod":{"name":"vlan-pod1","namespace":"testvlannetwork-qprush5v","containerID":"7c79866797a73d1e98b9018e03524c348bae65c82e4b7bb23ffdb7d56659c5d8","ifName":"eth2"}}}],"usage":{"total":13,"used":2}}
I0415 18:17:44.539751       1 allocator.go:154] "IP Pool update succeeded" pool="secnet-ipv6-3" allocation={"ipAddresses":[{"ipAddress":"10:2400::2","phase":"Allocated","owner":{"pod":{"name":"vlan-pod2","namespace":"testvlannetwork-qprush5v","containerID":"6ed33ecc46e5ba305c51d99d5211ac00f9e58d25e497312b0d0be95c8247ea51","ifName":"eth2"}}}],"usage":{"total":65535,"used":1}}
I0415 18:17:44.546449       1 pod_configuration.go:260] "Configured container interface" Pod="testvlannetwork-qprush5v/vlan-pod1" container="7c79866797a73d1e98b9018e03524c348bae65c82e4b7bb23ffdb7d56659c5d8" interface="eth2" hostInterface="vlan-pod-4f9443"
I0415 18:17:44.584129       1 server.go:564] "Received CmdDel request" request="cni_args:{container_id:\"5765fd5bf0b10e5ac209197afc1d77f0da632debf11db685348f7ad5b0133593\" netns:\"/var/run/netns/cni-e8b97ba8-09ee-0705-b75d-a90ff851c7c9\" ifname:\"eth0\" args:\"K8S_POD_NAME=coredns-76f75df574-mdjs4;K8S_POD_INFRA_CONTAINER_ID=5765fd5bf0b10e5ac209197afc1d77f0da632debf11db685348f7ad5b0133593;K8S_POD_UID=ff9296c5-873c-498f-a4e7-feb67b420e88;IgnoreUnknown=1;K8S_POD_NAMESPACE=kube-system\" path:\"/opt/cni/bin\" network_configuration:\"{\\\"cniVersion\\\":\\\"0.3.0\\\",\\\"ipam\\\":{\\\"type\\\":\\\"host-local\\\"},\\\"name\\\":\\\"antrea\\\",\\\"type\\\":\\\"antrea\\\"}\"}"
I0415 18:17:44.588840       1 pod_configuration.go:574] "Deleted container OVS port" container="5765fd5bf0b10e5ac209197afc1d77f0da632debf11db685348f7ad5b0133593" interface="coredns--4a2562"
I0415 18:17:44.643405       1 server.go:550] "Deleted interfaces for container" container="5765fd5bf0b10e5ac209197afc1d77f0da632debf11db685348f7ad5b0133593"
I0415 18:17:44.647260       1 server.go:558] "CmdDel for container succeeded" container="5765fd5bf0b10e5ac209197afc1d77f0da632debf11db685348f7ad5b0133593"
I0415 18:17:44.668347       1 server.go:564] "Received CmdDel request" request="cni_args:{container_id:\"e081278ff1c222dbd1e8abc119048e56e02d6c6347c993b21116549894e7813b\" ifname:\"eth0\" args:\"K8S_POD_UID=ff9296c5-873c-498f-a4e7-feb67b420e88;IgnoreUnknown=1;K8S_POD_NAMESPACE=kube-system;K8S_POD_NAME=coredns-76f75df574-mdjs4;K8S_POD_INFRA_CONTAINER_ID=e081278ff1c222dbd1e8abc119048e56e02d6c6347c993b21116549894e7813b\" path:\"/opt/cni/bin\" network_configuration:\"{\\\"cniVersion\\\":\\\"0.3.0\\\",\\\"ipam\\\":{\\\"type\\\":\\\"host-local\\\"},\\\"name\\\":\\\"antrea\\\",\\\"type\\\":\\\"antrea\\\"}\"}"
I0415 18:17:44.668519       1 server.go:550] "Deleted interfaces for container" container="e081278ff1c222dbd1e8abc119048e56e02d6c6347c993b21116549894e7813b"
I0415 18:17:44.672504       1 server.go:558] "CmdDel for container succeeded" container="e081278ff1c222dbd1e8abc119048e56e02d6c6347c993b21116549894e7813b"
E0415 18:17:44.830058       1 net_linux.go:475] "failed to get sysctl" path="ipv6/conf/eth0/disable_ipv6"

--->
E0415 18:31:00.538942       1 net_linux.go:475] "failed to get sysctl" path="ipv6/conf/eth0/disable_ipv6"
I0415 18:17:44.830462       1 net_linux.go:477] "sysctl" path="ipv6/conf/eth0/disable_ipv6" value=-1
I0415 18:17:44.830505       1 net_linux.go:482] "sysctl" path="ipv6/conf/eth0~/disable_ipv6" value=0
<---

I0415 18:31:23.964727       1 allocator.go:250] "IP Pool update succeeded" pool="secnet-ipv4-1" allocation={"usage":{"total":253,"used":0}}
I0415 18:31:23.965513       1 pod_configuration.go:574] "Deleted container OVS port" container="c2416b575410ded7efbf4329e0636565c2ed77577403023db612b11082d51a2a" interface="vlan-pod-192a59"
I0415 18:31:24.016187       1 allocator.go:250] "IP Pool update succeeded" pool="secnet-ipv4-3" allocation={"usage":{"total":253,"used":0}}
I0415 18:31:24.025363       1 allocator.go:250] "IP Pool update succeeded" pool="secnet-ipv6-3" allocation={"usage":{"total":65535,"used":0}}

--->
I0415 18:31:24.575449       1 net_linux.go:477] "sysctl" path="ipv6/conf/eth0/disable_ipv6" value=1
I0415 18:31:24.575549       1 net_linux.go:482] "sysctl" path="ipv6/conf/eth0~/disable_ipv6" value=0
<---

Do we need to provide some other_config for creating IPv6 enabled ovs internal port ?

jianjuns · 2024-04-15T21:11:12Z

pkg/agent/agent_linux.go

@@ -70,22 +70,25 @@ func (i *Initializer) prepareOVSBridgeForK8sNode() error {
 	uplinkNetConfig := i.nodeConfig.UplinkNetConfig
 	uplinkNetConfig.Name = adapter.Name
 	uplinkNetConfig.MAC = adapter.HardwareAddr
-	uplinkIPs, err := getAllIPNetsByName(adapter.Name)
+	_, uplinkIPs, uplinkRoutes, err := getInterfaceConfig(adapter.Name)


I feel we need not to get uplink IPs and Routes here, but can just get them in PrepareHostInterfaceConnection.

jianjuns · 2024-04-15T21:12:40Z

pkg/agent/agent_linux.go

+	}
+	var uplinkV4Routes []interface{}
+	for _, route := range uplinkRoutes {
+		if route.(netlink.Route).Gw.To4() == nil {


Guess no need to skip IPv6 routes. At least no harm to move them too.

jianjuns · 2024-04-15T21:16:15Z

pkg/agent/util/net_linux.go

+// actions taken in PrepareHostInterfaceConnection.
+func RestoreHostInterfaceConfiguration(brName string, interfaceName string) {
+	klog.InfoS("Restoring bridge config to host interface")
+	if interfaceName != "" {


Could we do the check in the caller. It looks unnatural to pass a "" interfaceName.

jianjuns · 2024-04-15T21:16:53Z

pkg/agent/util/net_linux.go

+// RestoreHostInterfaceConfiguration restore the configuration from bridge back to host interface, reverting the
+// actions taken in PrepareHostInterfaceConnection.
+func RestoreHostInterfaceConfiguration(brName string, interfaceName string) {
+	klog.InfoS("Restoring bridge config to host interface")


Would you increase verbose level for this log?

jianjuns · 2024-04-15T21:17:17Z

pkg/agent/util/net_linux.go

+					}
+				}
+			}
+			klog.InfoS("Finished to restore bridge config to host interface")


Could we log bridge name and interface name too?

We should return at some errors and do not log this message which sounds like restoration succeeded.

jianjuns · 2024-04-15T21:19:02Z

pkg/agent/util/net_linux.go

+	if interfaceName != "" {
+		bridgedName := GenerateUplinkInterfaceName(interfaceName)
+		// restore if interface eth0~ exists
+		if exists, err := interfaceExists(bridgedName); exists || err != nil {


I think we can just log an error and return if err != nil. The operation should not fail and if it can really fail the following operations (and we do not know if they are really nedded) can fail too.

pkg/agent/util/net_linux.go

jianjuns · 2024-04-15T21:31:56Z

@wenyingd : do you know the /proc/sys/netipv6/conf/eth0/disable_ipv6 problem on OVS port that Daman described: #6108 (comment)

wenyingd · 2024-04-19T04:50:37Z

@jianjuns I'm facing a weird problem. EnsureIPv6EnabledOnInterface succeeds but I end up with IPv6 disabled on the internal port and I ultimately end up losing the IPv6 address. I tired to log the value of sysctl periodically in a go-routine and noticed that the path /proc/sys/netipv6/conf/eth0/disable_ipv6 disappears for a few seconds and then magically appears back with the default value of 1.

I0415 18:17:43.764311       1 server.go:524] "CmdAdd for container succeeded" container="6ed33ecc46e5ba305c51d99d5211ac00f9e58d25e497312b0d0be95c8247ea51"

--->
I0415 18:17:43.829266       1 net_linux.go:477] "sysctl" path="ipv6/conf/eth0/disable_ipv6" value=0
I0415 18:17:43.829292       1 net_linux.go:482] "sysctl" path="ipv6/conf/eth0~/disable_ipv6" value=0
<---

I0415 18:17:44.506174       1 allocator.go:154] "IP Pool update succeeded" pool="secnet-ipv4-1" allocation={"ipAddresses":[{"ipAddress":"148.14.24.2","phase":"Allocated","owner":{"pod":{"name":"vlan-pod2","namespace":"testvlannetwork-qprush5v","containerID":"6ed33ecc46e5ba305c51d99d5211ac00f9e58d25e497312b0d0be95c8247ea51","ifName":"eth1"}}}],"usage":{"total":253,"used":1}}
I0415 18:17:44.512202       1 pod_configuration.go:260] "Configured container interface" Pod="testvlannetwork-qprush5v/vlan-pod2" container="6ed33ecc46e5ba305c51d99d5211ac00f9e58d25e497312b0d0be95c8247ea51" interface="eth1" hostInterface="vlan-pod-725b29"
I0415 18:17:44.519200       1 allocator.go:154] "IP Pool update succeeded" pool="secnet-ipv4-1" allocation={"ipAddresses":[{"ipAddress":"148.14.24.2","phase":"Allocated","owner":{"pod":{"name":"vlan-pod2","namespace":"testvlannetwork-qprush5v","containerID":"6ed33ecc46e5ba305c51d99d5211ac00f9e58d25e497312b0d0be95c8247ea51","ifName":"eth1"}}},{"ipAddress":"148.14.24.3","phase":"Allocated","owner":{"pod":{"name":"vlan-pod1","namespace":"testvlannetwork-qprush5v","containerID":"7c79866797a73d1e98b9018e03524c348bae65c82e4b7bb23ffdb7d56659c5d8","ifName":"eth1"}}}],"usage":{"total":253,"used":2}}
I0415 18:17:44.524549       1 pod_configuration.go:260] "Configured container interface" Pod="testvlannetwork-qprush5v/vlan-pod1" container="7c79866797a73d1e98b9018e03524c348bae65c82e4b7bb23ffdb7d56659c5d8" interface="eth1" hostInterface="vlan-pod-39ea04"
I0415 18:17:44.530864       1 allocator.go:154] "IP Pool update succeeded" pool="secnet-ipv4-3" allocation={"ipAddresses":[{"ipAddress":"148.14.26.2","phase":"Allocated","owner":{"pod":{"name":"vlan-pod2","namespace":"testvlannetwork-qprush5v","containerID":"6ed33ecc46e5ba305c51d99d5211ac00f9e58d25e497312b0d0be95c8247ea51","ifName":"eth2"}}}],"usage":{"total":253,"used":1}}
I0415 18:17:44.539065       1 allocator.go:154] "IP Pool update succeeded" pool="secnet-ipv4-2" allocation={"ipAddresses":[{"ipAddress":"148.14.25.111","phase":"Allocated","owner":{"pod":{"name":"vlan-pod3","namespace":"testvlannetwork-qprush5v","containerID":"15d503003f071d1805251ad02118496fd13aeb3c01c31b9c1be4fa06b2a6200e","ifName":"eth1"}}},{"ipAddress":"148.14.25.112","phase":"Allocated","owner":{"pod":{"name":"vlan-pod1","namespace":"testvlannetwork-qprush5v","containerID":"7c79866797a73d1e98b9018e03524c348bae65c82e4b7bb23ffdb7d56659c5d8","ifName":"eth2"}}}],"usage":{"total":13,"used":2}}
I0415 18:17:44.539751       1 allocator.go:154] "IP Pool update succeeded" pool="secnet-ipv6-3" allocation={"ipAddresses":[{"ipAddress":"10:2400::2","phase":"Allocated","owner":{"pod":{"name":"vlan-pod2","namespace":"testvlannetwork-qprush5v","containerID":"6ed33ecc46e5ba305c51d99d5211ac00f9e58d25e497312b0d0be95c8247ea51","ifName":"eth2"}}}],"usage":{"total":65535,"used":1}}
I0415 18:17:44.546449       1 pod_configuration.go:260] "Configured container interface" Pod="testvlannetwork-qprush5v/vlan-pod1" container="7c79866797a73d1e98b9018e03524c348bae65c82e4b7bb23ffdb7d56659c5d8" interface="eth2" hostInterface="vlan-pod-4f9443"
I0415 18:17:44.584129       1 server.go:564] "Received CmdDel request" request="cni_args:{container_id:\"5765fd5bf0b10e5ac209197afc1d77f0da632debf11db685348f7ad5b0133593\" netns:\"/var/run/netns/cni-e8b97ba8-09ee-0705-b75d-a90ff851c7c9\" ifname:\"eth0\" args:\"K8S_POD_NAME=coredns-76f75df574-mdjs4;K8S_POD_INFRA_CONTAINER_ID=5765fd5bf0b10e5ac209197afc1d77f0da632debf11db685348f7ad5b0133593;K8S_POD_UID=ff9296c5-873c-498f-a4e7-feb67b420e88;IgnoreUnknown=1;K8S_POD_NAMESPACE=kube-system\" path:\"/opt/cni/bin\" network_configuration:\"{\\\"cniVersion\\\":\\\"0.3.0\\\",\\\"ipam\\\":{\\\"type\\\":\\\"host-local\\\"},\\\"name\\\":\\\"antrea\\\",\\\"type\\\":\\\"antrea\\\"}\"}"
I0415 18:17:44.588840       1 pod_configuration.go:574] "Deleted container OVS port" container="5765fd5bf0b10e5ac209197afc1d77f0da632debf11db685348f7ad5b0133593" interface="coredns--4a2562"
I0415 18:17:44.643405       1 server.go:550] "Deleted interfaces for container" container="5765fd5bf0b10e5ac209197afc1d77f0da632debf11db685348f7ad5b0133593"
I0415 18:17:44.647260       1 server.go:558] "CmdDel for container succeeded" container="5765fd5bf0b10e5ac209197afc1d77f0da632debf11db685348f7ad5b0133593"
I0415 18:17:44.668347       1 server.go:564] "Received CmdDel request" request="cni_args:{container_id:\"e081278ff1c222dbd1e8abc119048e56e02d6c6347c993b21116549894e7813b\" ifname:\"eth0\" args:\"K8S_POD_UID=ff9296c5-873c-498f-a4e7-feb67b420e88;IgnoreUnknown=1;K8S_POD_NAMESPACE=kube-system;K8S_POD_NAME=coredns-76f75df574-mdjs4;K8S_POD_INFRA_CONTAINER_ID=e081278ff1c222dbd1e8abc119048e56e02d6c6347c993b21116549894e7813b\" path:\"/opt/cni/bin\" network_configuration:\"{\\\"cniVersion\\\":\\\"0.3.0\\\",\\\"ipam\\\":{\\\"type\\\":\\\"host-local\\\"},\\\"name\\\":\\\"antrea\\\",\\\"type\\\":\\\"antrea\\\"}\"}"
I0415 18:17:44.668519       1 server.go:550] "Deleted interfaces for container" container="e081278ff1c222dbd1e8abc119048e56e02d6c6347c993b21116549894e7813b"
I0415 18:17:44.672504       1 server.go:558] "CmdDel for container succeeded" container="e081278ff1c222dbd1e8abc119048e56e02d6c6347c993b21116549894e7813b"
E0415 18:17:44.830058       1 net_linux.go:475] "failed to get sysctl" path="ipv6/conf/eth0/disable_ipv6"

--->
E0415 18:31:00.538942       1 net_linux.go:475] "failed to get sysctl" path="ipv6/conf/eth0/disable_ipv6"
I0415 18:17:44.830462       1 net_linux.go:477] "sysctl" path="ipv6/conf/eth0/disable_ipv6" value=-1
I0415 18:17:44.830505       1 net_linux.go:482] "sysctl" path="ipv6/conf/eth0~/disable_ipv6" value=0
<---

I0415 18:31:23.964727       1 allocator.go:250] "IP Pool update succeeded" pool="secnet-ipv4-1" allocation={"usage":{"total":253,"used":0}}
I0415 18:31:23.965513       1 pod_configuration.go:574] "Deleted container OVS port" container="c2416b575410ded7efbf4329e0636565c2ed77577403023db612b11082d51a2a" interface="vlan-pod-192a59"
I0415 18:31:24.016187       1 allocator.go:250] "IP Pool update succeeded" pool="secnet-ipv4-3" allocation={"usage":{"total":253,"used":0}}
I0415 18:31:24.025363       1 allocator.go:250] "IP Pool update succeeded" pool="secnet-ipv6-3" allocation={"usage":{"total":65535,"used":0}}

--->
I0415 18:31:24.575449       1 net_linux.go:477] "sysctl" path="ipv6/conf/eth0/disable_ipv6" value=1
I0415 18:31:24.575549       1 net_linux.go:482] "sysctl" path="ipv6/conf/eth0~/disable_ipv6" value=0
<---

Do we need to provide some other_config for creating IPv6 enabled ovs internal port ?

May I know the default value for config /proc/sys/net/ipv6/conf/default/disable_ipv6 on your testbed? And did you explicitly set the value as "0" for the OVS internal port "eth0". If the default value is "1", and you didn't explicitly update the value for a given interface (e.g., the new netlink created by OVS internal port), the system is supposed to use the default value "1".

Besides, it seems some re-creation on "eth0" happened as we may see this log intermittently, which may happen when the netlink doesn't exist on the host. If that is true, we may see that a change history on the the value of "disable_ipv6", -1 -> 1.

 E0415 18:17:44.830058       1 net_linux.go:475] "failed to get sysctl" path="ipv6/conf/eth0/disable_ipv6"
...
 I0415 18:17:44.830462       1 net_linux.go:477] "sysctl" path="ipv6/conf/eth0/disable_ipv6" value=-1
 I0415 18:17:44.830505       1 net_linux.go:482] "sysctl" path="ipv6/conf/eth0~/disable_ipv6" value=0

A strange observation is why the initial value of eth0 is "0" if the default is 1, did you manually set it?

aroradaman · 2024-04-19T08:29:33Z

@wenyingd
I'm using kind(0.22.0) cluster on ubuntu (22.04.1 LTS <> 5.15.0-102-generic kernel) as testbed.

/proc/sys/net/ipv6/conf/default/disable_ipv6 is 1.
In the current setup eth0 (IPv6 enabled) is the default interface created by docker.
We rename eth0 to eth0~ and then create an ovs internal port named eth0. We explicitly enable IPv6 on this interface, and assign the IPs(both ip-family) and routes(both ip-family) to this interface. The assignment succeeds without any error.

Netlink is just an API, even if it disappears for a while it shouldn't affect the underlying value which we set initially, right?

wenyingd · 2024-04-19T08:44:59Z

Netlink is just an API, even if it disappears for a while it shouldn't affect the underlying value which we set initially, right?

I wonder the thing is it doesn't disappear for a while, but is deleted and then re-created. So the latter is a new one without your previous manual configurations.

To fix it, can we set the default value as "0" in advance. Then even for the new created ports, it can be enabled.

aroradaman · 2024-04-21T14:53:37Z

thanks @wenyingd!
After setting up default config to enable IPv6 the interface config comes back with IPv6 enabled.

I0421 14:47:14.876941       1 net_linux.go:417] "sysctl" path="ipv6/conf/eth0/disable_ipv6" value=0
I0421 14:47:14.877009       1 net_linux.go:420] "sysctl" path="ipv6/conf/eth0~/disable_ipv6" value=0
I0421 14:47:14.877049       1 net_linux.go:423] "sysctl" path="ipv6/conf/default/disable_ipv6" value=0
.......
.......
.......
I0421 14:47:44.158820       1 net_linux.go:417] "sysctl" path="ipv6/conf/eth0/disable_ipv6" value=-1
I0421 14:47:44.158953       1 net_linux.go:420] "sysctl" path="ipv6/conf/eth0~/disable_ipv6" value=0
I0421 14:47:44.159017       1 net_linux.go:423] "sysctl" path="ipv6/conf/default/disable_ipv6" value=0
.......
.......
.......
I0421 14:47:45.659625       1 net_linux.go:417] "sysctl" path="ipv6/conf/eth0/disable_ipv6" value=0
I0421 14:47:45.659695       1 net_linux.go:420] "sysctl" path="ipv6/conf/eth0~/disable_ipv6" value=0
I0421 14:47:45.659732       1 net_linux.go:423] "sysctl" path="ipv6/conf/default/disable_ipv6" value=0

@jianjuns should we update the default config ipv6/conf/default/disable_ipv6 as part of initialization?

pkg/agent/agent_linux.go

pkg/agent/util/net_linux.go

jianjuns · 2024-04-22T02:16:26Z

pkg/agent/util/net_linux.go

+			}
+
+			// rename host interface(eth0~ -> eth0)
+			if err = RenameInterface(bridgedName, interfaceName); err != nil {


But here, even interfaceName does not exist, we still want to rename eth0~ back to eth0.

Maybe we can add a boolean to record if an interface exists or not. Maybe the code is simpler to return at errors, or when bridgedName does not exist.

jianjuns · 2024-04-22T03:02:09Z

@jianjuns should we update the default config ipv6/conf/default/disable_ipv6 as part of initialization?

@wenyingd : for an IPv6 cluster, is the flag set to 1 or 0? If it can be 1, do we have another solution to resolve the issue with interface moving/creation without setting the flag to 0?

wenyingd · 2024-04-22T03:51:11Z

@jianjuns should we update the default config ipv6/conf/default/disable_ipv6 as part of initialization?

@wenyingd : for an IPv6 cluster, is the flag set to 1 or 0?

For IPv6 cluster, I didn't remember we have steps in the code to modify sysctl configurations dedicated for IPv6, so we may require that IPv6 is enabled by default on any Nodes in the cluster if they plan to run IPv6 (including the IPv6 configurations enabled on interface and the network forwarding configurations). We used to hit issues that IPv6 networking forwarding is not enabled by default in our lab env before (not received similar reports from users), and the solution is to manually enable it in the env as it is one-time setting.

If it can be 1, do we have another solution to resolve the issue with interface moving/creation without setting the flag to 0?

1 means IPv6 is disabled on the target interface, which is a kernel configuration. We may not have userspace workaround on it. If you mean a substitution to not update the default value, we can add the logic to enable the sysctl values in the implementations. A reference is ipam.ConfigureIface, we can think about calling the func in our logic for static configurations, which is also called when configuring Pod's IP for container cases.

pkg/agent/util/net_linux.go

jianjuns · 2024-04-23T16:37:23Z

/test-all

jianjuns · 2024-04-23T18:35:26Z

@aroradaman : seems there are some Windows build errors. Could you check the failed tests like "Go / Build Antrea Windows binaries"?

aroradaman · 2024-04-23T19:01:55Z

/test-all

jianjuns · 2024-04-23T21:42:23Z

There is another error. Check "Go / Golangci-lint (ubuntu-latest)". @aroradaman

aroradaman · 2024-04-24T12:34:42Z

Kind / E2e tests on a Kind cluster on Linux with all features enabled (pull_request) this fails at initializing secnet, empty config is passed. I guess this somehow bypassed validation.
Maybe we can exclude secnet from this test, as secondary bridge and secondary network needs to be created before hand.

jianjuns · 2024-04-24T15:56:21Z

Kind / E2e tests on a Kind cluster on Linux with all features enabled (pull_request) this fails at initializing secnet, empty config is passed. I guess this somehow bypassed validation. Maybe we can exclude secnet from this test, as secondary bridge and secondary network needs to be created before hand.

Could you share more info for me to understand the failure? @aroradaman

If primary nic is configured with secondary bridge antrea-agent will move primary nic configuration(IPs and Routes) to the secondary bridge, and will revert the change on shutdown. Signed-off-by: Daman Arora <aroradaman@gmail.com>

aroradaman · 2024-04-24T20:29:13Z

Could you share more info for me to understand the failure? @aroradaman

I was directly accessing the first ovs bridge of sec net config which was causing panic / index error.

secNetConfig.OVSBridges[0].PhysicalInterfaces

I thought the validation required at least one bridge for the secondary network config.
Checked validateSecondaryNetworkConfig, we do allow empty config.

jianjuns · 2024-04-24T20:43:16Z

pkg/agent/secondarynetwork/init.go

 	if err != nil {
 		return err
 	}

+	// We only support moving and restoring of interface configuration to OVS Bridge for the single physical interface case.
+	if len(secNetConfig.OVSBridges) != 0 {


Ah, I missed this one when reviewing the code.

jianjuns · 2024-04-24T20:56:08Z

/test-all

aroradaman force-pushed the secondary-net-primary-nic branch from 2fccd34 to 1805b30 Compare March 16, 2024 06:12

jianjuns reviewed Mar 18, 2024

View reviewed changes

pkg/agent/secondarynetwork/init.go Outdated Show resolved Hide resolved

pkg/agent/secondarynetwork/init.go Outdated Show resolved Hide resolved

pkg/agent/secondarynetwork/init.go Outdated Show resolved Hide resolved

aroradaman force-pushed the secondary-net-primary-nic branch from 1805b30 to 32825e4 Compare March 26, 2024 09:51

jianjuns reviewed Mar 26, 2024

View reviewed changes

aroradaman force-pushed the secondary-net-primary-nic branch from 32825e4 to 66f782c Compare March 27, 2024 08:57

aroradaman requested a review from jianjuns March 27, 2024 08:57

jianjuns reviewed Mar 27, 2024

View reviewed changes

aroradaman force-pushed the secondary-net-primary-nic branch from 66f782c to 0a5d866 Compare March 27, 2024 19:37

wenyingd reviewed Mar 29, 2024

View reviewed changes

aroradaman force-pushed the secondary-net-primary-nic branch from 0a5d866 to 27c1934 Compare March 31, 2024 18:41

jianjuns reviewed Apr 1, 2024

View reviewed changes

aroradaman force-pushed the secondary-net-primary-nic branch from 27c1934 to 8b79d5a Compare April 3, 2024 13:14

aroradaman requested a review from jianjuns April 3, 2024 13:15

jianjuns reviewed Apr 3, 2024

View reviewed changes

aroradaman force-pushed the secondary-net-primary-nic branch from 8b79d5a to 7f45a55 Compare April 4, 2024 17:10

aroradaman requested a review from jianjuns April 4, 2024 17:11

jianjuns reviewed Apr 4, 2024

View reviewed changes

aroradaman force-pushed the secondary-net-primary-nic branch from 7f45a55 to eb0dead Compare April 6, 2024 12:26

jianjuns reviewed Apr 8, 2024

View reviewed changes

aroradaman force-pushed the secondary-net-primary-nic branch from eb0dead to 7007314 Compare April 15, 2024 18:08

jianjuns reviewed Apr 15, 2024

View reviewed changes

aroradaman force-pushed the secondary-net-primary-nic branch from 7007314 to 4afe74c Compare April 21, 2024 14:28

jianjuns reviewed Apr 22, 2024

View reviewed changes

aroradaman force-pushed the secondary-net-primary-nic branch 2 times, most recently from 2f64197 to ac7192f Compare April 22, 2024 08:45

jianjuns reviewed Apr 22, 2024

View reviewed changes

pkg/agent/util/net_linux.go Show resolved Hide resolved

jianjuns reviewed Apr 23, 2024

View reviewed changes

pkg/agent/util/net_linux.go Show resolved Hide resolved

pkg/agent/util/net_linux.go Outdated Show resolved Hide resolved

pkg/agent/util/net_linux.go Show resolved Hide resolved

aroradaman force-pushed the secondary-net-primary-nic branch from ac7192f to 69241fb Compare April 23, 2024 16:32

jianjuns previously approved these changes Apr 23, 2024

View reviewed changes

aroradaman dismissed jianjuns’s stale review via 10ff49d April 23, 2024 19:00

aroradaman force-pushed the secondary-net-primary-nic branch from 69241fb to 10ff49d Compare April 23, 2024 19:00

aroradaman force-pushed the secondary-net-primary-nic branch from 10ff49d to 94e1327 Compare April 24, 2024 06:37

allow primary nic for secondary bridge

a0bbf0c

If primary nic is configured with secondary bridge antrea-agent will move primary nic configuration(IPs and Routes) to the secondary bridge, and will revert the change on shutdown. Signed-off-by: Daman Arora <aroradaman@gmail.com>

aroradaman force-pushed the secondary-net-primary-nic branch from 94e1327 to a0bbf0c Compare April 24, 2024 20:13

jianjuns reviewed Apr 24, 2024

View reviewed changes

jianjuns approved these changes Apr 24, 2024

View reviewed changes

jianjuns merged commit 6b0f4f4 into antrea-io:main Apr 25, 2024
51 of 56 checks passed

		@@ -166,30 +166,6 @@ func (i *Initializer) saveHostRoutes() error {
		return nil

Use a Node's primary NIC as the secondary OVS bridge physical interface #6108

Use a Node's primary NIC as the secondary OVS bridge physical interface #6108

Conversation

aroradaman commented Mar 14, 2024 • edited Loading

aroradaman commented Mar 16, 2024 • edited Loading

Choose a reason for hiding this comment

aroradaman Mar 27, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jianjuns Mar 27, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aroradaman Apr 3, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aroradaman Apr 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jianjuns Apr 11, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aroradaman commented Apr 15, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jianjuns commented Apr 15, 2024

wenyingd commented Apr 19, 2024 • edited Loading

aroradaman commented Apr 19, 2024 • edited Loading

wenyingd commented Apr 19, 2024 • edited Loading

aroradaman commented Apr 21, 2024

Choose a reason for hiding this comment

jianjuns commented Apr 22, 2024

wenyingd commented Apr 22, 2024 • edited Loading

jianjuns commented Apr 23, 2024

jianjuns commented Apr 23, 2024

aroradaman commented Apr 23, 2024

jianjuns commented Apr 23, 2024

aroradaman commented Apr 24, 2024 • edited Loading

jianjuns commented Apr 24, 2024

aroradaman commented Apr 24, 2024

Choose a reason for hiding this comment

jianjuns commented Apr 24, 2024

aroradaman commented Mar 14, 2024 •

edited

Loading

aroradaman commented Mar 16, 2024 •

edited

Loading

aroradaman Mar 27, 2024 •

edited

Loading

jianjuns Mar 27, 2024 •

edited

Loading

aroradaman Apr 3, 2024 •

edited

Loading

aroradaman Apr 5, 2024 •

edited

Loading

jianjuns Apr 11, 2024 •

edited

Loading

aroradaman commented Apr 15, 2024 •

edited

Loading

wenyingd commented Apr 19, 2024 •

edited

Loading

aroradaman commented Apr 19, 2024 •

edited

Loading

wenyingd commented Apr 19, 2024 •

edited

Loading

wenyingd commented Apr 22, 2024 •

edited

Loading

aroradaman commented Apr 24, 2024 •

edited

Loading