Enhance LinuxCollector to support detecting multiple app VIF IPs

Application VIF may have multiple IP address assigned. They can be
either assigned directly on the same interface, or used on separate
VLAN sub-interfaces which share the parent interface MAC address.

LinuxCollector should detect and publish all of them, instead of
flapping between them and generating many IP address change
notifications, which trigger a flood of NI and App info messages
published to the controller.

For DHCP assigned IP addresses, we use lease time to determine if
a previously detected IP address is still valid. For statically
assigned IP, we expect to see at least one associated ARP packet
every 10 minutes. Otherwise, we consider the IP address to be removed
or simply not used anymore.

This commit also enhances LinuxCollector to support enabling or
disabling ARP snooping in runtime, without requiring to recreate all
switch network instances or rebooting device.

Signed-off-by: Milan Lenco <milan@zededa.com>

docs/APP-CONNECTIVITY.md

@@ -216,6 +216,25 @@ to retrieve cloud-init configuration, obtain information from EVE (e.g. device U
 hostname, external IP address) or to download [patch envelopes](PATCH-ENVELOPES.md).
 More information about metadata server can be found in [ECO-METADATA.md](ECO-METADATA.md).
 
+#### IPAM
+
+Every Local Network Instance must be configured with an IPv4 network subnet and an IP
+range within this subnet for automatic IP allocations. Host IP addresses from this subnet
+that do not fall within the IP range are available for manual assignment.
+
+Whether an IP address is selected manually or dynamically assigned by EVE from the configured
+IP range, an internal DHCP server is used to distribute these IP addresses to applications.
+Container applications are deployed inside a "shim VM", which EVE prepares, ensuring that
+a DHCP client is running for every virtual interface connected to a network instance
+This guarantees that the IP address is received and applied before the application starts.
+In contrast, VM applications are responsible for starting their own DHCP client and applying
+the received IP addresses.
+
+Regardless of the application type, EVE does not automatically assume that the allocated
+IP address is actually in use. Instead, it monitors the set of IP leases granted by the internal
+DHCP server and updates the set of application IP addresses in the published info messages
+accordingly.
+
 ### Switch Network Instance
 
 Switch Network Instance is a simple L2-only bridge between connected applications and
@@ -241,6 +260,52 @@ inbound ACL rules are that much more important.
 A metadata HTTP server is run for a switch network instance only if it has a port attached
 that has an IP address.
 
+#### IP address detection
+
+Unlike a Local Network Instance, a switch network instance is configured without any IP
+configuration, and EVE does not run an internal DHCP server. Instead, if IP connectivity
+is required, IP addresses must be assigned statically within the connected applications
+or provided by an external DHCP server or another application offering DHCP services.
+
+Since EVE is not in control of IP address allocations and leases, it must monitor application
+traffic to learn which IP addresses are being used and report this information to the controller.
+
+In the case of an external DHCP server (IPv4), EVE captures the DHCPACK packet from the server,
+which confirms the leased IP address. Because EVE manages MAC address allocations, it knows
+the MAC address of every application's virtual interface (VIF). It can then map the CHADDR
+(Client Hardware Address) attribute to the corresponding application VIF and learn the assigned
+IP address from the YIADDR (your, i.e. client, IP Address) attribute. Additionally, EVE reads
+the DHCP option 51 (Lease Time), if available, to determine how long the leased IP address
+is valid. If EVE does not observe an IP renewal within this period, it assumes that the IP address
+is no longer in use and reports this change to the controller.
+
+For statically assigned IPv4 addresses, EVE captures both ARP reply and request packets to learn
+the application VIF IP assignment from either Sender IP + MAC or Target IP + MAC attribute
+pairs. Since ARP cache entries have a limited lifetime — typically around 2 minutes — EVE expects
+to see at least one ARP packet for every assigned IP within a 10-minute window (this is not
+configurable). If no ARP packet is observed within this period for a previously detected IP
+assignment, EVE assumes that the IP address has been removed and reports this change to
+the controller. EVE also captures ARP packets for IP addresses configured via DHCP, but these
+are ignored as the information from the previously captured DHCPACK takes precedence.
+Note that ARP-based IP detection is enabled by default but can be disabled by setting
+the configuration item `network.switch.enable.arpsnoop` to `false`. Change in this config
+options will apply to already deployed switch network instances.
+
+For an external DHCPv6 server, EVE captures DHCPv6 REPLY messages. It learns the target MAC
+address from the DUID option (Client Identifier, option code 1), while the IPv6 address
+and its valid lifetime are provided by the IA Address (option code 5).
+
+To learn IPv6 addresses assigned using SLAAC (Stateless Address Auto Configuration),
+EVE captures unicast ICMPv6 Neighbor Solicitation messages. These are sent from the interface
+with the assigned IPv6 address to check if the address is free or already in use by another
+host — a process known as Duplicate Address Detection (DAD). The ICMPv6 packet sent to detect
+IP duplicates for a particular VIF IP will have the VIF MAC address as the source address
+in the Ethernet header. EVE uses this, along with the "Target Address" field from the ICMPv6
+header, to identify the assigned IPv6 address.
+
+EVE is capable of detecting multiple IPs assigned to the same VIF MAC address. This is commonly
+seen when applications use VLAN sub-interfaces, which share the parent interface's MAC address.
+
 ### Network Instance Ports
 
 Network instances can be configured with one or more network adapters, which will be used

docs/CONFIG-PROPERTIES.md

@@ -55,7 +55,7 @@
 | netdump.topic.maxcount | integer | 10 | maximum number of netdumps that can be published for each topic. The oldest netdump is unpublished should a new netdump exceed the limit.
 | netdump.downloader.with.pcap | boolean | false | include packet captures inside netdumps for download requests. However, even if enabled, TCP segments carrying non-empty payload (i.e. content which is being downloaded) are excluded and the overall PCAP size is limited to 64MB. |
 | netdump.downloader.http.with.fieldvalue | boolean | false | include HTTP header field values in captured network traces for download requests (beware: may contain secrets, such as datastore credentials). |
-| network.switch.enable.arpsnoop | boolean | true | enable ARP Snooping on switch Network Instance, may need a device reboot to take effect |
+| network.switch.enable.arpsnoop | boolean | true | enable ARP Snooping on switch Network Instances |
 | wwan.query.visible.providers | bool | false | enable to periodically (once per hour) query the set of visible cellular service providers and publish them under WirelessStatus (for every modem) |
 | network.local.legacy.mac.address | bool | false | enables legacy MAC address generation for local network instances for those EVE nodes where changing MAC addresses in applications will lead to incorrect network configuration |
 

pkg/pillar/cmd/msrv/msrv_test.go

@@ -47,7 +47,14 @@ func TestPostKubeconfig(t *testing.T) {
 		},
 		AppNetAdapterList: []types.AppNetAdapterStatus{
 			{
-				AllocatedIPv4Addr: net.ParseIP("192.168.1.1"),
+				AssignedAddresses: types.AssignedAddrs{
+					IPv4Addrs: []types.AssignedAddr{
+						{
+							Address: net.ParseIP("192.168.1.1"),
+						},
+					},
+					IPv6Addrs: nil,
+				},
 			},
 		},
 	})
@@ -83,7 +90,11 @@ func TestPostKubeconfig(t *testing.T) {
 	niStatus := types.NetworkInstanceStatus{
 		NetworkInstanceInfo: types.NetworkInstanceInfo{
 			IPAssignments: map[string]types.AssignedAddrs{"k": {
-				IPv4Addr: net.ParseIP("192.168.1.1"),
+				IPv4Addrs: []types.AssignedAddr{
+					{
+						Address: net.ParseIP("192.168.1.1"),
+					},
+				},
 			}},
 		},
 	}
@@ -161,7 +172,14 @@ func TestRequestPatchEnvelopes(t *testing.T) {
 		},
 		AppNetAdapterList: []types.AppNetAdapterStatus{
 			{
-				AllocatedIPv4Addr: net.ParseIP("192.168.1.1"),
+				AssignedAddresses: types.AssignedAddrs{
+					IPv4Addrs: []types.AssignedAddr{
+						{
+							Address: net.ParseIP("192.168.1.1"),
+						},
+					},
+					IPv6Addrs: nil,
+				},
 			},
 		},
 	})
@@ -309,7 +327,14 @@ func TestHandleAppInstanceDiscovery(t *testing.T) {
 		},
 		AppNetAdapters: []types.AppNetAdapterStatus{
 			{
-				AllocatedIPv4Addr: net.ParseIP("192.168.1.1"),
+				AssignedAddresses: types.AssignedAddrs{
+					IPv4Addrs: []types.AssignedAddr{
+						{
+							Address: net.ParseIP("192.168.1.1"),
+						},
+					},
+					IPv6Addrs: nil,
+				},
 				AppNetAdapterConfig: types.AppNetAdapterConfig{
 					IfIdx:           2,
 					AllowToDiscover: true,
@@ -320,8 +345,15 @@ func TestHandleAppInstanceDiscovery(t *testing.T) {
 	err = appInstanceStatus.Publish(u.String(), a)
 	g.Expect(err).ToNot(gomega.HaveOccurred())
 	discoverableNet := types.AppNetAdapterStatus{
-		AllocatedIPv4Addr: net.ParseIP("192.168.1.2"),
-		VifInfo:           types.VifInfo{VifConfig: types.VifConfig{Vif: "eth0"}},
+		AssignedAddresses: types.AssignedAddrs{
+			IPv4Addrs: []types.AssignedAddr{
+				{
+					Address: net.ParseIP("192.168.1.2"),
+				},
+			},
+			IPv6Addrs: nil,
+		},
+		VifInfo: types.VifInfo{VifConfig: types.VifConfig{Vif: "eth0"}},
 	}
 	a1 := types.AppInstanceStatus{
 		UUIDandVersion: types.UUIDandVersion{
@@ -367,7 +399,7 @@ func TestHandleAppInstanceDiscovery(t *testing.T) {
 	expected := map[string][]msrv.AppInstDiscovery{
 		u1.String(): {{
 			Port:    discoverableNet.Vif,
-			Address: discoverableNet.AllocatedIPv4Addr.String(),
+			Address: discoverableNet.AssignedAddresses.IPv4Addrs[0].Address.String(),
 		}},
 	}
 	g.Expect(got).To(gomega.BeEquivalentTo(expected))

pkg/pillar/cmd/msrv/pubsub.go

@@ -24,14 +24,42 @@ func (srv *Msrv) lookupAppNetworkStatusByAppIP(ip net.IP) *types.AppNetworkStatu
 	for _, st := range items {
 		status := st.(types.AppNetworkStatus)
 		for _, adapterStatus := range status.AppNetAdapterList {
-			if adapterStatus.AllocatedIPv4Addr.Equal(ip) {
-				return &status
+			for _, adapterIP := range adapterStatus.AssignedAddresses.IPv4Addrs {
+				if adapterIP.Address.Equal(ip) {
+					return &status
+				}
+			}
+			for _, adapterIP := range adapterStatus.AssignedAddresses.IPv6Addrs {
+				if adapterIP.Address.Equal(ip) {
+					return &status
+				}
 			}
 		}
 	}
 	return nil
 }
 
+func (srv *Msrv) lookupAppInstStatusByAppIP(ip net.IP) (*types.AppInstanceStatus, bool) {
+	sub := srv.subAppInstanceStatus
+	items := sub.GetAll()
+	for _, sc := range items {
+		status := sc.(types.AppInstanceStatus)
+		for _, adapterStatus := range status.AppNetAdapters {
+			for _, adapterIP := range adapterStatus.AssignedAddresses.IPv4Addrs {
+				if adapterIP.Address.Equal(ip) {
+					return &status, true
+				}
+			}
+			for _, adapterIP := range adapterStatus.AssignedAddresses.IPv6Addrs {
+				if adapterIP.Address.Equal(ip) {
+					return &status, true
+				}
+			}
+		}
+	}
+	return nil, false
+}
+
 func (srv *Msrv) getExternalIPsForApp(remoteIP net.IP) ([]net.IP, int) {
 	netstatus := srv.lookupNetworkInstanceStatusByAppIP(remoteIP)
 	if netstatus == nil {
@@ -80,11 +108,13 @@ func (srv *Msrv) lookupNetworkInstanceStatusByAppIP(
 	for _, st := range items {
 		status := st.(types.NetworkInstanceStatus)
 		for _, addrs := range status.IPAssignments {
-			if ip.Equal(addrs.IPv4Addr) {
-				return &status
+			for _, assignedIP := range addrs.IPv4Addrs {
+				if ip.Equal(assignedIP.Address) {
+					return &status
+				}
 			}
-			for _, nip := range addrs.IPv6Addrs {
-				if ip.Equal(nip) {
+			for _, assignedIP := range addrs.IPv6Addrs {
+				if ip.Equal(assignedIP.Address) {
 					return &status
 				}
 			}
@@ -483,21 +513,6 @@ func (srv *Msrv) handleAppInstDelete(ctxArg interface{}, key string,
 	srv.Log.Functionf("handleAppInstDelete(%s) done", key)
 }
 
-func (srv *Msrv) lookupAppInstStatusByAppIP(ip net.IP) (*types.AppInstanceStatus, bool) {
-	sub := srv.subAppInstanceStatus
-	items := sub.GetAll()
-	for _, sc := range items {
-		status := sc.(types.AppInstanceStatus)
-		for _, adapterStatus := range status.AppNetAdapters {
-			if adapterStatus.AllocatedIPv4Addr.Equal(ip) {
-				return &status, adapterStatus.AllowToDiscover
-			}
-		}
-	}
-
-	return nil, false
-}
-
 // AppInstDiscovery is a struct which AppInstances see, when they request discoverable
 type AppInstDiscovery struct {
 	Port    string `json:"port"`
@@ -514,13 +529,18 @@ func (srv *Msrv) composeAppInstancesIPAddresses(UUIDToSkip uuid.UUID) map[string
 		}
 		adapters := make([]AppInstDiscovery, 0)
 		for _, adapterStatus := range status.AppNetAdapters {
-			if adapterStatus.AllocatedIPv4Addr == nil || adapterStatus.AllocatedIPv4Addr.String() == "" {
-				continue
+			for _, ip := range adapterStatus.AssignedAddresses.IPv4Addrs {
+				adapters = append(adapters, AppInstDiscovery{
+					Port:    adapterStatus.Vif,
+					Address: ip.Address.String(),
+				})
+			}
+			for _, ip := range adapterStatus.AssignedAddresses.IPv6Addrs {
+				adapters = append(adapters, AppInstDiscovery{
+					Port:    adapterStatus.Vif,
+					Address: ip.Address.String(),
+				})
 			}
-			adapters = append(adapters, AppInstDiscovery{
-				Port:    adapterStatus.Vif,
-				Address: adapterStatus.AllocatedIPv4Addr.String(),
-			})
 		}
 		res[status.UUIDandVersion.UUID.String()] = adapters
 	}

pkg/pillar/cmd/zedagent/handleconfig.go

@@ -1210,8 +1210,22 @@ func updateLocalServerMap(getconfigCtx *getconfigContext, localServerURL string)
 				continue
 			}
 			if localServerIP != nil {
-				// check if the defined IP of localServer equals the allocated IP of the app
-				if adapterStatus.AllocatedIPv4Addr.Equal(localServerIP) {
+				// Check if the defined IP of localServer equals one of the IPs
+				// allocated to the app.
+				var matchesApp bool
+				for _, ip := range adapterStatus.AssignedAddresses.IPv4Addrs {
+					if ip.Address.Equal(localServerIP) {
+						matchesApp = true
+						break
+					}
+				}
+				for _, ip := range adapterStatus.AssignedAddresses.IPv6Addrs {
+					if ip.Address.Equal(localServerIP) {
+						matchesApp = true
+						break
+					}
+				}
+				if matchesApp {
 					srvAddr := localServerAddr{
 						localServerAddr: localServerURL,
 						bridgeIP:        adapterStatus.BridgeIPAddr,

pkg/pillar/cmd/zedagent/handlemetrics.go

@@ -1112,16 +1112,18 @@ func PublishAppInfoToZedCloud(ctx *zedagentContext, uuid string,
 		for _, ifname := range ifNames {
 			networkInfo := new(info.ZInfoNetwork)
 			networkInfo.LocalName = *proto.String(ifname)
-			ipv4Addr, ipv6Addrs, allocated, macAddr, ipAddrMismatch :=
-				getAppIP(ctx, aiStatus, ifname)
-			if ipv4Addr != nil {
-				networkInfo.IPAddrs = append(networkInfo.IPAddrs, ipv4Addr.String())
+			addrs, hasIPv4Addr, macAddr, ipAddrMismatch :=
+				getAppIPs(ctx, aiStatus, ifname)
+			for _, ipv4Addr := range addrs.IPv4Addrs {
+				networkInfo.IPAddrs = append(networkInfo.IPAddrs,
+					ipv4Addr.Address.String())
 			}
-			for _, ipv6Addr := range ipv6Addrs {
-				networkInfo.IPAddrs = append(networkInfo.IPAddrs, ipv6Addr.String())
+			for _, ipv6Addr := range addrs.IPv6Addrs {
+				networkInfo.IPAddrs = append(networkInfo.IPAddrs,
+					ipv6Addr.Address.String())
 			}
 			networkInfo.MacAddr = *proto.String(macAddr.String())
-			networkInfo.Ipv4Up = allocated
+			networkInfo.Ipv4Up = hasIPv4Addr
 			networkInfo.IpAddrMisMatch = ipAddrMismatch
 			name := appIfnameToName(aiStatus, ifname)
 			log.Tracef("app %s/%s localName %s devName %s",
@@ -1560,21 +1562,22 @@ func sendMetricsProtobuf(ctx *getconfigContext,
 
 // Use the ifname/vifname to find the AppNetAdapter status
 // and from there the (ip, allocated, mac) addresses for the app
-func getAppIP(ctx *zedagentContext, aiStatus *types.AppInstanceStatus,
-	vifname string) (net.IP, []net.IP, bool, net.HardwareAddr, bool) {
+func getAppIPs(ctx *zedagentContext, aiStatus *types.AppInstanceStatus,
+	vifname string) (types.AssignedAddrs, bool, net.HardwareAddr, bool) {
 
 	log.Tracef("getAppIP(%s, %s)", aiStatus.Key(), vifname)
 	for _, adapterStatus := range aiStatus.AppNetAdapters {
 		if adapterStatus.VifUsed != vifname {
 			continue
 		}
-		log.Tracef("getAppIP(%s, %s) found AppIP v4: %s, v6: %s, ipv4 assigned %v mac %s",
-			aiStatus.Key(), vifname, adapterStatus.AllocatedIPv4Addr,
-			adapterStatus.AllocatedIPv6List, adapterStatus.IPv4Assigned, adapterStatus.Mac)
-		return adapterStatus.AllocatedIPv4Addr, adapterStatus.AllocatedIPv6List, adapterStatus.IPv4Assigned,
+		log.Tracef("getAppIP(%s, %s) found AppIPs v4: %v, v6: %v, ipv4 assigned %v mac %s",
+			aiStatus.Key(), vifname, adapterStatus.AssignedAddresses.IPv4Addrs,
+			adapterStatus.AssignedAddresses.IPv6Addrs, adapterStatus.IPv4Assigned,
+			adapterStatus.Mac)
+		return adapterStatus.AssignedAddresses, adapterStatus.IPv4Assigned,
 			adapterStatus.Mac, adapterStatus.IPAddrMisMatch
 	}
-	return nil, nil, false, nil, false
+	return types.AssignedAddrs{}, false, nil, false
 }
 
 func createVolumeInstanceMetrics(ctx *zedagentContext, reportMetrics *metrics.ZMetricMsg) {

pkg/pillar/cmd/zedagent/handlenetworkinstance.go

@@ -8,7 +8,6 @@ package zedagent
 import (
 	"bytes"
 	"fmt"
-	"net"
 	"time"
 
 	"github.com/golang/protobuf/ptypes/timestamp"
@@ -134,11 +133,13 @@ func prepareAndPublishNetworkInstanceInfoMsg(ctx *zedagentContext,
 		for mac, addrs := range status.IPAssignments {
 			assignment := new(zinfo.ZmetIPAssignmentEntry)
 			assignment.MacAddress = mac
-			if !addrs.IPv4Addr.Equal(net.IP{}) {
-				assignment.IpAddress = append(assignment.IpAddress, addrs.IPv4Addr.String())
+			for _, assignedIP := range addrs.IPv4Addrs {
+				assignment.IpAddress = append(assignment.IpAddress,
+					assignedIP.Address.String())
 			}
-			for _, ip := range addrs.IPv6Addrs {
-				assignment.IpAddress = append(assignment.IpAddress, ip.String())
+			for _, assignedIP := range addrs.IPv6Addrs {
+				assignment.IpAddress = append(assignment.IpAddress,
+					assignedIP.Address.String())
 			}
 			info.IpAssignments = append(info.IpAssignments, assignment)
 		}