proposal: net: fetch all interfaces with associated addresses in a single call

[mandarjog]

Background

At present if we want to get all interface and associated Addresses we write the following

ifaceByName := make(map[string]net.Interface)
addrByName := make(map[string][]net.IP)

interfaces, _ := net.Interfaces()

for i := 0; i < len(interfaces); i++ {
    intf := interfaces[i]
    addrByName[intf.Name] = intf.Addrs()
}

I would like a function that returns this entire information.

Why is it needed ?

intf.Addrs() call is inefficient because it results in getting the entire RIB syscall.NetlinkRIB(syscall.RTM_GETADDR, syscall.AF_UNSPEC) every time. The size of the RIB is proportional to the number of interfaces/Addrs. If on a router you have 1000s of interfaces this O(n^2) behaviour becomes untenable.
When number of interfaces is small this problem does not show up.

details

The flamegraphs show excessive time spent making syscall and repeatedly parsing messages.
I also ran call count using bpftrace

/usr/local/bin/bpftrace -p ${prog-pid}\
    -e 'uprobe:/usr/local/bin/prog:*vmsysmon.*,uprobe:/usr/local/bin/prog:net.int*,uprobe:/usr/local/bin/prog:syscall.* { @[func] = count(); }'

@[prog.(*IfMon).calcStats]: 5922
@[net.interfaceAddrTable]: 6169
@[syscall.Close]: 6318
@[syscall.RawSyscall.abi0]: 10287
@[syscall.Syscall.abi0]: 10583
@[syscall.(*SockaddrNetlink).sockaddr]: 12320
@[syscall.ParseNetlinkRouteAttr]: 16582
@[syscall.recvfrom]: 398880
@[syscall.anyToSockaddr]: 506789
@[syscall.Syscall6.abi0]: 568181
@[syscall.Recvfrom]: 622379
@[syscall.ParseNetlinkMessage]: 655167

calcStats function results in

After I made the change to fetch the entire RIB and associate addresses in a single loop it makes significant change in the call counts and times.

@[syscall.Syscall.abi0]: 2265
@[syscall.recvfrom]: 3330
@[syscall.Recvfrom]: 3334
@[syscall.ParseNetlinkMessage]: 3335
@[syscall.anyToSockaddr]: 3347
@[syscall.Syscall6.abi0]: 3482
@[aviatrix.com/vmsysmon.(*IfMon).calcStats]: 5912
@[syscall.ParseNetlinkRouteAttr]: 18326

Proposed API

Option 1

// InterfaceAddrsWithIndex returns a map from interface index to a list of
// associated unicast addresses.
func InterfaceAddrsWithIndex()(map[int][]Addr, error)  {
	return nil, nil
}

Keeps the flavor of existing InterfaceAddrs API.
Typical use.

func testListInterfaceDetails() {
	interfaces, _ := Interfaces()
	addrsMap, _ := InterfaceAddrsWithIndex()

	for i := 0; i < len(interfaces); i++ {
		intf := interfaces[i]
		if addrs, found := addrsMap[intf.Index]; found {
			fmt.Printf("%v: %v\n", intf, addrs)
		}
	}
}

Option 2

Treat all the information above as a single DB and return it. More optimizations are possible with this API without changing the API signature.

// IntfRecord keeps interface and associated Addresses.
type IntfRecord struct {
	Intf  Interface
	Addrs []Addr
}
type IntfDB map[int]IntfRecord

// InterfaceDB returns a map from interface index to an IntfRecord.
// This is the most efficient way feth the entire Interface database.
func InterfaceDB() (IntfDB, error) {
	return nil, nil
}

[ianlancetaylor]

What specific new API do you propose to address this? Thanks.

[mandarjog]

@ianlancetaylor I have updated the issue with the API proposal.

[mandarjog]

@ianlancetaylor can you point me to benchmark tests that test with a number of interfaces? I found benchmark tests in interface_tests.go however they operate with a small number of interfaces (perhaps just loopback?). Is there test infra to create a large number of interfaces, managing network namespaces to do that and cleanup ?

[seankhliao]

Alternate API: the net.Addrs returned by net.InterfaceAddrs() could expose an additional method for getting the associated interface

[ianlancetaylor]

Is there test infra to create a large number of interfaces, managing network namespaces to do that and cleanup ?

Not to my knowledge.

[seankhliao]

#51934 is a report for efficiency / race / ignoring failures from k8s

[mandarjog]

@seankhliao as you can see in the usage example I have specified, devices and interfaces are snapshots at slightly different times. This means that we should expect to see device indexes returned by 2nd call to be unknown to the interfaces() call. The only way to fix it is to get a full snapshot from the kernel, which i don’t think is available.

[mandarjog]

Alternate API: the net.Addrs returned by net.InterfaceAddrs() could expose an additional method for getting the associated interface

interesting idea however,
@seankhliao addresses are a property of an interface not the other way around. So from a modeling perspective it does not make sense.

[mandarjog]

@ianlancetaylor any comments? Does a PR need to be implemented for all platforms or can start with Linux only ?

[ianlancetaylor]

This will go to review by the proposal review committee as described at https://go.dev/s/proposal.

If we adopt this change, it needs to do something sensible on all platforms.

Thanks.

[wjordan]

Alternate API: the net.Addrs returned by net.InterfaceAddrs() could expose an additional method for getting the associated interface

interesting idea however, @seankhliao addresses are a property of an interface not the other way around. So from a modeling perspective it does not make sense.

I think it makes perfect sense, both from a 'modeling perspective' (it's called a 'belongs to' relation) and based on the existing underlying ifaddrmsg structure (which contains an 'interface index' property).

That said, this seems like a Linux-specific issue that may not require any changes to the high-level API to resolve. An alternate platform-implementation-only fix would be for the Linux implementation of net.InterfaceAddrs() (specifically newAddr()) to return a subtype of net.IPNet that contains an additional Index field, which would make it possible for user-level code to associate this Linux-specific address type with interfaces returned by net.Interfaces(), skipping the extra calls to intf.Addrs() as a platform-specific performance optimization.