Tune "flags" regex for ip and ifconfig output + add test case

[claireyywang]

Currently the flags are only parsed in ifconfig command pattern. It would be nice to have that in ip too since ifconfig is deprecated. Thanks!

[benjaoming]

Hi @claireyywang - thanks for filing the issue! A PR is most welcome.

I think there's already test data for this purpose. The behavior can be implemented and already existing unit tests can be expanded to cover flags.

[shr-project]

And even with ifconfig command it doesn't work with some versions of net-tools

e.g.

ifconfig --version
net-tools 1.60
ifconfig 1.42 (2001-04-13)

with this format:

root@qemux86-64:~# ifconfig
eth0      Link encap:Ethernet  HWaddr 52:54:00:12:34:02  
          inet addr:192.168.7.2  Bcast:192.168.7.255  Mask:255.255.255.0
          inet6 addr: fe80::5054:ff:fe12:3402/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:6 errors:0 dropped:0 overruns:0 frame:0
          TX packets:106 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:496 (496.0 B)  TX bytes:42832 (41.8 KiB)

doesn't parse the flags, while net-tools 1.60+git20180626.aebd88e-1ubuntu1 in ubuntu works with this format:

# ifconfig eth0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.17.0.2  netmask 255.255.0.0  broadcast 172.17.255.255
        ether 02:42:ac:11:00:02  txqueuelen 0  (Ethernet)
        RX packets 39299  bytes 89614740 (89.6 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 33767  bytes 3807354 (3.8 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ifconfig --version
net-tools 2.10-alpha

works

The format was probably changed in:
https://sourceforge.net/p/net-tools/code/ci/eb04ef31571f6c707eacaba6846feeebfab518e6
which unfortunately doesn't mention anything about format change in commit message and considering state of net-tools package in various distributions it's hard to tell which distributions have this included, e.g. debian probably included it since:

net-tools (1.60+git20150829.73cef8a-1) unstable; urgency=medium

  * After 14 years without an upstream release, I am producing a new package
    based on today's upstream repository.
    Closes: #391495, #486448, #323261, #260587, #545328, #511395.
  * Remove many patches now merged upstream, delete unmaintainable and
    undocumented local changes, and update the rest.

 -- Martín Ferrari <tincho@debian.org>  Mon, 07 Sep 2015 01:54:07 +0000

[benjaoming]

@shr-project the first example (that's the broken one, right) has a test case written:

python-ifcfg/tests/ifconfig_out.py

Lines 4 to 23 in d936468

	LINUX2 = """
	eth0 Link encap:Ethernet HWaddr 1a:2b:3c:4d:5e:6f
	inet addr:192.168.0.1 Bcast:192.168.0.255 Mask:255.255.255.0
	inet6 addr: fe80::4240:36ff:fe38:a121/64 Scope:Link
	UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
	RX packets:4041381 errors:0 dropped:0 overruns:0 frame:0
	TX packets:3851783 errors:0 dropped:0 overruns:0 carrier:0
	collisions:0 txqueuelen:1000
	RX bytes:1123058554 (1.0 GiB) TX bytes:737462074 (703.2 MiB)
	Interrupt:24
	lo Link encap:Local Loopback
	inet addr:127.0.0.1 Mask:255.0.0.0
	inet6 addr: ::1/128 Scope:Host
	UP LOOPBACK RUNNING MTU:16436 Metric:1
	RX packets:126491 errors:0 dropped:0 overruns:0 frame:0
	TX packets:126491 errors:0 dropped:0 overruns:0 carrier:0
	collisions:0 txqueuelen:0
	RX bytes:43170341 (41.1 MiB) TX bytes:43170341 (41.1 MiB)
	"""

...but upon further investigation it seems like the test case doesn't check if the flags were found:

python-ifcfg/tests/ifconfig_tests.py

Lines 43 to 52 in d936468

	def test_linux2(self):
	ifcfg.distro = 'Linux'
	ifcfg.Parser = LinuxParser
	parser = ifcfg.get_parser(ifconfig=ifconfig_out.LINUX2)
	interfaces = parser.interfaces
	self.assertEqual(len(interfaces.keys()), 2)
	eq_(interfaces['eth0']['ether'], '1a:2b:3c:4d:5e:6f')
	eq_(interfaces['eth0']['inet'], '192.168.0.1')
	eq_(interfaces['eth0']['broadcast'], '192.168.0.255')
	eq_(interfaces['eth0']['netmask'], '255.255.255.0')

[shr-project]

Sorry about the spam from meta-ros commits (it's useful to mention related issue ticket in commit message and in many cases I didn't want to trigger notification from the other side, but github is just too clever).

Anyway, I've also sent patch to OpenEmbedded (where I've seen the issue first) to update the ifconfig format to match what debian/ubuntu currently use:
https://lists.openembedded.org/g/openembedded-core/message/139567
now python-ifcfg works fine for me and ros is happily finding those flags.

If you can extend the regex to cover older ifconfig (which will probably work on busybox's version of ifconfig as well) then it would be still nice for other people I guess (especially as you already have a test data for it).

Thanks

[benjaoming]

In #61, @dre1902 has implemented a very simple approach to parsing and storing flags from ip. It's not platform transparent, but does it work well enough for you cases?

[shr-project]

@benjaoming
Thanks for update, I've reverted the change in net-tools to reproduce this issue again and backported the change from #61 to python-ifcfg-0.21. Unfortunately it still doesn't parse correctly:

root@qemux86-64:~# grep -B 3 flags= /usr/lib/python3.8/site-packages/ifcfg/parser.py
    @classmethod
    def get_patterns(cls):
        return [
            r'(?P<device>^[-a-zA-Z0-9:\.]+): flags=(?P<flags>.*) mtu (?P<mtu>\d+)',

root@qemux86-64:~# ifconfig
eth0      Link encap:Ethernet  HWaddr 52:54:00:12:34:02  
          inet addr:192.168.7.2  Bcast:192.168.7.255  Mask:255.255.255.0
          inet6 addr: fe80::5054:ff:fe12:3402/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:6 errors:0 dropped:0 overruns:0 frame:0
          TX packets:49 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:496 (496.0 B)  TX bytes:15914 (15.5 KiB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:340 errors:0 dropped:0 overruns:0 frame:0
          TX packets:340 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:22124 (21.6 KiB)  TX bytes:22124 (21.6 KiB

root@qemux86-64:~# python3 -m ifcfg.cli | python3 -m json.tool
{
    "eth0": {
        "inet": "192.168.7.2",
        "inet4": [
            "192.168.7.2"
        ],
        "ether": "52:54:00:12:34:02",
        "inet6": [
            "add",
            "fe80::5054:ff:fe12:3402"
        ],
        "netmask": "255.255.255.0",
        "device": "eth0",
        "broadcast": "192.168.7.255",
        "mtu": "1500",
        "rxbytes": "496",
        "txbytes": "18864"
    },
    "lo": {
        "inet": "127.0.0.1",
        "inet4": [
            "127.0.0.1"
        ],
        "ether": null,
        "inet6": [
            "add",
            "::1"
        ],
        "netmask": "255.0.0.0",
        "device": "lo",
        "mtu": "65536",
        "rxbytes": "25540",
        "txbytes": "25540"
    },
    "sit0": {
        "inet": null,
        "inet4": [],
        "ether": "00-00-00-00-30-30-30-00-00-00-00-00-00-00-00-00",
        "inet6": [],
        "netmask": null,
        "device": "sit0",
        "mtu": "1480",
        "rxbytes": "0",
        "txbytes": "0"
    }
}

the old format from net-tools's ifconfig as well as busybox's (as in v1.33.0) ifconfig doesn't have 'flags=' at all, the flags are listed only on this line:

UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

For my case (https://github.com/ros/meta-ros) I've just backported the format change to net-tools for python-ifcfg to parse the flags correctly, I don't know how many distributions still ship older net-tools especially now with new upstream release 2.10 (from 2021-01-07) which includes the format change from 2003 https://sourceforge.net/p/net-tools/code/ci/eb04ef31571f6c707eacaba6846feeebfab518e6

[benjaoming]

@shr-project thanks for testing. IIRC (don't have time to verify it ATM) then ifconfig is used if it exists. I guess on second thought that since the new behavior only works for systems that only have the ip command.