From 310de5a9d649481e23ec3b048cbe90242c6e063f Mon Sep 17 00:00:00 2001 From: Harish Venkatraman Date: Mon, 6 Aug 2018 12:47:57 -0700 Subject: [PATCH] [SONIC] Management VRF design doc This pull request has Management VRF design. Signed-off-by: Harish Venkatraman --- mgmt-vrf-design-doc.md | 313 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 313 insertions(+) create mode 100755 mgmt-vrf-design-doc.md diff --git a/mgmt-vrf-design-doc.md b/mgmt-vrf-design-doc.md new file mode 100755 index 0000000000..a18fef2bca --- /dev/null +++ b/mgmt-vrf-design-doc.md @@ -0,0 +1,313 @@ +Management VRF Design Document + +Introduction +------------ + +Management VRF is a subset of VRF (virtual routing tables and forwarding) and provides a separation between the out-of-band management network and the in-band data plane network. For all VRFs, the main routing table is the default table for all of the data plane switch ports. With management VRF, a second table, mgmt, is used for routing through the Ethernet ports of the switch. The following design for mgmt-vrf uses the l3mdev approach for implementating mgmt-vrf on SONiC. + +Requirements +------------ + Table - 1 + | Req No | Description | Priority | Comments | + |--------|------------------------------------------------------------------------------------------------------------|----------|----------| + | 1 | Develop and implement a separate Management VRF that provide management plane and Data Plane Isolation | | | + | 2 | Management VRF should be associated with a separate L3 Routing table and the management interface | | | + | 3 | Management VRF and Default VRF should support IP services like Ping and traceroute in its context | | | + | 4 | Management VRF should provide the ability to be polled via SNMP both via default VRF and management VRF | | | + | 5 | Management VRF will provide the ability to send SNMP traps both via the management VRF and data network. | | | + | 6 | Enable DHCP services to run in Management VRF context | | | + | | ˇ Dhcpd | | | + | | ˇ Dhcrelay – Default VRF support | | | + | | ˇ dhclient | | | + | 7 | Enable SSH services to run in the Management VRF context and Default VRF context | | + | 8 | Enable TFTP services to run in the Management VRF context and Default VRF context | | + | 9 | Management VRF should support NTP services in its context and default VRF context | | | + | 10 | Management VRF and Default VRF should support wget. Curl and HTTPS services | | | + | 11 | Management VRF and Default VRF should support apt-get package managers in the Management VRF context | | | + | 12 | Management VRF will provide TACACS+ support. | Low | | + + +Use-Cases +--------- + + 1.Out of band management + ---------------------- + + Only Management port is part of management VRF; No FEP is part of management VRF + + Dest IP = Mgmt. IP Dest IP ≠ Mgmt. IP + ------------------ ------------------ + Packet coming via mgmt. port All IP services listed in Table 1 should Packet dropped. (No one arm routing?) + be supported + + Packet coming via Front end port Treat the packet like any other data traffic. Treat the packet like any other data traffic. + To be sent out as per the routing table lookup To be sent out as per the routing table lookup + Dest IP = FEP IP in Default VRF Dest IP ≠ FEP IP in Default VRF + + Packet coming via mgmt. port Packet dropped Packet dropped + + Packet coming via Front end port All IP services listed in Table 1 should Treat the packet like any other data traffic. + be supported. To be sent out as per the routing table lookup + + 2.In band management + ------------------ + + Only Front-end port is part of management VRF; No management port connectivity (typically) + + Dest IP = Mgmt. IP Dest IP ≠ Mgmt. IP + ------------------ ------------------ + Packet coming via mgmt. port Not applicable Not applicable + + Packet coming via Front end port All IP services listed in Table 1 should (One arm routing supported? Need to + be supported test in FTOS/OS10) + Dest IP = Front end port IP Dest IP ≠ Front end port IP + + Packet coming via mgmt. port Not applicable Not applicable + + Packet coming via Front end port All IP services listed in Table 1 should Treat the packet like any other data traffic. + be supported To be sent out as per the routing table lookup + in management VRF + 3.Switch originated traffic + ------------------------- + + Unless explicitly specified to use mamangment VRF, all the services will use the default routing table to egress out of switch. + +Design Approach +--------------- + + | Features | Approach – Name space | Approach – L3MDev | + |------------------|----------------------------------|-----------------------------------------------| + | Kernel Support | Yes | Yes | + | Scalability | Less scalable | Better Scalable | + | Protocol support | IP services should be replicated | IP services can be enslaved to a L3 interface | + | Performance | Overhead of replicated services | Shared services | + +Network namespaces were designed to provide a complete isolation of the entire network stack —devices, network addresses, neighbor and routing tables, protocol ports and sockets. +VRF On the other hand is a network layer 3 feature and as such should really only impact FIB lookups. While the isolation provided by a namespace includes the route tables, a network namespace will have a significant overhead as we will end up having two network stacks. + +Using network name space as VRF’s will involve having services replicated across all VRF’s – for example we will have run two instances of lldp/ssh for Management and Default VRF’s respectively. +Also in terms of extensibility, using namespaces for Multi VRF means N-VRF’s which will have N-application instances and also the additional complexity of aggregating all the data across these instances. + +With L3-Mdev on 4.9 , VRF’s are created and enslaved to an L3 Interface and services like ssh can be shared across VRF’s thereby avoiding overhead of running multiple instances of IP services. + + +Phase-1 +------- + +In this Phase mgmt-vrf feature will be enabled by default when SONiC boots with Debian 4.9 kernel, configuration of mgmt-vrf will be +done using linux commands. + + +DB Schema +--------- + +The config_db.json schema is updated to have the vrfname in MGMT_INTERFACE. The schema representation will look like below + + "MGMT_INTERFACE": { + "eth0|10.11.150.19/24": { + "gwaddr": "10.11.150.254" + "vrfname": "mgmt-vrf" + } + } + +The vrfname if set to mgmt-vrf will program the linux kernel with the vrf configuration and create a mgmt-vrf for the mgmt +port eth0. If it is set to "None" then the mgmt-vrf will not be configured on the system. Management VRF will be enabled by default when SONiC boots. + + +Flow +----- + +By default, mgmt-vrf will be configured in Debian 4.9 kernel, when SONiC boots there will be isolation of traffic between +mgmt traffic and data traffic. This is acheived by setting the vrfname to mgmt-vrf in the minigraph.py file as default setting. +The interfaces.j2 file checks for the vrfname tag to see if it is set to mgmt-vrf. If the vrfname is set then it creates the +mgmt-vrf configuration by executing the lniux commands. Debian 4.9 kernel does not support running services per VRF, by enabling +the tcp_l3mdev_accept=1 the services will work across all VRF's. Below diagram shows the flows of event for creating mgmt-vrf on +SONiC using Debian 4.9 + +![](https://i.imgur.com/Ge6SNX4.png) + + +Phase-2 +------- + +Later when kernel is upgraded to 4.15 or later support for running services in separate VRF will be available. New CLI command will be +provided to create and delete VRF using config commands. Config-save command will be used to save the configuration to config_db. + +Proposed New CLI +---------------- + +The following CLI can be used to Configure and show vrf's. + + config vrf add/del + config vrf member add/del + show vrf config + show vrf brief + show vrf + + + +Flow +----- + +The following modules will be affected in phase-2 for management VRF configuration. + + swss + - cfgmgr + - vrfmgrd.cpp + - vrfmgr.cpp + +Configure vrf using the config cli above, this triggers the vrfmgrd to create/delete the vrf in linux. Changes to configuration files for services is required for services to run per VRF instance. + + +mgmt-vrf Configuration Commands +------------------------------- + # management vrf configuration + 1. Create a VRF + ip link add name type vrf table + + 2. Set the vrf to up + ip link set dev up + + 3. Assign a Network Interface to a VRF + // Network interfaces are assigned to VRF by enslaving the netdevice to VRF device. + ip link set dev eth0 master + + 4. Add a route to the mgmt-vrf + ip route add vrf 0.0.0.0/0 via + +mgmt-vrf Linux Show Commands +---------------------------- + show commands + ------------- + 1. Default VRF show command + ip route show + + 2. mgmt-vrf show command + //Table id is the id used in create VRF command above. + ip route show table + + 3. List's all the VRF that are created. + // -d option shows the table number or id + ip -d link show vrf + + 4. show command to dump brief output of vrf. + ip -br link show type vrf + + 5. show command to dump vrf output + ip link show type vrf + + 6. show command to dump vrf addr + ip addr show vrf + +mgmt-vrf Show Command (SONiC Wrapper) +--------------------- + + | SONiC Utility command | Linux command | Description | + |---------------------------------|--------------------------------|--------------------------------------| + | show vrf | ip link show type vrf | Display the VRF's configured | + | show vrf brief | ip -br link show type vrf | Display VRF brief info | + | show vrf detail | ip -d link show type vrf | Displays VRF detailed info | + | show vrf route | ip route show | Displays the default VRF routes | + | show vrf route table | ip route show table | Displays the VRF routes for table-id | + | show vrf address | ip address show vrf | Displays IP related info for VRF | + +Utilities and Tools +------------------- + 1. ping commands + // default VRF ping command + ping x.x.x.x + + //mgmt-vrf ping command + ping -I mgmt-vrf x.x.x.x + + 2. Traceroute commands + // default VRF traceroute command + traceroute x.x.x.x + + //mgmt-vrf traceroute command + traceroute -i x.x.x.x + +Example Output Of Commands +-------------------------- + root@sonic:~# ip route show + 1.1.1.0/24 dev Ethernet4 proto kernel scope link src 1.1.1.2 + 2.1.1.0/24 dev Ethernet5 proto kernel scope link src 2.1.1.2 + 10.0.0.0/31 dev PortChannel1 proto kernel scope link src 10.0.0.0 linkdown + 10.0.0.4/31 dev PortChannel5 proto kernel scope link src 10.0.0.4 linkdown + 10.0.0.8/31 dev PortChannel16 proto kernel scope link src 10.0.0.8 linkdown + 10.0.0.12/31 dev PortChannel20 proto kernel scope link src 10.0.0.12 linkdown + 172.0.0.0/26 dev Vlan2 proto kernel scope link src 172.0.0.1 linkdown + 240.127.1.0/24 dev docker0 proto kernel scope link src 240.127.1.1 linkdown + root@sonic:~# + root@sonic:~# ip route show table 1 + default via 10.11.55.254 dev eth0 + broadcast 10.11.55.0 dev eth0 proto kernel scope link src 10.11.55.38 + 10.11.55.0/24 dev eth0 proto kernel scope link src 10.11.55.38 + local 10.11.55.38 dev eth0 proto kernel scope host src 10.11.55.38 + broadcast 10.11.55.255 dev eth0 proto kernel scope link src 10.11.55.38 + root@sonic:~# + root@sonic:~# + root@sonic:~# ip -d link show type vrf + 145: mgmt-vrf: mtu 65536 qdisc noqueue state UP mode DEFAULT group default qlen 1000 + link/ether 0e:bb:43:3e:18:ae brd ff:ff:ff:ff:ff:ff promiscuity 0 + vrf table 1 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 + root@sonic:~# + root@sonic:~# + root@sonic:~# ip link show type vrf + 145: mgmt-vrf: mtu 65536 qdisc noqueue state UP mode DEFAULT group default qlen 1000 + link/ether 0e:bb:43:3e:18:ae brd ff:ff:ff:ff:ff:ff + root@sonic:~# + root@sonic:~# + root@sonic:~# + root@sonic:~# ip -br link show type vrf + mgmt-vrf UP 0e:bb:43:3e:18:ae + root@sonic:~# + root@sonic:~# + + root@sonic:~# ip addr show vrf mgmt-vrf + 2: eth0: mtu 1500 qdisc mq master mgmt-vrf state UP group default qlen 1000 + link/ether 4c:76:25:f5:54:80 brd ff:ff:ff:ff:ff:ff + inet 10.11.55.38/24 brd 10.11.55.255 scope global eth0 + valid_lft forever preferred_lft forever + inet6 fe80::4e76:25ff:fef5:5480/64 scope link + valid_lft forever preferred_lft forever + root@sonic:~# ip addr show type vrf + 145: mgmt-vrf: mtu 65536 qdisc noqueue state UP group default qlen 1000 + link/ether 0e:bb:43:3e:18:ae brd ff:ff:ff:ff:ff:ff + root@sonic:~# + root@sonic:~# ping -I mgmt-vrf 8.8.8.8 + ping: Warning: source address might be selected on device other than mgmt-vrf. + PING 8.8.8.8 (8.8.8.8) from 10.11.55.38 mgmt-vrf: 56(84) bytes of data. + 64 bytes from 8.8.8.8: icmp_seq=1 ttl=119 time=3.55 ms + 64 bytes from 8.8.8.8: icmp_seq=2 ttl=119 time=2.77 ms + + --- 8.8.8.8 ping statistics --- + 2 packets transmitted, 2 received, 0% packet loss, time 1001ms + rtt min/avg/max/mdev = 2.772/3.165/3.559/0.397 ms + + root@sonic:~# traceroute -i mgmt-vrf 8.8.8.8 + traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets + 1 10.11.55.254 (10.11.55.254) 0.468 ms 0.441 ms 0.468 ms + 2 10.11.3.254 (10.11.3.254) 0.599 ms 10.11.3.253 (10.11.3.253) 0.885 ms 10.11.3.254 (10.11.3.254) 0.886 ms + 3 10.11.27.254 (10.11.27.254) 0.521 ms 0.482 ms 0.798 ms + 4 63.80.56.65 (63.80.56.65) 1.141 ms 1.278 ms 2.977 ms + 5 65.198.30.33 (65.198.30.33) 1.982 ms 2.871 ms 1.999 ms + 6 152.179.99.173 (152.179.99.173) 3.307 ms 2.442 ms 4.390 ms + 7 140.222.237.207 (140.222.237.207) 6.141 ms 140.222.237.205 (140.222.237.205) 8.052 ms 140.222.237.207 (140.222.237.207) 6.091 ms + 8 152.179.48.74 (152.179.48.74) 4.229 ms 2.930 ms 5.132 ms + 9 108.170.242.81 (108.170.242.81) 3.316 ms 108.170.243.1 (108.170.243.1) 7.184 ms 5.406 ms + 10 216.239.46.167 (216.239.46.167) 6.463 ms 74.125.37.41 (74.125.37.41) 4.269 ms 108.170.230.87 (108.170.230.87) 6.593 ms + 11 8.8.8.8 (8.8.8.8) 4.861 ms 5.541 ms 4.087 ms + + + +Future Changes For Linux Upgrade +-------------------------------- + + Minimal changes required to support multiple services or configuration files across multiple VRF's. Can leverage the latest feature that are + supported as we upgrade Linux. + + Changes will be required to run IP services per VRF, we will vist this and update the design accordingly in future. + + ip vrf exec command will be available in the next linux upgrade when iproute2 utilities will also be upgraded. +