[SNMP][IPv6][202012]: Fix SNMP IPv6 reachability issue in certain scenarios #16329
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…onic-net#15487) Modify snmpd.conf to start snmpd to listen on specific management and loopback ips instead of listening on any ip. #### Why I did it SNMP over IPv6 is not working for all scenarios for a single asic platforms. The expectation is that SNMP query over IPv6 should work over Management or Loopback0 addresses. **Specific scenario where this issue is seen** In case of Lab T0 device, when SNMP request is sent from a directly connected T1 neighbor over Loopback IP, SNMP response was not received. This was because the SRC IP address in SNMP response was not Loopback IP, it was the PortChannel IP connected to the neighboring device. ``` 23:18:51.620897 In 22:26:27:e6:e0:07 ethertype IPv6 (0x86dd), length 105: fc00::72.41725 > **fc00:1::32**.161: C="msft" **GetRequest**(28) .1.3.6.1.2.1.1.1.0 23:18:51.621441 Out 28:99:3a:a0:97:30 ethertype IPv6 (0x86dd), length 241: **fc00::71**.161 > fc00::72.41725: C="msft" **GetResponse**(162) .1.3.6.1.2.1.1.1.0="SONiC Software Version: SONiC.xxx - HwSku: xx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64" ``` In case of IPv4, the SRC IP in SNMP response was correctly set to Loopback IP. ``` 23:25:32.769712 In 22:26:27:e6:e0:07 ethertype IPv4 (0x0800), length 85: 10.0.0.57.56701 > **10.1.0.32**.161: C="msft" **GetRequest**(28) .1.3.6.1.2.1.1.1.0 23:25:32.975967 Out 28:99:3a:a0:97:30 ethertype IPv4 (0x0800), length 221: **10.1.0.32**.161 > 10.0.0.57.56701: C="msft" **GetResponse**(162) .1.3.6.1.2.1.1.1.0="SONiC Software Version: SONiC.xxx - HwSku: xx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64" ``` **Sequence of SNMP request and response** 1. SNMP request will be sent with SRC IP fc00::72 DST IP fc00:1::32 2. SNMP request is received at SONiC device is sent to snmpd which is listening on port 161 :::161/ 3. snmpd process will parse the request create a response and sent to DST IP fc00::72. snmpd process does not track the DST IP on which the SNMP request was received, which in this case is Loopback IP. snmpd process will only keep track what is tht IP to which the response should be sent to. 4. snmpd process will send the response packet. 5. Kernel will do a route look up on destination IP and find the best path. ip -6 route get fc00::72 fc00::72 from :: dev PortChannel101 proto kernel src fc00::71 metric 256 pref medium 5. Using the "src" ip from about, the response is sent out. This SRC ip is that of the PortChannel and not the device Loopback IP. The same issue is seen when SNMP query is sent from a remote server over Management IP. SONiC device eth0 --------- Remote server SNMP request comes with SRC IP <Remote_server> DST IP <Mgmt IP> If kernel finds best route to Remote_server_IP is via BGP neighbors, then it will send the response via front-panel interface with SRC IP as Loopback IP instead of Management IP. Main issue is that in case of IPv6, snmpd ignores the IP address to which SNMP request was sent, in case of IPv6. In case of IPv4, snmpd keeps track of DST IP of SNMP request, it will keep track if the SNMP request was sent to mgmt IP or Loopback IP. Later, this IP is used in ipi_spec_dst as SRC IP which helps kernel to find the route based on DST IP using the right SRC IP. https://github.com/net-snmp/net-snmp/blob/master/snmplib/transports/snmpUDPBaseDomain.c#L300 ipi.ipi_spec_dst.s_addr = srcip->s_addr Reference: https://man7.org/linux/man-pages/man7/ip.7.html ``` If IP_PKTINFO is passed to sendmsg(2) and ipi_spec_dst is not zero, then it is used as the local source address for the routing table lookup and for setting up IP source route options. When ipi_ifindex is not zero, the primary local address of the interface specified by the index overwrites ipi_spec_dst for the routing table lookup. ``` **This issue is not seen on multi-asic platform, why?** on multi-asic platform, there exists different network namespaces. SNMP docker with snmpd process runs on host namespace. Management interface belongs to host namespace. Loopback0 is configured on asic namespaces. Additional inforamtion on how the packet coming over Loopback IP reaches snmpd process running on host namespace: sonic-net#5420 Because of this separation of network namespaces, the route lookup of destination IP is confined to routing table of specific namespace where packet is received. if packet is received over management interface, SNMP response also is sent out of management interface. Same goes with packet received over Loopback Ip. ##### Work item tracking - Microsoft ADO **17537063**: #### How I did it Have snmpd listen on specific Management and Loopback IPs specifically instead of listening on any IP for single-asic platform. Before Fix ``` admin@xx:~$ sudo netstat -tulnp | grep 161 udp 0 0 0.0.0.0:161 0.0.0.0:* 15631/snmpd udp6 0 0 :::161 :::* 15631/snmpd ``` After fix ``` admin@device:~$ sudo netstat -tulnp | grep 161 udp 0 0 10.1.0.32:161 0.0.0.0:* 215899/snmpd udp 0 0 10.3.1.1:161 0.0.0.0:* 215899/snmpd udp6 0 0 fc00:1::32:161 :::* 215899/snmpd udp6 0 0 fc00:2::32:161 :::* 215899/snmpd ``` **How this change helps with the issue?** To see snmpd trace logs, modify snmpd to start using the below parameters, in supervisord.conf file ``` /usr/sbin/snmpd -f -LS0-7i -Lf /var/log/snmpd.log ``` When snmpd listens on any IP, snmpd binds to IPv4 and IPv6 sockets as below: ``` netsnmp_udpbase: binding socket: 7 to UDP: [0.0.0.0]:0->[0.0.0.0]:161 trace: netsnmp_udp6_transport_bind(): transports/snmpUDPIPv6Domain.c, 303: netsnmp_udpbase: binding socket: 8 to UDP/IPv6: [::]:161 ``` When IPv4 response is sent, it goes out of fd 7 and IPv6 response goes out of fd 8. When IPv6 response is sent, it does not have the right SRC IP and it can lead to the issue described. When snmpd listens on specific Loopback/Management IPs, snmpd binds to different sockets: ``` trace: netsnmp_udpipv4base_transport_bind(): transports/snmpUDPIPv4BaseDomain.c, 207: netsnmp_udpbase: binding socket: 7 to UDP: [0.0.0.0]:0->[10.250.0.101]:161 trace: netsnmp_udpipv4base_transport_bind(): transports/snmpUDPIPv4BaseDomain.c, 207: netsnmp_udpbase: binding socket: 8 to UDP: [0.0.0.0]:0->[10.1.0.32]:161 trace: netsnmp_register_agent_nsap(): snmp_agent.c, 1261: netsnmp_register_agent_nsap: fd 8 netsnmp_udpbase: binding socket: 10 to UDP/IPv6: [fc00:1::32]:161 trace: netsnmp_register_agent_nsap(): snmp_agent.c, 1261: netsnmp_register_agent_nsap: fd 10 netsnmp_ipv6: fmtaddr: t = (nil), data = 0x7fffed4c85d0, len = 28 trace: netsnmp_udp6_transport_bind(): transports/snmpUDPIPv6Domain.c, 303: netsnmp_udpbase: binding socket: 9 to UDP/IPv6: [fc00:2::32]:161 ``` When SNMP request comes in via Loopback IPv4, SNMP response is sent out of fd 8 ``` trace: netsnmp_udpbase_send(): transports/snmpUDPBaseDomain.c, 511: netsnmp_udp: send 170 bytes from 0x5581f2fbe30a to UDP: [10.0.0.33]:46089->[10.1.0.32]:161 on fd 8 ``` When SNMP request comes in via Loopback IPv6, SNMP response is sent out of fd 10 ``` netsnmp_ipv6: fmtaddr: t = (nil), data = 0x5581f2fc2ff0, len = 28 trace: netsnmp_udp6_send(): transports/snmpUDPIPv6Domain.c, 164: netsnmp_udp6: send 170 bytes from 0x5581f2fbe30a to UDP/IPv6: [fc00::42]:43750 on fd 10 ``` #### How to verify it Verified on single asic and multi-asic devices. Single asic SNMP query with Loopback ``` ARISTA01T1#bash snmpget -v2c -c xxx 10.1.0.32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: Arista-7260xx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64 ARISTA01T1#bash snmpget -v2c -c xxx fc00:1::32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: Arista-7260xxx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64 ``` On multi-asic -- no change. ``` sudo netstat -tulnp | grep 161 udp 0 0 0.0.0.0:161 0.0.0.0:* 17978/snmpd udp6 0 0 :::161 :::* 17978/snmpd ``` Query result using Loopback IP from a directly connected BGP neighbor ``` ARISTA01T2#bash snmpget -v2c -c xxx 10.1.0.32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: xx - Distribution: Debian 9.13 - Kernel: 4.9.0-14-2-amd64 ARISTA01T2#bash snmpget -v2c -c xxx fc00:1::32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: xx - Distribution: Debian 9.13 - Kernel: 4.9.0-14-2-amd64 ``` <!-- If PR needs to be backported, then the PR must be tested against the base branch and the earliest backport release branch and provide tested image version on these two branches. For example, if the PR is requested for master, 202211 and 202012, then the requester needs to provide test results on master and 202012. --> (cherry picked from commit 9864dfe)
…sonic-net#16013) <!-- Please make sure you've read and understood our contributing guidelines: https://github.com/Azure/SONiC/blob/gh-pages/CONTRIBUTING.md ** Make sure all your commits include a signature generated with `git commit -s` ** If this is a bug fix, make sure your description includes "fixes #xxxx", or "closes #xxxx" or "resolves #xxxx" Please provide the following information: --> #### Why I did it fixes: sonic-net#16001 Caused by: sonic-net#15487 The above PR introduced change to use Management and Loopback Ipv4 and ipv6 addresses as snmpagent address in snmpd.conf file. With this change, if Link local IP address is configured as management or Loopback IPv6 address, then snmpd tries to open socket on that ipv6 address and fails with the below error: ``` Error opening specified endpoint "udp6:[fe80::5054:ff:fe6f:16f0]:161" Server Exiting with code 1 ``` From RFC4007, if we need to specify non-global ipv6 address without ambiguity, we need to use zone id along with the ipv6 address: <address>%<zone_id> Reference: https://datatracker.ietf.org/doc/html/rfc4007 ##### Work item tracking - Microsoft ADO **(number only)**: #### How I did it Modify snmpd.conf file to use the %zone_id representation for ipv6 address. #### How to verify it In VS testbed, modify config_db to use link local ipv6 address as management address: "MGMT_INTERFACE": { "eth0|10.250.0.101/24": { "forced_mgmt_routes": [ "172.17.0.1/24" ], "gwaddr": "10.250.0.1" }, "eth0|fe80::5054:ff:fe6f:16f0/64": { "gwaddr": "fe80::1" } }, Execute config_reload after the above change. snmpd comes up and check if snmpd is listening on ipv4 and ipv6 addresses: ``` admin@vlab-01:~$ sudo netstat -tulnp | grep 161 tcp 0 0 127.0.0.1:3161 0.0.0.0:* LISTEN 274060/snmpd udp 0 0 10.1.0.32:161 0.0.0.0:* 274060/snmpd udp 0 0 10.250.0.101:161 0.0.0.0:* 274060/snmpd udp6 0 0 fc00:1::32:161 :::* 274060/snmpd udp6 0 0 fe80::5054:ff:fe6f::161 :::* 274060/snmpd -- Link local admin@vlab-01:~$ sudo ifconfig eth0 eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 10.250.0.101 netmask 255.255.255.0 broadcast 10.250.0.255 inet6 fe80::5054:ff:fe6f:16f0 prefixlen 64 scopeid 0x20<link> ether 52:54:00:6f:16:f0 txqueuelen 1000 (Ethernet) RX packets 36384 bytes 22878123 (21.8 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 261265 bytes 46585948 (44.4 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 admin@vlab-01:~$ docker exec -it snmp snmpget -v2c -c public fe80::5054:ff:fe6f:16f0 1.3.6.1.2.1.1.1.0 iso.3.6.1.2.1.1.1.0 = STRING: "SONiC Software Version: SONiC.master.327516-04a6031b2 - HwSku: Force10-S6000 - Distribution: Debian 11.7 - Kernel: 5.10.0-18-2-amd64" ``` Logs from snmpd: ``` Turning on AgentX master support. NET-SNMP version 5.9 Connection from UDP/IPv6: [fe80::5054:ff:fe6f:16f0%eth0]:44308 ``` Ran test_snmp_loopback test to check if loopback ipv4 and ipv6 works: ``` ./run_tests.sh -n vms-kvm-t0 -d vlab-01 -c snmp/test_snmp_loopback.py -f vtestbed.yaml -i ../ansible/veos_vtb -e "--skip_sanity --disable_loganalyzer" -u === Running tests in groups === Running: pytest snmp/test_snmp_loopback.py --inventory ../ansible/veos_vtb --host-pattern vlab-01 --testbed vms-kvm-t0 --testbed_file vtestbed.yaml --log-cli-level warning --log-file-level debug --kube_master unset --showlocals --assert plain --show-capture no -rav --allow_recover --ignore=ptftests --ignore=acstests --ignore=saitests --ignore=scripts --ignore=k8s --ignore=sai_qualify --junit-xml=logs/tr.xml --log-file=logs/test.log --skip_sanity --disable_loganalyzer .. snmp/test_snmp_loopback.py::test_snmp_loopback[vlab-01] PASSED ``` <!-- If PR needs to be backported, then the PR must be tested against the base branch and the earliest backport release branch and provide tested image version on these two branches. For example, if the PR is requested for master, 202211 and 202012, then the requester needs to provide test results on master and 202012. --> #### Which release branch to backport (provide reason below if selected) <!-- - Note we only backport fixes to a release branch, *not* features! - Please also provide a reason for the backporting below. - e.g. - [x] 202006 --> - [ ] 201811 - [ ] 201911 - [ ] 202006 - [x] 202012 - [x] 202106 - [x] 202111 - [x] 202205 - [x] 202211 - [x] 202305 #### Tested branch (Please provide the tested image version) <!-- - Please provide tested image version - e.g. - [x] 20201231.100 --> - [ ] <!-- image version 1 --> - [ ] <!-- image version 2 --> #### Description for the changelog <!-- Write a short (one line) summary that describes the changes in this pull request for inclusion in the changelog: --> <!-- Ensure to add label/tag for the feature raised. example - PR#2174 under sonic-utilities repo. where, Generic Config and Update feature has been labelled as GCU. --> #### Link to config_db schema for YANG module changes <!-- Provide a link to config_db schema for the table for which YANG model is defined Link should point to correct section on https://github.com/Azure/sonic-buildimage/blob/master/src/sonic-yang-models/doc/Configuration.md --> #### A picture of a cute animal (not mandatory but encouraged) (cherry picked from commit 803c71c)
Signed-off-by: Suvarna Meenakshi <sumeenak@microsoft.com>
Signed-off-by: Suvarna Meenakshi <sumeenak@microsoft.com>
qiluo-msft
approved these changes
Sep 6, 2023
|
i do not recommend to backport this. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why I did it
Cherry pick of #15487 and #16013 along fixes in snmpd.conf for snmpd 5.7.3 version
SNMP over IPv6 is not working for all scenarios for a single asic platforms.
The expectation is that SNMP query over IPv6 should work over Management or Loopback0 addresses.
Specific scenario where this issue is seen
In case of Lab T0 device, when SNMP request is sent from a directly connected T1 neighbor over Loopback IP, SNMP response was not received.
This was because the SRC IP address in SNMP response was not Loopback IP, it was the PortChannel IP connected to the neighboring device.
In case of IPv4, the SRC IP in SNMP response was correctly set to Loopback IP.
Sequence of SNMP request and response
snmpd process does not track the DST IP on which the SNMP request was received, which in this case is Loopback IP.
snmpd process will only keep track what is tht IP to which the response should be sent to.
ip -6 route get fc00::72
fc00::72 from :: dev PortChannel101 proto kernel src fc00::71 metric 256 pref medium
The same issue is seen when SNMP query is sent from a remote server over Management IP.
SONiC device eth0 --------- Remote server
SNMP request comes with SRC IP <Remote_server> DST IP
If kernel finds best route to Remote_server_IP is via BGP neighbors, then it will send the response via front-panel interface with SRC IP as Loopback IP instead of Management IP.
Main issue is that in case of IPv6, snmpd ignores the IP address to which SNMP request was sent, in case of IPv6.
In case of IPv4, snmpd keeps track of DST IP of SNMP request, it will keep track if the SNMP request was sent to mgmt IP or Loopback IP.
Later, this IP is used in ipi_spec_dst as SRC IP which helps kernel to find the route based on DST IP using the right SRC IP.
https://github.com/net-snmp/net-snmp/blob/master/snmplib/transports/snmpUDPBaseDomain.c#L300
ipi.ipi_spec_dst.s_addr = srcip->s_addr
Reference: https://man7.org/linux/man-pages/man7/ip.7.html
This issue is not seen on multi-asic platform, why?
on multi-asic platform, there exists different network namespaces.
SNMP docker with snmpd process runs on host namespace.
Management interface belongs to host namespace.
Loopback0 is configured on asic namespaces.
Additional inforamtion on how the packet coming over Loopback IP reaches snmpd process running on host namespace: #5420
Because of this separation of network namespaces, the route lookup of destination IP is confined to routing table of specific namespace where packet is received.
if packet is received over management interface, SNMP response also is sent out of management interface. Same goes with packet received over Loopback Ip.
Work item tracking
How I did it
Have snmpd listen on specific Management and Loopback IPs specifically instead of listening on any IP for single-asic platform.
With this, snmpd will bind to different socket for each IP.
Any request coming in from Mgmt or Loopback IP will go through specific socket and response will have also use the same socket ensuring that the src IP of the response packet matches the dst ip of request packet.
How to verify it
on multi-asic image: no change.
Verified on singl-asic DUT with test_snmp_link_local and test_snmp_loopback test cases.
on single-asic VS image:
snmpd.conf compatible with snmpd 5.7.3:
snmpd.conf with linklocal ip:
Ran IPv6 SNMP test cases:
Which release branch to backport (provide reason below if selected)
Tested branch (Please provide the tested image version)
Description for the changelog
Link to config_db schema for YANG module changes
A picture of a cute animal (not mandatory but encouraged)