Skip to content

Commit

Permalink
Merge pull request sonic-net#267 from michaelli10/cpu_packet_debug_co…
Browse files Browse the repository at this point in the history 
…unters

Updating CPU Packet Debug Counters HLD
  • Loading branch information
@michaelli10
michaelli10 authored Sep 1, 2021
2 parents 6599556 + 900c900 commit 05a797d
Showing 1 changed file with 320 additions and 34 deletions.
354 changes: 320 additions & 34 deletions system/SONiC_CPU_Packet_Debug_Counters.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,8 @@ SONiC CPU Packet Debug Counters
# Revision
| Rev | Date | Author | Change Description |
|:---:|:-----------:|:------------------:|-----------------------------------|
| 0.1 | 04/13/2019 | Michael Li | Initial version |
| 0.1 | 04/13/2021 | Michael Li | Initial version |
| 0.2 | 08/16/2021 | Michael Li | Design detail updates |

# About this Manual
This document provides general information about the SONiC CPU Packet Debug Counters feature implementation in SONiC.
Expand All @@ -28,50 +29,335 @@ This document describes the high level design of SONiC CPU Packet Debug Counters
### Table 1: Abbreviations
| **Term** | **Meaning** |
|--------------------------|-------------------------------------|
| XYZ | Term description |
| KNET | Kernel Network Driver |

# 1 Feature Overview
The CPU Packet Debug Counters provides BRCM KNET driver level Rx and Tx statistics for each protocol type. SONiC KNET currently provides CPU packet counters based on CPU queues which indirectly provides Rx per protocol counters since each protocol is assigned a separate CPU queue. This feature enhances CPU packet statistics by adding per protocol counters for CPU Rx and Tx directions.
The CPU Packet Debug Counters provides BRCM KNET driver level Rx and Tx statistics for each protocol type at switch and interface levels. SONiC KNET currently provides CPU packet counters based on CPU queues which indirectly provides Rx per protocol counters since each protocol is assigned a separate CPU queue. This feature enhances CPU packet statistics by adding per protocol counters per interface for CPU Rx and Tx directions. Packet drop reason details per protocol/queue per interface are also added for more debug visibility.

## 1.1 Requirements

### 1.1.1 Functional Requirements

1. Support Tx and Rx per protocol KNET driver level packet counters from KNET procfs
2. Support Tx and Rx per protocol Linux/KNET dropped packet counters from KNET procfs
3. Support clearing of Tx and Rx counters
4. Output of statistics will be similar to the following format:
1. Support switch and per interface Tx and Rx per protocol KNET driver level packet counters
2. Support switch and per interface Tx and Rx per protocol Linux/KNET dropped packet counters
3. Support switch and per interface Tx and Rx KNET error reason detail counters
4. Support switch and per interface clearing of Tx and Rx counters

### 1.1.2 Configuration and Management Requirements
Statistics will be provided via click CLIs and KNET procfs files

### 1.1.3 Scaling and Performance Requirements

There should be minimal impact on CPU Pkt IO performance and latencies. CPU Pkt IO performance will be profiled to measure the impact of adding pkt type processing on CPU Pkt IO performance and latencies.

### 1.1.4 Warm Boot Requirements
Not applicable

# 2 Functionality
## 2.1 Target Deployment Use Cases
This debugging enhancement provides more granular debug counters and additional visibility into the CPU Pkt IO network driver (KNET) and Linux kernel pkt path that will help with debugging SONiC systems.

## 2.2 Functional Description
See feature overview

# 3 Design
## 3.1 Overview
Protocol classification logic will be added to the Broadcom KNET driver pkt Tx and Rx callbacks to identify protocols and enable pkt protocol type accounting per physical interface. The statistics output will be available via KNET driver procfs files and click CLIs for convenience.

### 3.1.1 Rx Classification and Packet drop counting
The Broadcom KNET driver filter infrastructure will be used to classify Rx packets (and CPU queues from pkt metadata) and tag the protocol type from added logic in the KNET callback module. This will allow managing KNET packet stats accounting to have per protocol and per interface information when managing stats accounting.

Any drops before KNET filter processing (matching src port) will not have src port information so pkt drops will count towards the global protocol switch counters and will not update the protocol interface counters. After KNET filter processing determines the src port, both global and per interface protocol counters will be updated.

### 3.1.2 Tx Classification
The Broadcom KNET driver will be modified to provide a similar KNET filter infrastructure as the Rx path to allow classification of pkt types as they are received by the KNET driver from the Linux kernel. The pkt skbuff will have destination port information from the Linux kernel so per interface protocol counters and any drop counters will be updated.

### 3.1.3 Protocol Classification Criteria
Packets will be parsed and classified according to the criteria in the following table. Any packets that cannot be classified will have counters updated in the "unknown" category.

References:
- https://www.iana.org/assignments/ieee-802-numbers/ieee-802-numbers.xhtml
- https://www.iana.org/assignments/protocol-numbers/protocol-numbers.xhtml
- https://www.iana.org/assignments/service-names-port-numbers/service-names-port-numbers.xhtml
- SAI Hostinterface protocol qualifier implementations

### **Table 1: Protocol Classification Criteria**

| Protocols | Queue | Classification Criteria
|----------------------|-------|----------------------------
| LACP | 23 | (EType==0x8809) && (MAC DA == {01:80:C2:00:00:02})
| UDLD | 22 | (MAC DA == {01:00:0C:CC:CC:CC})
| STP | 21 | (MAC DA == {01:80:C2:00:00:00})
| PVRSTP | 21 | (MAC DA == {01:00:0C:CC:CC:CD}
| BFDv4 | 20 | (EType==0x800) && (IpProtocol == 17) && (L4DstPort == 3784 or 4784)
| BFDv6 | 20 | (EType==0x86DD) && (IpProtocol == 17) && (L4DstPort == 3784 or 4784)
| PTP | 19 | (EType==0x88F7)
| LLDP | 18 | (EType==0x88CC) && (MAC DA == {01:80:C2:00:00:00} or {01:80:C2:00:00:03} or {01:80:C2:00:00:0e})
| VRRP | 17 | (EType==0x800) && (IpProtocol == 112)
| ICCP | 16 | (IpProtocol == 6) && (IpProtocol == 8888)
| OSPFv4 | 15 | (EType==0x800) && (IpProtocol == 89)
| BGPv4 | 14 | (EType==0x800) && (IpProtocol == 6) && (L4Port == 179)
| BGPv6 | 14 | (EType==0x86DD) && (IpProtocol == 6) && (L4Port == 179)
| PIM | 13 | (IpProtocol == 103)
| IGMP | 12 | (EType==0x800) && (IpProtocol == 2)
| ARP Request | 10 | (EType == 0x806) && (ArpOpCode == 1)
| ARP Reply | 10 | (EType == 0x806) && (ArpOpCode == 2)
| ARP | 10 | (EType == 0x806)
| NDP | 10 | (EType == 0x86DD) && (IPv6NextHeader == 58) && (IPv6HopLimit == 255) && (Type Code is 133 ~ 137)
| DHCPv4 | 9 | (EType == 0x800) && (IpProtocol == 17) && ((L4SrcPort == 68) && (L4DstPort == 67)) or (L4SrcPort == 67) && (L4DstPort == 68))
| DHCPv6 | 9 | (EType == 0x86DD) && (IpProtocol == 17) && (L4DstPort == 547 or 546)
| ICMP Echo Request | 8 | (IpProtocol == 1) && (ICMP type == 0)
| ICMP Echo Reply | 8 | (IpProtocol == 1) && (ICMP type == 8)
| ICMP | 8 | (IpProtocol == 1)
| SSH | 7 | (IpProtocol == 6) && ((L4Port == 22))|
| INET | 7 | (Etype==0x800)
| INETv6 | 7 | (Etype==0x88DD)
| Subnet | 6 | Not classified (Rx queue counters only)
| NAT | 5 | Not classified (Rx queue counters only)
| MTU | 4 | Not classified (Rx queue counters only)
| Sflow | 3 | Rx: Pkt sample metadata, Tx: (L4Port == 6343 for inband sflow datagram to collector)
| TTL | 0 | TTL == 0 or TTL == 1

### 3.1.4 Packet Drop Error types
Errors in KNET (and in Rx kernel path) which result in packet drops are counted per protocol/queue on each interface. Some common errors are listed below.
### **Table 2: Packet Drop Errors**
| Error | Description
|----------------------|-----------------------------------
| LINK_DOWN | Kernel network interface was in down state
| NO_SKB | Packet SKB alloc failed
| NO_BUFFER | No DMA buffer resource available (congested PktIO)
| KERNEL_DROP | Rx packet drop in Linux kernel network stack
| NO_FILTER_MATCH | Rx packet did not match any KNET filters (no handlers)
| UNKN_NETIF | Unknown src network interface for Rx packet
| HW_RESET | Packet dropped due to HW reset cleanup in progress

## 3.2 DB Changes
There are no new changes to SONiC DBs.

## 3.3 Switch State Service Design
### 3.3.1 Orchestration Agent
There are no new changes to Orchagent.
## 3.4 SyncD
There are no new changes to SyncD.

## 3.5 Manageability
There are no new changes to manageability infrastructure

## 3.6 CLI
KNET Packet stats at switch and interface levels can be shown through click commands or by dumping KNET pkt_stats procfs file. Rx and Tx error counter details are available for protocol and rx queue counters at switch and interface levels.

### 3.6.1 Protocol Stats
* Show protocol stats (real output will not show entries with zero counts)
```
DUT# cat /proc/bcm/knet-cb/pkt_stats
DUT# show knet stats protocol
or
DUT# cat /proc/bcm/knet/stats/protocol
Total rx = 101127
Total tx = 235644
protocol rx rx_err tx tx_err
--------------------------------------------------
UDLD 0 0 0 0
STP 0 0 0 0
BFD 0 0 0 0
PTP 0 0 0 0
LLDP 24291 0 54 0
VRRP 0 0 0 0
ICCP 0 0 0 0
OSPF 0 0 0 0
BGP 0 0 0 0
PIM 0 0 0 0
IGMP 0 0 0 0
ARP_SUPPRESS 0 0 0 0
ARP 8953 0 9103 0
DHCP 0 0 0 0
ICMP 0 0 0 0
IP2ME 0 0 0 0
IP2ME_SUBNET 0 0 0 0
NAT 0 0 0 0
IPMTU 0 0 0 0
SFLOW 0 0 0 0
DROP 0 0 0 0
UNSPEC 0 0 0 0
protocol rx rx_err tx tx_err
------------------------------------------------------------
LACP 0 0 0 0
UDLD 0 0 0 0
STP 0 0 0 0
PVRSTP 0 0 0 0
BFDv4 0 0 0 0
BFDv6 0 0 0 0
PTP 0 0 0 0
LLDP 24291 0 54 0
VRRP 0 0 0 0
ICCP 0 0 0 0
OSPF 0 0 0 0
BGPv4 0 0 0 0
BGPv6 0 0 0 0
PIM 0 0 0 0
IGMP 0 0 0 0
ARP Req 4428 0 4428 0
ARP Rply 4428 0 4428 0
ARP 8856 0 8856 0
NDP 0 0 0 0
DHCPv4 0 0 0 0
DHCPv6 0 0 0 0
ICMP EchoReq 0 0 0 0
ICMP EchoRply 0 0 0 0
ICMP 0 0 0 0
SSH 0 0 0 0
SFLOW 0 0 0 0
TTL 0 0 0 0
INETv4 0 0 0 0
INETv6 0 0 0 0
UNKNOWN 0 0 0 0
```
* Show interface level protocol stats
```
DUT# show knet Ethernet0 stats protocol
or
DUT# cat /proc/bcm/knet/Ethernet0/stats/protocol
Total Ethernet0 rx = 101127
Total Ethernet0 tx = 235644
protocol rx rx_err tx tx_err
------------------------------------------------------------
LACP 0 0 0 0
UDLD 0 0 0 0
STP 0 0 0 0
PVRSTP 0 0 0 0
BFDv4 0 0 0 0
...
```
* Show interface level protocol stats with error details
```
DUT# show knet Ethernet0 stats protocol details
or
DUT# echo "detail=1" > /proc/bcm/knet/Ethernet0/stats/protocol
DUT# cat /proc/bcm/knet/Ethernet0/stats/protocol
DUT# echo "detail=0" > /proc/bcm/knet/Ethernet0/stats/protocol
Total Ethernet0 rx = 101127
Total Ethernet0 tx = 235644
protocol rx rx_err tx tx_err
------------------------------------------------------------
LACP 0 0 0 0
UDLD 0 0 0 0
STP 0 0 0 0
PVRSTP 685 222 0 193
LINK_DOWN 208 168
NO_SKB 2 5
NO_BUFFER 0 20
KERNEL_DROP 12 0
BFDv4 0 0 0 0
...
```

### 1.1.2 Configuration and Management Requirements
Statistics will be provided via KNET procfs file

* Clear protocol pkt statistics
```
DUT# sonic-clear knet stats protocol
DUT# sonic-clear knet Ethernet0 stats protocol
or
DUT# echo "clear" > /proc/bcm/knet/stats/protocol
DUT# echo "clear" > /proc/bcm/knet/Ethernet0/stats/protocol
```
### 3.6.2 Rx Queue Stats
```
DUT# show knet stats rx_queue
or
DUT# cat /proc/bcm/knet/stats/rx_queue
Total rx = 24918
name queue rx rx_err
----------------------------------------------
SFLOW 3 0 0
ARP 10 238 0
LLDP 18 24291 0
LACP 23 389 0
```

* Show interface level rx queue stats
```
DUT# show knet Ethernet0 stats rx_queue
or
DUT# cat /proc/bcm/knet/Ethernet0/stats/rx_queue
Total Ethernet0 rx = 24918
name queue rx rx_err
----------------------------------------------
SFLOW 3 0 0
ARP 10 238 0
LLDP 18 24291 0
LACP 23 389 0
```

* Show interface level rx queue stats with error details
```
DUT# show knet Ethernet0 stats rx_queue details
or
DUT# echo "detail=1" > /proc/bcm/knet/Ethernet0/stats/rx_queue
DUT# cat /proc/bcm/knet/Ethernet0/stats/rx_queue
DUT# echo "detail=0" > /proc/bcm/knet/Ethernet0/stats/rx_queue
Total Ethernet0 rx = 24918
name queue rx rx_err
----------------------------------------------
SFLOW 3 0 0
ARP 10 238 220
LINK_DOWN 208
KERNEL_DROP 12
LLDP 18 24291 15
KERNEL_DROP 15
LACP 23 389 0
DEFAULT 0 0 7
NO_FILTER_MATCH 2
UNKN_NETIF 2
UNKN_DEST_TYPE 3
```

* Clear Rx Queue stats
```
DUT# sonic-clear knet stats rx_queue
DUT# sonic-clear knet Ethernet0 stats rx_queue
or
DUT# echo "clear" > /proc/bcm/knet/stats/rx_queue
DUT# echo "clear" > /proc/bcm/knet/Ethernet0/stats/rx_queue
```

# 4 Flow Diagrams

# 5 Error Handling
Not applicable

# 6 Serviceability and Debug
The new debug counters introduced at switch and interface levels (with drop details) will be added to techsupport collection with 2 samples each

# 7 Warm Boot Support
No warmboot support. CPU Pkt IO counters are maintained in KNET kernel network driver which gets cleared on CPU reset on a warmboot operation.

# 8 Scalability
## 8.1 Profiling and Tuning
The CPU packet IO performance impact on throughput and latency with the addition of processing to identify pkt type for counters will be profiled. It is expected there will be minimal impact.

# 9 Unit Test
## 9.1 Functional Test Cases
### 9.1.1 Protocol Stats
1. Verify Rx and Tx pkt counters on switch/interface levels for each protocol
2. Verify KERNEL_DROP Rx drop counter on switch/interface levels for selected protocols
3. Verify LINK_DOWN Rx and Tx drop counters on switch/interface levels for selected protocols
4. Verify NO_BUFFER Tx drop counters on switch/interface levels for selected protocols
5. Verify clear stats for switch and interface level protocol counters

### 9.1.2 Rx Queue Stats
1. Verify Rx queues and description match COPP config
2. Verify Rx queue pkt counters on switch/interface levels for each protocol
3. Verify KERNEL_DROP Rx queue drop counter on switch/interface levels for selected protocols
4. Verify LINK_DOWN Rx queue drop counters on switch/interface levels for selected protocols
5. Verify clear stats for switch and interface level Rx queue counters


## 9.2 Warm Boot Test Cases
Not applicable

## 9.3 Negative Test Cases

# 10 Internal Design Information
## 10.1 Guidance on simulating triggers for testing packet drops
### 10.1.1 LINK_DOWN pkt drops
* The following command can directly set netdevice link status down while physical link is up. Packets will continue to be punted up to CPU but will drop with LINK_DOWN reason. CPU Tx direction should also show drops
```
DUT# echo "Ethernet0=down" > /proc/bcm/knet/link
DUT# cat /proc/bcm/knet/link
Software link status:
Ethernet0 down
```
### 10.1.2 KERNEL_DROP Rx pkt drops
* Send LACP packets to the DUT without the src port configured on a portchannel. The Linux kernel will drop the packet since the src port has not been registered as a portchannel member in the teamd kernel module (portchannel kernel component).

### 10.1.3 NO_BUFFER Tx pkt drops
* Decrease the KNET Tx buffers to a low value (8 or 4) to increase chance of congestion then send flood ping to congest the CPU Tx path and induce out of DMA buffer resource Tx drops. This operation should be used for testing only with all interfaces down. Changing the max number of Tx buffers may cause a crash if packets are being processed in KNET.
```
echo "max_tx_dcbs=4" > /proc/bcm/knet/dma
```
* Then check the Tx high watermark stats after sending CPU Tx burst.
```
DUT# cat /proc/bcm/knet/dstats | grep "Tx used DCBs hi wm"
Tx used DCBs hi wm 4
```

0 comments on commit 05a797d

Please sign in to comment.