Netdevice Statistics

Table of Contents

1. Link Statistics
2. CPU Hit Statistics
3. L3 HW Statistics
4. ethtool Statistics

Features by Version

Kernel Version
5.18	L3 HW Statistics

Link Statistics

On netdevices representing front panel ports, link statistics include the traffic taking place in the HW datapath. On any other netdevice, it only includes the traffic taking place in SW datapath.

If ip stats is available, it can be used to obtain link statistics under group link:

# ip stats show dev swp6 group link
53: swp6: group link
    RX:  bytes packets errors dropped  missed   mcast
      10002698    1284      0       0       0      31
    TX:  bytes packets errors dropped carrier collsns
        306328    4707      0       0       0       0

Otherwise it is always possible to use ip -s link show:

# ip -s link show dev swp6
53: swp6: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel state DOWN mode DEFAULT group default qlen 1000
    link/ether 1c:34:da:c6:35:63 brd ff:ff:ff:ff:ff:ff
    RX:  bytes packets errors dropped  missed   mcast
      10002698    1284      0       0       0      31
    TX:  bytes packets errors dropped carrier collsns
        306328    4707      0       0       0       0

CPU Hit Statistics

CPU hit statistics account for packets trapped to the CPU or packets sent from the CPU. If ip stats is available, it can be used to display the statistics:

# ip stats show dev swp6 group offload subgroup cpu_hit
53: swp6: group offload subgroup cpu_hit
    RX:  bytes packets errors dropped  missed   mcast
          3136      29      0       0       0       0
    TX:  bytes packets errors dropped carrier collsns
          3256      30      0       0       0       0

Otherwise, ifstat can be used:

# ifstat -a -x cpu_hit swp6
#kernel
Interface        RX Pkts/Rate    TX Pkts/Rate    RX Data/Rate    TX Data/Rate
                 RX Errs/Drop    TX Errs/Drop    RX Over/Rate    TX Coll/Rate
swp6                  29 0            30 0          3136 0          3256 0
                       0 0             0 0             0 0             0 0

L3 HW Statistics

The Spectrum ASIC permits installation of traffic counters on router interfaces. As of Linux 5.18, installation of these counters is possible through the mechanism of "L3 HW statistics". These can be enabled on any device, but mlxsw will only actually install the counters on netdevices for which it has a corresponding router interface. Use ip stats to configure L3 HW stats:

# ip -br address show dev swp2
swp2             UP             192.0.2.2/28 fe80::1e34:daff:fec6:356b/64
# ip stats set dev swp2 l3_stats on

To see whether the counter was installed (used), inspect group offload, subgroup hw_stats_info in ip stats:

# ip stats show dev swp2 group offload subgroup hw_stats_info
57: swp2: group offload subgroup hw_stats_info
    l3_stats on used on

To read the stats, use group offload, subgroup l3_stats. This also shows whether the stats are installed at the moment:

# ip stats show dev swp2 group offload subgroup l3_stats
57: swp2: group offload subgroup l3_stats on used on
    RX: bytes packets errors dropped   mcast
        22416     267     13       0       1
    TX: bytes packets errors dropped
        22344     266      0       0

The stats do not go away when the router interface disappears:

# ip -br address show dev swp2
swp2             DOWN
# ip stats show dev swp2 group offload subgroup l3_stats
57: swp2: group offload subgroup l3_stats on used off
    RX: bytes packets errors dropped   mcast
        27792     331     16       0       1
    TX: bytes packets errors dropped
        27720     330      0       0

Only when the counter is disabled again will the statistics be wiped:

# ip stats set dev swp2 l3_stats off
# ip stats show dev swp2 group offload subgroup l3_stats
57: swp2: group offload subgroup l3_stats off used off

ethtool Statistics

Front panel ports support a number of statistical counters. These account for all packets going through the port, including those trapped to or originating from the CPU.

For example:

$ ethtool -S sw1p5
NIC statistics:
     a_frames_transmitted_ok: 8500
     a_frames_received_ok: 772
     a_frame_check_sequence_errors: 0
     a_alignment_errors: 0
     a_octets_transmitted_ok: 874212
     a_octets_received_ok: 67968
     a_multicast_frames_xmitted_ok: 308
     a_broadcast_frames_xmitted_ok: 0
     a_multicast_frames_received_ok: 290
     a_broadcast_frames_received_ok: 0
     a_in_range_length_errors: 0
     a_out_of_range_length_field: 0
     a_frame_too_long_errors: 0
     a_symbol_error_during_carrier: 0
     a_mac_control_frames_transmitted: 0
     a_mac_control_frames_received: 0
     a_unsupported_opcodes_received: 0
     a_pause_mac_ctrl_frames_received: 0
     a_pause_mac_ctrl_frames_xmitted: 0
     if_in_discards: 0
     if_out_discards: 0
     if_out_errors: 0
     ether_stats_undersize_pkts: 0
     ether_stats_oversize_pkts: 0
     ether_stats_fragments: 0
     ether_pkts64octets: 0
     ...
     ether_pkts65to127octets: 0
     ...
     dot3stats_fcs_errors: 0
     dot3stats_symbol_errors: 0
     dot3control_in_unknown_opcodes: 0
     dot3in_pause_frames: 0
     discard_ingress_general: 0
     discard_ingress_policy_engine: 0
     discard_ingress_vlan_membership: 0
     discard_ingress_tag_frame_type: 0
     discard_egress_vlan_membership: 0
     discard_loopback_filter: 0
     discard_egress_general: 0
     discard_egress_hoq: 0
     discard_egress_policy_engine: 0
     discard_ingress_tx_link_down: 0
     discard_egress_stp_filter: 0
     discard_egress_sll: 0
     rx_octets_prio_0: 67968
     rx_frames_prio_0: 772
     tx_octets_prio_0: 874212
     tx_frames_prio_0: 8500
     rx_pause_prio_0: 0
     rx_pause_duration_prio_0: 0
     tx_pause_prio_0: 0
     tx_pause_duration_prio_0: 0
     ...
     tc_transmit_queue_tc_0: 0
     tc_no_buffer_discard_uc_tc_0: 0
     ...
     transceiver_overheat: 0

Notes

1. a_frames_transmitted_ok: Includes PAUSE frames transmitted by the port. This applies for a_octets_transmitted_ok as well.

2. a_frames_received_ok: Includes packets later discarded due to insufficient space in the port's headroom or not admitted to the switch's shared buffer. This applies for a_octets_received_ok as well.

3. a_pause_mac_ctrl_frames_received: Includes both PAUSE and PFC frames. This applies for a_pause_mac_ctrl_frames_xmitted as well.

4. As part of [RFC 2863][4]:

   if_in_discards - The number of inbound packets which were chosen to be discarded even though no errors had been detected to prevent them from being deliverable to a higher-layer protocol.

   if_out_discards - The number of outbound packets which were chosen to be discarded even though no errors had been detected to prevent them from being transmitted.

   if_out_errors - The number of outbound packets that could not be transmitted because of errors.

5. As part of [RFC 2819][5]:

   ether_stats_undersize_pkts - The total number of packets received that were less than 64 octets long (excluding framing bits, but including FCS octets) and were otherwise well formed.

   ether_stats_oversize_pkts - The total number of packets received that were longer than MTU octets (excluding framing bits, but including FCS octets) but were otherwise well formed.

   ether_stats_fragments - The total number of packets received that were less than 64 octets in length (excluding framing bits but including FCS octets) and had either a bad FCS with an integral number of octets (FCS error) or a bad FCS with a non-integral number of octets (alignment error).

   ether_pkts64octets - The total number of packets (including bad packets) received that were 64 octets in length (excluding framing bits but including FCS octets).

   ether_pkts<X>to<Y>octets - The total number of packets (including bad packets) received that were between X and Y octets in length (excluding framing bits but including FCS octets).

6. As part of [RFC 3635][6]:

   dot3stats_fcs_errors - A count of frames received that are an integral number of octets in length but do not pass the FCS check. This count does not include frames received with frame-too-long or frame-too-short errors.

   dot3stats_symbol_errors - The number of times the receiving media is non-idle (a carrier event) for a period of time equal to or greater than minFrameSize, and during which there was at least one occurrence of an event that causes the PHY to indicate 'Receive Error'.

   dot3control_in_unknown_opcodes - A count of MAC Control frames received that contain an opcode that is not supported.

   dot3in_pause_frames - count of MAC Control frames received with an opcode indicating the PAUSE operation.

7. Hardware specific discard counters:

   discard_egress_general - In Spectrum, counts only MTU discards.

   discard_egress_hoq - Head-of-Queue time-out discards.

   discard_egress_sll - Number of packets dropped, because the Switch Lifetime Limit was exceeded.

8. rx_pause_prio_X: Number of PFC frames received from the far-end port with priority X. PAUSE frames increment the counters of all priorities.

9. rx_pause_duration_prio_X: The total time in microseconds in which transmission of packets with priority X to the far-end port has been paused. PAUSE frames increment the counters of all priorities.

10. tx_pause_prio_X: Number of PFC frames sent to the far-end port with priority X. PAUSE frames increment the counters of all priorities.

11. tx_pause_duration_prio_X: The total time in microseconds that transmission of packets with priority X from the far-end port has been requested to pause.

12. tc_transmit_queue_tc_X: The transmit queue depth in bytes of traffic class X.

13. tc_no_buffer_discard_uc_tc_X: The number of unicast packets with traffic class X dropped due to lack of shared buffer resources.

14. ptp_X: on Spectrum-1, number of garbage-collected PTP packets and timestamps. See PTP: Untimestamped Packets for details.

15. transceiver_overheat: The number of times a transceiver's temperature was higher than its configured threshold.