Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ERR syncd#syncd: :- collectPortCounters: Failed to get stats of port 0x100000071: -5 ---> Seen in master 281 #4586

Closed
mini-nair-dell opened this issue May 13, 2020 · 11 comments · Fixed by #4737
Closed

ERR syncd#syncd: :- collectPortCounters: Failed to get stats of port 0x100000071: -5 ---> Seen in master 281 #4586

mini-nair-dell opened this issue May 13, 2020 · 11 comments · Fixed by #4737
Assignees
@daall

Comments

@mini-nair-dell
Copy link

I see the syslogs of T0 S6100 filled with the below logs in the master image – 281. The same logs were not present in the image – 259.

This issue doesn’t look like the syncd crash because, the logs are always seen, and even when there is no crash.

The same is not seen in T1 TB.

root@sonic-s6100-07:/var/log# tail -f syslog
May 13 05:43:16.087433 sonic-s6100-07 ERR syncd#syncd: [none] brcm_sai_get_port_stats:2149 Multi stats get failed with error Invalid parameter (0xfffffffc).
May 13 05:43:16.087480 sonic-s6100-07 ERR syncd#syncd: :- collectPortCounters: Failed to get stats of port 0x100000071: -5
May 13 05:43:16.087527 sonic-s6100-07 ERR syncd#syncd: [none] brcm_sai_get_port_stats:2149 Multi stats get failed with error Invalid parameter (0xfffffffc).
May 13 05:43:16.087573 sonic-s6100-07 ERR syncd#syncd: :- collectPortCounters: Failed to get stats of port 0x100000072: -5
May 13 05:43:16.087619 sonic-s6100-07 ERR syncd#syncd: [none] brcm_sai_get_port_stats:2149 Multi stats get failed with error Invalid parameter (0xfffffffc).
May 13 05:43:16.087666 sonic-s6100-07 ERR syncd#syncd: :- collectPortCounters: Failed to get stats of port 0x100000073: -5
May 13 05:43:16.087712 sonic-s6100-07 ERR syncd#syncd: [none] brcm_sai_get_port_stats:2149 Multi stats get failed with error Invalid parameter (0xfffffffc).
May 13 05:43:16.087759 sonic-s6100-07 ERR syncd#syncd: :- collectPortCounters: Failed to get stats of port 0x100000074: -5
May 13 05:43:16.087805 sonic-s6100-07 ERR syncd#syncd: [none] brcm_sai_get_port_stats:2149 Multi stats get failed with error Invalid parameter (0xfffffffc).
May 13 05:43:16.087852 sonic-s6100-07 ERR syncd#syncd: :- collectPortCounters: Failed to get stats of port 0x100000075: -5
May 13 05:43:17.077916 sonic-s6100-07 ERR syncd#syncd: [none] brcm_sai_get_port_stats:2149 Multi stats get failed with error Invalid parameter (0xfffffffc).
May 13 05:43:17.077916 sonic-s6100-07 ERR syncd#syncd: :- collectPortCounters: Failed to get stats of port 0x100000001: -5
May 13 05:43:17.077916 sonic-s6100-07 ERR syncd#syncd: [none] brcm_sai_get_port_stats:2149 Multi stats get failed with error Invalid parameter (0xfffffffc).
May 13 05:43:17.077969 sonic-s6100-07 ERR syncd#syncd: :- collectPortCounters: Failed to get stats of port 0x100000002: -5
May 13 05:43:17.078137 sonic-s6100-07 ERR syncd#syncd: [none] brcm_sai_get_port_stats:2149 Multi stats get failed with error Invalid parameter (0xfffffffc).

root@sonic-s6100-07:/var/log# date
Wed 13 May 2020 05:43:54 AM UTC
root@sonic-s6100-07:/var/log# cd ../core

root@sonic-s6100-07:/var/core# ls -ltr
total 31740
-rw-rw-rw- 1 root root 3933970 May 11 18:21 zebra.1589221293.40.core.gz
-rw-rw-rw- 1 root root 3904129 May 11 18:52 zebra.1589223131.40.core.gz
-rw-rw-rw- 1 root root 3795330 May 11 20:33 zebra.1589229221.40.core.gz
drwxr-xr-x 2 root root 4096 May 12 07:58 old
-rw-rw-rw- 1 root root 10320167 May 12 07:58 syncd.1589270320.29.core.gz
-rw-rw-rw- 1 root root 10531380 May 12 12:22 syncd.1589286121.30.core.gz
@mini-nair-dell
Copy link
Author

syslog1-syncd.docx
syslog-syncd.docx

@mini-nair-dell
Copy link
Author

root@sonic-s6100-07:/var/core# bcmcmd "a"
a
Attach: Unit 0 (BCM56960_B1): attached (current unit)
drivshell>
root@sonic-s6100-07:/var/core# bcmcmd "version"
version
Broadcom Command Monitor: Copyright (c) 1998-2020 Broadcom
Release: sdk-6.5.16 built 20200417 (Fri Apr 17 02:10:18 2020)
From sonicbld@9cacac0fd10c:/var/sonicbld/workspace/Build/broadcom/broadcom_sai/20-sai-build-brcm-3.7/output/x86-xgs5-deb80//sdk/bcmsdk
Platform: X86
OS: Unix (Posix)
Chips:

   BCM56640_A0,
   BCM56850_A0,
   BCM56340_A0,
   BCM56960_A0, BCM56860_A0,




   BCM56970_A0, BCM56870_A0,
   BCM56980_A0, BCM56980_B0,

PHYs: BCM5400, BCM54182, BCM54185, BCM54180,
BCM54140, BCM54192, BCM54195, BCM54190,
BCM54194, BCM54210, BCM54220, BCM54280,
BCM54282, BCM54240, BCM54285, BCM5428X,
BCM54290, BCM54292, BCM54294, BCM54295,
BCM54296, BCM8750, BCM8752, BCM8754,
BCM84740, BCM84164, BCM84758, BCM84780,
BCM84784, BCM84318, BCM84328, Sesto,
copper sfp

drivshell>
root@sonic-s6100-07:/var/core# bcmcmd "soc"
soc
Unit 0 Driver Control Structure:
Chip=BCM56960_B1 Rev=0x12 Driver=BCM56960_A0
Flags=0x48103: attached initialized mem-clear-use-dma; board type 0x0
CM: Base=0x7ff6cb6f1000
Disabled: reg_flags=0x100 mem_flags=0x0
SchanOps=14968129 MMUdbg=0 LinkPause=2
Counter: int=2000000us per=2099324us dmaBuf=0x7ff6bd5a6000
EvictInvalidPoolIdCount=0
Timeout: Schan=0(300000us) MIIM=0(1000000us)
Intr: Total=1908022 Sc=0 ScErr=0 MMU/ARLErr=0
LinkStat=0 PCIfatal=0 PCIparity=0
ARLdrop=0 ARLmbuf=0 ARLxfer=0 ARLcnt0=0
TableDMA=1901932 TSLAM-DMA=6079 CCM-DMA=0 SW=0
MemCmd[BSE]=0 MemCmd[CSE]=0 MemCmd[HSE]=0
ChipFunc[0]=0 ChipFunc[1]=0 ChipFunc[2]=0
ChipFunc[3]=0 ChipFunc[4]=0
FifoDma[0]=0 FifoDma[1]=0 FifoDma[2]=0 FifoDma[3]=0
I2C=0 MII=0 StatsDMA=0 Desc=0 Chain=0 PciTimeOut=0
Error: SDRAM=0 CFAP=0 Fcell=0 MmuSR=0
SER events(mem=0 reg=0 nak=0 stat=0 ecc=0 direct=0 fifo=0 tcam=0)
SER corrections(fix=0 clear=0 restore=0 special=0 err:0)
PKT DMA: dcb=t32 tpkt=0 tbyt=0 rpkt=0 rbyt=0
DV: List: max-q=32 cur-tq=0 cur-rq=0 dv-size=160
DV: Statistics: allocs=8 frees=0 alloc-q=0
Mem cache (count=379 size=26393344 vmap size=188480 errmap size=8552)
Reg cache (count=30 size=3223200)
dma-ch-0 TX Idle Queue=0 ((nil)) default intr mbm
dma-ch-1 RX Active Queue=8 (0x5622a1aaf118) default intr mbm
dma-ch-2 RX Idle Queue=0 ((nil)) intr mbm
dma-ch-3 RX Idle Queue=0 ((nil)) intr mbm
dma-ch-4 -- Idle Queue=0 ((nil)) intr no-mbm
dma-ch-5 -- Idle Queue=0 ((nil)) intr no-mbm
dma-ch-6 -- Idle Queue=0 ((nil)) intr no-mbm
dma-ch-7 -- Idle Queue=0 ((nil)) intr no-mbm
dma-ch-8 -- Idle Queue=0 ((nil)) intr no-mbm
dma-ch-9 -- Idle Queue=0 ((nil)) intr no-mbm
dma-ch-10 -- Idle Queue=0 ((nil)) intr no-mbm
dma-ch-11 -- Idle Queue=0 ((nil)) intr no-mbm
drivshell>

root@sonic-s6100-07:/var/core# bcmcmd "ps"
ps
ena/ speed/ link auto STP lrn inter max cut loop
port link Lns duplex scan neg? state pause discrd ops face frame thru? back
xe0(104) up 2 40G FD SW No Forward None F CR4 9122 No
xe1(105) up 2 40G FD SW No Forward None F CR4 9122 No
xe2(102) !ena 2 40G FD SW No Forward None FA CR4 9122 No
xe3(103) !ena 2 40G FD SW No Forward None FA CR4 9122 No
xe4( 70) up 2 40G FD SW No Forward None F CR4 9122 No
xe5( 71) up 2 40G FD SW No Forward None F CR4 9122 No
xe6( 68) up 2 40G FD SW No Forward None FA CR4 9122 No
xe7( 69) up 2 40G FD SW No Forward None FA CR4 9122 No
xe8( 44) up 2 40G FD SW No Forward None FA CR4 9122 No
xe9( 45) up 2 40G FD SW No Forward None FA CR4 9122 No
xe10( 42) up 2 40G FD SW No Forward None FA CR4 9122 No
xe11( 43) up 2 40G FD SW No Forward None FA CR4 9122 No
xe12( 11) up 2 40G FD SW No Forward None FA CR4 9122 No
xe13( 12) up 2 40G FD SW No Forward None FA CR4 9122 No
xe14( 9) up 2 40G FD SW No Forward None FA CR4 9122 No
xe15( 10) up 2 40G FD SW No Forward None FA CR4 9122 No
xe16( 13) up 2 40G FD SW No Forward None F CR4 9122 No
xe17( 14) up 2 40G FD SW No Forward None F CR4 9122 No
xe18( 15) !ena 2 40G FD SW No Forward None FA CR4 9122 No
xe19( 16) !ena 2 40G FD SW No Forward None FA CR4 9122 No
xe20( 46) up 2 40G FD SW No Forward None F CR4 9122 No
xe21( 47) up 2 40G FD SW No Forward None F CR4 9122 No
xe22( 48) up 2 40G FD SW No Forward None FA CR4 9122 No
xe23( 49) up 2 40G FD SW No Forward None FA CR4 9122 No
xe24( 72) up 2 40G FD SW No Forward None FA CR4 9122 No
xe25( 73) up 2 40G FD SW No Forward None FA CR4 9122 No
xe26( 74) up 2 40G FD SW No Forward None FA CR4 9122 No
xe27( 75) up 2 40G FD SW No Forward None FA CR4 9122 No
xe28(106) up 2 40G FD SW No Forward None FA CR4 9122 No
xe29(107) up 2 40G FD SW No Forward None FA CR4 9122 No
xe30(108) up 2 40G FD SW No Forward None FA CR4 9122 No
xe31(109) up 2 40G FD SW No Forward None FA CR4 9122 No
xe32( 7) up 2 40G FD SW No Forward None FA CR4 9122 No
xe33( 8) !ena 2 40G FD SW No Forward None FA CR4 9122 No
xe34( 5) !ena 2 40G FD SW No Forward None FA CR4 9122 No
xe35( 6) !ena 2 40G FD SW No Forward None FA CR4 9122 No
xe36(116) up 2 40G FD SW No Forward None FA CR4 9122 No
xe37(117) up 2 40G FD SW No Forward None FA CR4 9122 No
xe38(114) up 2 40G FD SW No Forward None FA CR4 9122 No
xe39(115) up 2 40G FD SW No Forward None FA CR4 9122 No
xe40( 82) up 2 40G FD SW No Forward None FA CR4 9122 No
xe41( 83) up 2 40G FD SW No Forward None FA CR4 9122 No
xe42( 80) up 2 40G FD SW No Forward None FA CR4 9122 No
xe43( 81) !ena 2 40G FD SW No Forward None FA CR4 9122 No
xe44( 40) !ena 2 40G FD SW No Forward None FA CR4 9122 No
xe45( 41) !ena 2 40G FD SW No Forward None FA CR4 9122 No
xe46( 38) !ena 2 40G FD SW No Forward None FA CR4 9122 No
xe47( 39) !ena 2 40G FD SW No Forward None FA CR4 9122 No
xe48(110) up 2 40G FD SW No Forward None FA CR4 9122 No
xe49(111) !ena 2 40G FD SW No Forward None FA CR4 9122 No
xe50(112) !ena 2 40G FD SW No Forward None FA CR4 9122 No
xe51(113) !ena 2 40G FD SW No Forward None FA CR4 9122 No
xe52( 1) up 2 40G FD SW No Forward None FA CR4 9122 No
xe53( 2) up 2 40G FD SW No Forward None FA CR4 9122 No
xe54( 3) up 2 40G FD SW No Forward None FA CR4 9122 No
xe55( 4) up 2 40G FD SW No Forward None FA CR4 9122 No
xe56( 34) up 2 40G FD SW No Forward None FA CR4 9122 No
xe57( 35) up 2 40G FD SW No Forward None FA CR4 9122 No
xe58( 36) up 2 40G FD SW No Forward None FA CR4 9122 No
xe59( 37) !ena 2 40G FD SW No Forward None FA CR4 9122 No
xe60( 76) !ena 2 40G FD SW No Forward None FA CR4 9122 No
xe61( 77) !ena 2 40G FD SW No Forward None FA CR4 9122 No
xe62( 78) !ena 2 40G FD SW No Forward None FA CR4 9122 No
xe63( 79) !ena 2 40G FD SW No Forward None FA CR4 9122 No
drivshell>

@mini-nair-dell
Copy link
Author

root@sonic-s6100-07:~# show ver

SONiC Software Version: SONiC.master.281-286aa35a
Distribution: Debian 10.4
Kernel: 4.19.0-6-amd64
Build commit: 286aa35
Build date: Sun May 10 11:13:59 UTC 2020
Built by: johnar@jenkins-worker-8

Platform: x86_64-dell_s6100_c2538-r0
HwSKU: Force10-S6100

@yxieca
Copy link
Contributor

yxieca commented May 13, 2020

@mini-nair-dell please attach show techsupport to this ticket. We need the logs to triage this issue.

@rlhui rlhui assigned daall and unassigned rlhui May 13, 2020
@mini-nair-dell
Copy link
Author

the show tech is > 10 MB and unable to attach.

@yxieca
Copy link
Contributor

yxieca commented May 18, 2020

@mini-nair-dell can you use box to upload and share a link?

@mini-nair-dell
Copy link
Author

sonic_dump_sonic-s6100-07_20200527_130305.tar.gz

@mini-nair-dell
Copy link
Author

Pls find the show tech attached

@xinliu-seattle
Copy link
Contributor

@daall please take a look.

@daall
Copy link
Contributor

daall commented May 27, 2020

ack. Thank you for the show tech, I'll take a look today and try to sort this out.

@daall
Copy link
Contributor

daall commented Jun 2, 2020

It looks like it's related to this change: sonic-net/sonic-swss#1237.

It looks like there is some sort of issue polling those two counters introduced in the 3.7.3.3-4 SAI update, but this issue is only present in master, not 201911.

We're going to need do more investigating to figure out what exactly is wrong here, but for now I've opened a PR to revert the counter change to clean-up the master branch: sonic-net/sonic-swss#1308

Once it merges I can update the submodule and resolve this issue.

@daall daall mentioned this issue Jun 9, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@daall daall

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[submodule] Update swss submodule daall/sonic-buildimage
5 participants
@yxieca @xinliu-seattle @rlhui @daall @mini-nair-dell