removed call to saidump to fix timeouts to redis-db during techsupport #696
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
In a node with tons of configuration, issuing techsupport results in call to saidump which locks redis-db for a long time causing other daemons to timeout and crash. This commit removes the saidump call as the same information is already saved in call to redis-dump.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In a node with tons of configuration, issuing techsupport results in
call to saidump which locks redis-db for a long time causing other
daemons to timeout and crash. This commit removes the saidump call as
the same information is already saved in call to redis-dump.
- What I did
fix timeout errors that causes orchagent to crash during show techsupport when switch has a lot of configuration.
the crash in orchagent is attributed to redis-db in techsupport script that is locking up redis-db for a long time trying to retrieve data.
- How I did it
removed call to saidump to fix timeout errors to redis-db during techsupport. There is a redis-dump call that is already retrieving same data prior to call to redis-db
- How to verify it
add a lot configuration (i.e. max ip addresses) then issue techsupport
- Previous command output (if the output of a command-line utility has changed)
admin@sonic:~$ show techsupp
main
mkdir: created directory '/var/dump/sonic_dump_sonic_20190721_130617'
'/var/dump/sonic_dump_sonic_20190721_130617/generate_dump' -> '/usr/bin/generate_dump'
sonic_dump_sonic_20190721_130617/
sonic_dump_sonic_20190721_130617/generate_dump
mkdir: created directory '/var/dump/sonic_dump_sonic_20190721_130617/proc'
'/proc/buddyinfo' -> '/var/dump/sonic_dump_sonic_20190721_130617/proc/buddyinfo'
'/proc/cmdline' -> '/var/dump/sonic_dump_sonic_20190721_130617/proc/cmdline'
'/proc/consoles' -> '/var/dump/sonic_dump_sonic_20190721_130617/proc/consoles'
'/proc/cpuinfo' -> '/var/dump/sonic_dump_sonic_20190721_130617/proc/cpuinfo'
'/proc/devices' -> '/var/dump/sonic_dump_sonic_20190721_130617/proc/devices'
'/proc/diskstats' -> '/var/dump/sonic_dump_sonic_20190721_130617/proc/diskstats'
'/proc/dma' -> '/var/dump/sonic_dump_sonic_20190721_130617/proc/dma'
'/proc/interrupts' -> '/var/dump/sonic_dump_sonic_20190721_130617/proc/interrupts'
'/proc/iomem' -> '/var/dump/sonic_dump_sonic_20190721_130617/proc/iomem'
'/proc/ioports' -> '/var/dump/sonic_dump_sonic_20190721_130617/proc/ioports'
'/proc/kallsyms' -> '/var/dump/sonic_dump_sonic_20190721_130617/proc/kallsyms'
'/proc/loadavg' -> '/var/dump/sonic_dump_sonic_20190721_130617/proc/loadavg'
'/proc/locks' -> '/var/dump/sonic_dump_sonic_20190721_130617/proc/locks'
'/proc/meminfo' -> '/var/dump/sonic_dump_sonic_20190721_130617/proc/meminfo'
'/proc/misc' -> '/var/dump/sonic_dump_sonic_20190721_130617/proc/misc'
'/proc/modules' -> '/var/dump/sonic_dump_sonic_20190721_130617/proc/modules'
'/proc/self/mounts' -> '/var/dump/sonic_dump_sonic_20190721_130617/proc/mounts'
'/proc/self/net' -> '/var/dump/sonic_dump_sonic_20190721_130617/proc/net'
:
:
:
sonic_dump_sonic_20190721_130617/dump/stp.log
removed '/var/dump/sonic_dump_sonic_20190721_130617/dump/stp.log'
sonic_dump_sonic_20190721_130617/dump/ps.aux
removed '/var/dump/sonic_dump_sonic_20190721_130617/dump/ps.aux'
sonic_dump_sonic_20190721_130617/dump/free
removed '/var/dump/sonic_dump_sonic_20190721_130617/dump/free'
sonic_dump_sonic_20190721_130617/dump/vmstat
removed '/var/dump/sonic_dump_sonic_20190721_130617/dump/vmstat'
sonic_dump_sonic_20190721_130617/dump/vmstat.m
removed '/var/dump/sonic_dump_sonic_20190721_130617/dump/vmstat.m'
sonic_dump_sonic_20190721_130617/dump/vmstat.s
removed '/var/dump/sonic_dump_sonic_20190721_130617/dump/vmstat.s'
sonic_dump_sonic_20190721_130617/dump/mount
removed '/var/dump/sonic_dump_sonic_20190721_130617/dump/mount'
sonic_dump_sonic_20190721_130617/dump/df
removed '/var/dump/sonic_dump_sonic_20190721_130617/dump/df'
sonic_dump_sonic_20190721_130617/dump/dmesg
removed '/var/dump/sonic_dump_sonic_20190721_130617/dump/dmesg'
sonic_dump_sonic_20190721_130617/dump/APP_DB.json
removed '/var/dump/sonic_dump_sonic_20190721_130617/dump/APP_DB.json'
sonic_dump_sonic_20190721_130617/dump/ASIC_DB.json
removed '/var/dump/sonic_dump_sonic_20190721_130617/dump/ASIC_DB.json'
sonic_dump_sonic_20190721_130617/dump/COUNTERS_DB.json
removed '/var/dump/sonic_dump_sonic_20190721_130617/dump/COUNTERS_DB.json'
sonic_dump_sonic_20190721_130617/dump/CONFIG_DB.json
removed '/var/dump/sonic_dump_sonic_20190721_130617/dump/CONFIG_DB.json'
sonic_dump_sonic_20190721_130617/dump/docker.ps
removed '/var/dump/sonic_dump_sonic_20190721_130617/dump/docker.ps'
sonic_dump_sonic_20190721_130617/dump/docker.pmon
removed '/var/dump/sonic_dump_sonic_20190721_130617/dump/docker.pmon'
Jul 21 13:08:57.807649 sonic INFO swss#supervisor-proc-exit-listener: Process orchagent exited unxepectedly. Terminating supervisor...
- New command output (if the output of a command-line utility has changed)
admin@sonic:~$ show techsupp
main
mkdir: created directory '/var/dump/sonic_dump_sonic_20190721_130617'
'/var/dump/sonic_dump_sonic_20190721_130617/generate_dump' -> '/usr/bin/generate_dump'
sonic_dump_sonic_20190721_130617/
sonic_dump_sonic_20190721_130617/generate_dump
mkdir: created directory '/var/dump/sonic_dump_sonic_20190721_130617/proc'
'/proc/buddyinfo' -> '/var/dump/sonic_dump_sonic_20190721_130617/proc/buddyinfo'
'/proc/cmdline' -> '/var/dump/sonic_dump_sonic_20190721_130617/proc/cmdline'
'/proc/consoles' -> '/var/dump/sonic_dump_sonic_20190721_130617/proc/consoles'
'/proc/cpuinfo' -> '/var/dump/sonic_dump_sonic_20190721_130617/proc/cpuinfo'
'/proc/devices' -> '/var/dump/sonic_dump_sonic_20190721_130617/proc/devices'
'/proc/diskstats' -> '/var/dump/sonic_dump_sonic_20190721_130617/proc/diskstats'
'/proc/dma' -> '/var/dump/sonic_dump_sonic_20190721_130617/proc/dma'
'/proc/interrupts' -> '/var/dump/sonic_dump_sonic_20190721_130617/proc/interrupts'
'/proc/iomem' -> '/var/dump/sonic_dump_sonic_20190721_130617/proc/iomem'
'/proc/ioports' -> '/var/dump/sonic_dump_sonic_20190721_130617/proc/ioports'
'/proc/kallsyms' -> '/var/dump/sonic_dump_sonic_20190721_130617/proc/kallsyms'
'/proc/loadavg' -> '/var/dump/sonic_dump_sonic_20190721_130617/proc/loadavg'
'/proc/locks' -> '/var/dump/sonic_dump_sonic_20190721_130617/proc/locks'
'/proc/meminfo' -> '/var/dump/sonic_dump_sonic_20190721_130617/proc/meminfo'
'/proc/misc' -> '/var/dump/sonic_dump_sonic_20190721_130617/proc/misc'
'/proc/modules' -> '/var/dump/sonic_dump_sonic_20190721_130617/proc/modules'
'/proc/self/mounts' -> '/var/dump/sonic_dump_sonic_20190721_130617/proc/mounts'
'/proc/self/net' -> '/var/dump/sonic_dump_sonic_20190721_130617/proc/net'
:
:
:
sonic_dump_sonic_20190721_130617/dump/stp.log
removed '/var/dump/sonic_dump_sonic_20190721_130617/dump/stp.log'
sonic_dump_sonic_20190721_130617/dump/ps.aux
removed '/var/dump/sonic_dump_sonic_20190721_130617/dump/ps.aux'
sonic_dump_sonic_20190721_130617/dump/free
removed '/var/dump/sonic_dump_sonic_20190721_130617/dump/free'
sonic_dump_sonic_20190721_130617/dump/vmstat
removed '/var/dump/sonic_dump_sonic_20190721_130617/dump/vmstat'
sonic_dump_sonic_20190721_130617/dump/vmstat.m
removed '/var/dump/sonic_dump_sonic_20190721_130617/dump/vmstat.m'
sonic_dump_sonic_20190721_130617/dump/vmstat.s
removed '/var/dump/sonic_dump_sonic_20190721_130617/dump/vmstat.s'
sonic_dump_sonic_20190721_130617/dump/mount
removed '/var/dump/sonic_dump_sonic_20190721_130617/dump/mount'
sonic_dump_sonic_20190721_130617/dump/df
removed '/var/dump/sonic_dump_sonic_20190721_130617/dump/df'
sonic_dump_sonic_20190721_130617/dump/dmesg
removed '/var/dump/sonic_dump_sonic_20190721_130617/dump/dmesg'
sonic_dump_sonic_20190721_130617/dump/APP_DB.json
removed '/var/dump/sonic_dump_sonic_20190721_130617/dump/APP_DB.json'
sonic_dump_sonic_20190721_130617/dump/ASIC_DB.json
removed '/var/dump/sonic_dump_sonic_20190721_130617/dump/ASIC_DB.json'
sonic_dump_sonic_20190721_130617/dump/COUNTERS_DB.json
removed '/var/dump/sonic_dump_sonic_20190721_130617/dump/COUNTERS_DB.json'
sonic_dump_sonic_20190721_130617/dump/CONFIG_DB.json
removed '/var/dump/sonic_dump_sonic_20190721_130617/dump/CONFIG_DB.json'
sonic_dump_sonic_20190721_130617/dump/docker.ps
removed '/var/dump/sonic_dump_sonic_20190721_130617/dump/docker.ps'
sonic_dump_sonic_20190721_130617/dump/docker.pmon
removed '/var/dump/sonic_dump_sonic_20190721_130617/dump/docker.pmon'
-->