Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[201803] Restart SwSS, syncd and dependent services if a critical process in syncd container exits #2615

Merged
merged 3 commits into from
Mar 7, 2019
Merged

[201803] Restart SwSS, syncd and dependent services if a critical process in syncd container exits #2615

merged 3 commits into from
Mar 7, 2019

Conversation

jleveque
Copy link
Contributor

- What I did

  • Add process event listener in syncd docker containers to listen for critical process exits. If a critical process exits, syncd container will stop, causing syncd service to stop.
  • Modified SwSS service to start both SwSS and syncd Docker containers and then run a loop which checks that both containers are running. If either container stops, SwSS service will stop and, as long as it hasn't met its retry limit, will be restarted, along with dependent services.

@jleveque
Add process exit listeners to syncd containers so that containers sto…
daccd58 
…p upon unexpected critical process exit
@jleveque
swss.service unit file now starts both swss and syncd containers as E…
613fe0d 
…xecStartPre steps, and for ExecStart, it indefinitely checks that the containers are running
@jleveque jleveque self-assigned this Feb 28, 2019
@jleveque jleveque requested a review from lguohan February 28, 2019 02:05
@lguohan
Copy link
Collaborator

lguohan commented Mar 1, 2019

i prefer the wait. here is the docker-py implementation.

https://github.com/docker/docker-py/blob/master/docker/api/container.py#L1253

we should be able to do two post and wait for each one to return?

@jleveque
Copy link
Contributor Author

jleveque commented Mar 5, 2019

@lguohan: Addressed in 8827676. I wrote a new application, docker-wait-any which utilizes the docker-py package and will wait on one or more containers in a threaded fashion. If any containers stop, the application will exit.

@lguohan lguohan merged commit 9e8518c into sonic-net:201803 Mar 7, 2019
@jleveque jleveque deleted the restart_swss_syncd_201803 branch March 7, 2019 18:51
lguohan added a commit that referenced this pull request Apr 6, 2019
@lguohan
Revert "[201803] Restart SwSS, syncd and dependent services if a crit…
968a0df 
…ical process in syncd container exits (#2615)"

This reverts commit 9e8518c.
@prsunny prsunny mentioned this pull request Jan 17, 2023
Merged
8 tasks
qiluo-msft pushed a commit that referenced this pull request Jan 18, 2023
@prsunny
[Submodule] Update for sonic-swss (#13405)
b500e0e 
#### Why I did it

Update for following swss commits:

96180bf - 2023-01-13 : [202012] Bfd default multiplier change  (#2615) [siqbal1986]
07506ac - 2023-01-11 : Add missing parameter to on_switch_shutdown_request method. (#2567) [Hua Liu]
3253cc8 - 2022-11-30 : Use github code scanning instead of LGTM (#2546) [Liu Shilong]
f4df524 - 2023-01-11 : [orchagent]:add local_discriminator to state_db (#2587) [Baorong Liu]
f3cd02d - 2022-12-05 : [202012][muxorch] Adding case for maintaining current state (#2500) [Nikola Dancejic]
@mssonicbld mssonicbld mentioned this pull request Feb 3, 2023
Closed
dprital added a commit to dprital/sonic-buildimage that referenced this pull request Feb 7, 2023
@dprital
[submodule] Advance sonic-utilities pointer
ce8cce8 
Update sonic-utilities submodule pointer to include the following:
* f9130d1c [db_migrator] make LOG_LEVEL_DB migration more robust ([sonic-net#2651](sonic-net/sonic-utilities#2651))
* a2520e60 Fixed a bug in show vnet routes all causing screen overrun. ([sonic-net#2644](sonic-net/sonic-utilities#2644))
* c57c3fad show logging CLI support for logs stored in tmpfs ([sonic-net#2641](sonic-net/sonic-utilities#2641))
* 5d23934f [chassis][voq] Add asic id for linecards so show fabric counters queue/port can work. ([sonic-net#2499](sonic-net/sonic-utilities#2499))
* 79ffd9fd Add Transceiver PM basic CLI support to show output from TRANSCEIVER_PM table for ZR ([sonic-net#2615](sonic-net/sonic-utilities#2615))
* 1b71985e [masic support] 'show run bgp' support for multi-asic ([sonic-net#2427](sonic-net/sonic-utilities#2427))
* 8239e9ab Making 'show feature autorestart' more resilient to missing auto_restart config in CONFIG_DB ([sonic-net#2592](sonic-net/sonic-utilities#2592))
* 9ee6ac29 [doc] Update docs for dhcp_relay config cli ([sonic-net#2598](sonic-net/sonic-utilities#2598))
* c3c92a47 Skip saidump for Spine Router as this can take more than 5 sec ([sonic-net#2637](sonic-net/sonic-utilities#2637))
* 6fe85992 Secure upgrade ([sonic-net#2337](sonic-net/sonic-utilities#2337))

Signed-off-by: dprital <drorp@nvidia.com>
@siqbal1986 siqbal1986 mentioned this pull request Feb 7, 2023
Closed
8 tasks
liat-grozovik pushed a commit that referenced this pull request Feb 8, 2023
@dprital
[submodule] Advance sonic-utilities pointer (#13706)
11926de 
Update sonic-utilities submodule pointer to include the following:
* f9130d1c [db_migrator] make LOG_LEVEL_DB migration more robust ([#2651](sonic-net/sonic-utilities#2651))
* a2520e60 Fixed a bug in show vnet routes all causing screen overrun. ([#2644](sonic-net/sonic-utilities#2644))
* c57c3fad show logging CLI support for logs stored in tmpfs ([#2641](sonic-net/sonic-utilities#2641))
* 5d23934f [chassis][voq] Add asic id for linecards so show fabric counters queue/port can work. ([#2499](sonic-net/sonic-utilities#2499))
* 79ffd9fd Add Transceiver PM basic CLI support to show output from TRANSCEIVER_PM table for ZR ([#2615](sonic-net/sonic-utilities#2615))
* 1b71985e [masic support] 'show run bgp' support for multi-asic ([#2427](sonic-net/sonic-utilities#2427))
* 8239e9ab Making 'show feature autorestart' more resilient to missing auto_restart config in CONFIG_DB ([#2592](sonic-net/sonic-utilities#2592))
* 9ee6ac29 [doc] Update docs for dhcp_relay config cli ([#2598](sonic-net/sonic-utilities#2598))
* c3c92a47 Skip saidump for Spine Router as this can take more than 5 sec ([#2637](sonic-net/sonic-utilities#2637))
* 6fe85992 Secure upgrade ([#2337](sonic-net/sonic-utilities#2337))

Signed-off-by: dprital <drorp@nvidia.com>
@StormLiangMS StormLiangMS mentioned this pull request Feb 11, 2023
Merged
8 tasks
StormLiangMS added a commit that referenced this pull request Feb 17, 2023
@StormLiangMS
[submodules] advance submodules for 202211 #13781
1d155b8 
Why I did it
Submodule advances:
sonic-utilities

8e8e6088 - [202211][dhcp_relay] Remove add field of vlanid to DHCP_RELAY table while adding vlan ([201811 sub-module] advance sub-modules: utilities, swss, swss-common #2679) (16 hours ago) [Yaqiang Zhu]
1400fb94 - [GCU] Ignore bgpraw in GCU applier (Fix sfputil indexing for 7170-Q59S20 #2623) (15 hours ago) [jingwenxie]
f76a6364 - [vlan] Refresh dhcpv6_relay config while adding/deleting a vlan ([sonic-py-swsssdk] Update submodule #2660) (15 hours ago) [Yaqiang Zhu]
7849e18d - [db_migrator] make LOG_LEVEL_DB migration more robust (Mellanox platform: attach queues 2 and 6 to lossy profile using generic buffer template #2651) (16 hours ago) [Stepan Blyshchak]
c7df6dfa - Fixed a bug in "show vnet routes all" causing screen overrun. (Add hook to allow customizing link cable lengths #2644) (16 hours ago) [siqbal1986]
a5505f02 - show logging CLI support for logs stored in tmpfs (Traceback error seen while issuing show interface commands with if_names #2641) (16 hours ago) [mihirpat1]
bbacb91a - [system-health] Fix issue: show system-health CLI crashes (Updating deb package for platform and sai #2635) (16 hours ago) [Junchao-Mellanox]
8d724024 - [sai_failure_dump]Invoking dump during SAI failure ([dockers]: Upgrade LLDP docker to stretch build #2633) (16 hours ago) [Sudharsan Dhamal Gopalarathnam]
3c3be526 - Add transceiver info CLI support to show output from TRANSCEIVER_INFO for ZR ([submodule]: Update sonic-sairedis pointer #2630) (16 hours ago) [mihirpat1]
37f41666 - [show] add support for gRPC show commands for active-active ([bitmap-vnet]: Bitmap vnet test image [DO NOT MERGE] #2629) (16 hours ago) [vdahiya12]
b06d7fe4 - [show_bfd] add local discriminator in show bfd command ([Pmon] Selectively load pmon container daemons #2625) (16 hours ago) [Baorong Liu]
6adcd3e8 - [GCU] Ignore bgpraw table in GCU operation ([Mellanox] Fix SAI version #2628) (16 hours ago) [jingwenxie]
c65bdc35 - [muxcable][config] Add support to enable/disable ceasing to be an advertisement interface when radv service is stopped (Add knob in ConfigDB to enable/disable telemetry container #2622) (16 hours ago) [Jing Zhang]
91e9457f - Add Transceiver PM basic CLI support to show output from TRANSCEIVER_PM table for ZR ([201803] Restart SwSS, syncd and dependent services if a critical process in syncd container exits #2615) (16 hours ago) [longhuan-cisco]
54cc8c5a - Remove TODO comment which is no longer relevant (Warm-reboot: teamd warm restart caused neighbor deleted and learned again.  #2600) (16 hours ago) [Lior Avramov]
6891b4fb - Making 'show feature autorestart' more resilient to missing auto_restart config in CONFIG_DB ([submodule] update mellanox hw-mgmgt pointer (V.2.0.0061) #2592) (16 hours ago) [kartik-arista]
1e8bea37 - [storyteller] add link prober state change to story teller ([sonic-buildimage] New feature managementVRF(L3mdev) #2585) (16 hours ago) [Jing Zhang]
7481a20f - Extend fast-reboot STATE_DB entry timer ([submodule]: update sonic-swss-common, sonic-py-swsssdk, sonic-snmpagent #2577) (16 hours ago) [Aryeh Feigin]
0e08701c - [sonic_installer] use /etc/resolv.conf from the host when migrating packages (Set a rate limit on syslog messages from all Docker containers #2573) (16 hours ago) [Stepan Blyshchak]
06096780 - Fixed admin state config CLI for Backport interfaces (Prior to install a new ONIE SONiC image, delete all partitions except EFI/ONIE #2557) (16 hours ago) [anamehra]
9f1f13e4 - [show] Add bgpraw to show run all (Fixed typo on paragraph #40 #2537) (16 hours ago) [jingwenxie]
98bc8bd2 - [chassis][voq] Add "show fabric reachability" command. ([ntp]: Build 4.2.6 locally. #2528) (16 hours ago) [jfeng-arista]
3a50b63f - Preserve copp tables through DB migration ([docker-radvd]: upgrade docker radvd to stretch based #2524) (16 hours ago) [Aryeh Feigin]
28f6b127 - [masic] 'show interfaces counters' reminds to use '-d all' option to check for internal links (solve dependency issue #2466) (16 hours ago) [wenyiz2021]
15026e14 - suppport multi asic for show queue counter ([dockers] Prevent old supervisord messages from gettting re-logged to syslog #2439) (16 hours ago) [zhixzhu]
2d773e17 - [masic support] 'show run bgp' support for multi-asic (lo address not synced to the asic #2427) (16 hours ago) [wenyiz2021]
sonic-swss

4f304bc - [EVPN]Handling race condition when remote VNI arrives before tunnel map entry ([sonic-quagga] Function defect, do NOT cancel route while connect IP down #2642) (15 hours ago) [Sudharsan Dhamal Gopalarathnam]
34fc615 - [sai_failure_dump]Invoking dump during SAI failure (Add hook to allow customizing link cable lengths #2644) (15 hours ago) [Sudharsan Dhamal Gopalarathnam]
b817695 - [autoneg]Fixing adv interface types to be set when AN is disabled (Fix issue with platform file path name #2638) (15 hours ago) [Sudharsan Dhamal Gopalarathnam]
ab36bd4 - [bfdorch] add local discriminator to state DB ([bitmap-vnet]: Bitmap vnet test image [DO NOT MERGE] #2629) (15 hours ago) [Baorong Liu]
6343471 - Remove TODO comments that are no longer relevant (Add knob in ConfigDB to enable/disable telemetry container #2622) (15 hours ago) [Lior Avramov]
2b1869c - [refactor]Refactoring sai handle status (Rollback kernel submodule update. #2621) (15 hours ago) [Sudharsan Dhamal Gopalarathnam]
c41a1b7 - Fix issue ARP entry is out of sync between kernel and APPL_DB after warm reboot if the ARP entry is updated more than once during warm reboot in PFC watchdog warm reboot test #13341 ARP entry can be out of sync between kernel and APPL_DB if multiple updates are received from RTNL ([sub module] advance sonic-utilities sub module for 201811 branch #2619) (15 hours ago) [Stephen Sun]
da0cf7a - Changed the BFD default detect multiplier to 10x ("failed to load plugin io.containerd.snapshotter..." seen during linux boot up #2614) (15 hours ago) [siqbal1986]
13b5adf - [vstest] Only collect stdout of orchagent_restart_check in vstest ([submodules] update swss and utilities pointers #2597) (15 hours ago) [bingwang-ms]
2b9d94d - Avoid aborting orchagent when setting TUNNEL attributes (build failing for PLATFORM=p4 #2591) (15 hours ago) [Stephen Sun]
99b7d3b - Only collect stdout of orchagent_restart_check in vstest ( [saibcm-modules]: import new bcm modules #2578) (15 hours ago) [bingwang-ms]
5209c42 - dereg acl-rule counters during acl-table del ([201803] Set a rate limit on syslog messages from all Docker containers #2574) (15 hours ago) [Vivek]
ae68054 - Fixed set mtu for deleted subintf due to late notification ([vs]: Add option to specify platform name for DVS orchagent #2571) (15 hours ago) [EdenGri]
ab13dfa - Remove TODO comments which are no longer needed (support set timezone in ConfigDB #2568) (15 hours ago) [Junchao-Mellanox]
a3545cf - Modify coppmgr mergeConfig to support preserving copp tables through reboot. (Added new SN3700/SN3700C Mellanox platforms #2548) (15 hours ago) [Aryeh Feigin]
be16e79 - Use github code scanning instead of LGTM ([201803] [services] Restart SwSS service upon unexpected critical process exit #2546) (15 hours ago) [Liu Shilong]
63c0234 - Updated handling of VRF_VNI mapping and VLAN_VNI mapping for same VNI ID (Move warm_restart enable/disable config to stateDB WARM_RESTART_ENABL… #2538) (15 hours ago) [Tapash Das]
4844111 - Fix potential risks ([mlnx] Fix sai xml path for boxer platform #2516) (15 hours ago) [Liran-Ar]
6420808 - [p4orch]: PINS Extension tables support ([build] When generating image version, handle case where current commit has no reachable tags #2506) (15 hours ago) [svshah-intel]
sonic-swss-common

1badd46 - Increase the netlink buffer size from 3MB to 16MB. (arp_update doesn't sleep 300 between each execution #739) (14 hours ago) [KISHORE KUNAL]
6555057 - Refactor eventpublisher deinit ([acl] Add default deny rule for l3 table #734) (14 hours ago) [Zain Budhwani]
f4d6de7 - Use github code scanning instead of LGTM ([sonic-quagga]:update submodule #718) (14 hours ago) [Liu Shilong]
sonic-linux-kernel

74f9a8f - Update linux kernel for hw-mgmt V.7.0020.4104 (Move template files to /usr/share/sonic/templates #305) (14 hours ago) [Stephen Sun]
6365701 - Fixes for emmc unreliability ([build_debian.sh]: Integrate system dump script #270) (14 hours ago) [Samuel Angebault]
How I did it
How to verify it
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lguohan lguohan lguohan approved these changes

Assignees

@jleveque jleveque

Labels
Enhancement ➕
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@jleveque @lguohan