[201803] [services] Restart SwSS service upon unexpected critical process exit

[jleveque]

- What I did
Restart SwSS service (and also restart dependent services) if orchagent process exits abnormally.

NOTE: Will need to port this patch to the master branch in the near future, as well.

- How I did it

• Add supervisor-proc-exit-listener event listener plugin for Supervisor running in SwSS Docker container which is set to listen for abnormal orchagent exits
• Configure swss.service to always auto-restart the service if it stops, with a delay of 30 seconds [Also set a rate limit of 3 restarts within 20 minutes (1200 seconds). If this is exceeded, systemd will stop attempting to restart the service. To restart the service after entering this state, one must first run systemctl reset-failed -- we should probably call this command in config load_minigraph before restarting services]
• Add dependency between teamd and SwSS to ensure teamd also gets restarted along with SwSS
• Also change the way the DHCP relay Docker container waits for interfaces to be ready before starting the relay agent process. Rather than using ip commands, now check STATE_DB for interface entries with "state" == "ok"
• Also add "WantedBy=swss.service" option to unit files of services which need to be started with SwSS (currently teamd, snmp, dhcp_relay and radv). The "Requires=swss.service" option causes the dependent services to stop and restart along with SwSS (when calling systemctl stop swss.service and systemctl restart swss.service). However this will not cause them to start with SwSS (when calling systemctl start swss.service). This functionality is enabled with the addition of the "WantedBy=" option.
• supervisor-proc-exit-listener script resides in files/scripts/ so that the same script can be installed in multiple Docker containers. To add this solution to another container, one simply needs to do the following:
  1. Add the script to the container's "_FILES" variable in the container's Makefile, and ensure it gets copied into the container in the container's Dockerfile
  2. Add a /etc/supervisor/critical_processes file to the container specifying all critical processes, one per line
  3. Add the event listener as a process to the container's supervisor config file

- How it Works

• If orchagent process crashes/exits abnormally, supervisor-proc-exit-listener will send a SIGTERM signal to Supervisor, causing it to exit also
• Since Supervisor is running as PID 1 within the Docker container, when Supervisor process exits, it will cause the container to stop
• When the SwSS Docker container stops, systemd will consider the swss.service to have stopped unexpectedly. Systemd will wait 30 seconds, then stop dependent services, restart SwSS, and lastly restart dependent services (currently teamd, snmp, dhcp_relay and radv).

- How to verify it

Send a signal to orchagent to cause it to appear to exit abnormally (e.g., pkill -11 orchagent). Ensure the swss, syncd, teamd, snmp, dhcp_relay and radv services get restarted per the above details.

- A picture of a cute animal (not mandatory but encouraged)

               )\._.,--....,'``.       
 .b--.        /;   _.. \   _\  (`._ ,. 
`=,-,-'~~~   `----(,_..'--(,_..'`-.;.'

[jleveque]

Retest this please

[renukamanavalan]

@ e87b9ba#diff-527fc13fdccd498aa46471bac1c51a2d

supervisor-proc-exit-listener python script:
Will there be any more additional info in payload that can be logged to get more insight ?

[jleveque]

@renukamanavalan: More info with regards to what? The reason the process exited? If so, Supervisor already logs this info upon process exit. For example, INFO exited: orchagent (terminated by SIGSEGV (core dumped); not expected).

[renukamanavalan]

The changes looks good to me, per my little knowledge on the system.

Personally, I would need to understand more at higher level. For example, the script wait_for_intf.sh (both old & current) waits forever, until the IF is up. Wonder what would happen if interface init has failed and reporting some error, which would need some explicit action to fix, at the system level. Like, what logs gets fired and what auto-mitigation happens or possibilities for such ...

[jleveque]

Please do not close pull request. Please use the "Review changes" feature to add a review, either approving or requesting changes. PR owner or repo maintainer will merge PR once approved.

[renukamanavalan]

Sorry for closing, by mistake.

I don't have any request for any code changes. I will leave approval to Guohan, as I am still learning the system.

[jleveque]

Regarding the wait_for_intf.sh script, this is to ensure all interfaces required by the processes in the container are up and ready before starting them to ensure they do not get into a bad state. Auto-mitigation should be handled outside of this container, by orchagent or a related process.

[marian-pritsak]

Have you considered also printing message to users' terminals? Usinng wall command or something?

[jleveque]

No, this was not considered. Thanks for the suggestion.

@lguohan: What do you think about also broadcasting this message to all logged-in users' terminals?

[lguohan]

we should put this script as a SCRIPT so it can be shared by all dockers, then the process list should be move to a separate file instead of being hard-coded in the script.

[jleveque]

@lguohan: Where do you suggest we store the event listener script in the repo? We currently don't have files which are shared by multiple containers.

We could potentially add the script to the docker-base image and have it source a file containing the list of critical processes, but this might not be very straightforward.

[lguohan]

check rules/script.mk the CONFIGDB_LOAD_SCRIPT, it is shared by multiple containers.

[jleveque]

Done in commit b4c0ab2

[jipanyang]

Are you planning to make similar change for master/201811 branches? We may even consider unplanned warm restart support beyond this.

[jleveque]

@jipanyang: Yes, a similar change will be made to the master/201811 branches. The changes are a bit more complex due to warm reboot. After this PR is merged, I will begin work on a solution for those branches.