[reboot] stop docker service before rebooting #423
Conversation
After upgrading to docker-ce 18.09.0. Shutdown SONiC takes 5-10 minutes waiting for all the running docker containers to quit. They all enventually hit the timeout and caused the huge delay. This is a work-around until proper solution is found. Signed-off-by: Ying Xie <ying.xie@microsoft.com>
|
While we are figuring out the root cause, do you see needs to make any changes in fastboot script after docker upgrade? BTW: "5-10 seconds" should be "5-10 minutes" in the description. :) |
|
The fast reboot script is the same base as warm reboot, I suspect it would be fine but some test is needed. Thanks for noticing the wording error :-) |
|
Sorry, I guess I didn't answer your question in the way you expected. fast/warm reboot script already stopped docker service before reboot. So they are fine. |
|
This change is no longer needed. |
|
Oops. The reboot issue wasn't fixed by the proper installation of docker engine. This change is still needed for now. |
- add a protection of 30 seconds timeout. Signed-off-by: Ying Xie <ying.xie@microsoft.com>
|
what is the root cause of the 5 to 10 minutes delay? |
|
The root cause is not clear at the moment. The problem doesn't happen when issuing "docker stop <any container" or "systemctl stop docker". But it is somehow 100% repeatable during regular reboot. |
|
Is it related to the docker corruption issue: sonic-net/sonic-buildimage#2382? It looks the docker failure was seen with docker engine 1.11. With 18.09.0, I have not seen the docker hang issue. |
|
Jipan, I think this issue is not the same as the issue you referenced. The issue I am working on, is very particular to shutdown path of reboot. And I didn't see docker corruption afterwards, yet. Regards, |
|
retest this please |
This reverts commit 220e269.
- What I did
After upgrading to docker-ce 18.09.0. Shutdown SONiC takes 5-10 minutes
waiting for all the running docker containers to quit. They all enventually
hit the timeout and caused the huge delay.
This is a work-around until proper solution is found.
- How to verify it
with the change, reboot takes little time after upgrading to docker-ce 18.09.0. Otherwise the reboot takes about 5-10 minutes on shutdown path.