file system full could cause service not running after reboot

[yxieca]

We noticed that some devices running old sonic releases could fill /var/log/ partition to 100%.

When /var/log partition is full, we noticed that some containers (if not all) would stay down after o.s. reboot.

The issue of filling /var/log partition has been solved in recent release (swss releases log file to allow logrotate to happen).

However, when we upgrade from old releases with full partitions to new releases, the storage space is not automatically freed. Therefore, human intervention is needed to clear these partitions.

As result, we would like to see that the upgrade process format the /var/log partition to mitigate the issue where a full /var/log partition could prevent rebooted device to start service.

[grozovik]

we discussed that during the sonic to sonic upgrade and we agreed this is not correct. when we upgrade we would like to keep the /var/log so we will see the reboot flow as well.
Also, when we install a version it does not recent automatically so it means that this will be removed even before the upgrade which is not correct.

I do think that the install procedure on these cases should be manually.
appreciate if you can refer to the above before we start the implementation.

[marian-pritsak]

As @grozovik pointed, logs are not cleared by design.
The rationale behind this is to keep as much information as possible.

I'm more concerned about a problem that containers are inoperable when log partition is full. The main purpose of moving logs to separate mount was to limit its size so that in a case of flooding from some component switch won't stop working due to 'no space left' error.

@jleveque , are you aware of this problem? The switch shouldn't become inoperable even if we have /var/log full.

[lguohan]

@grozovik, even when clear the /var/log we should still see the reboot flow.

@grozovik, can you explain this? I do not fully understand this.

"
Also, when we install a version it does not recent automatically so it means that this will be removed even before the upgrade which is not correct.

I do think that the install procedure on these cases should be manually."

@marian-pritsak , swss won't start if the log is full as it tries to open swss/sairedis log file.

[liatgrozovik]

sonic-installer can be used at any time for future use and will not necessarily followed by immediate rebooted. This was one of the reasons we didnt implement the log clear when we run the install. We agreed that if the user would like to clear the log he should do it manually. As this is only relevant to the version which does not support log size limit i dont think we should add a clear log to any sonic-installer call. If you wajt it it should be only as an option param which by default no clear log is done. Get Outlook for Android<https://aka.ms/ghei36>

…

________________________________ From: lguohan <notifications@github.com> Sent: Tuesday, July 25, 2017 7:44:18 PM To: Azure/sonic-buildimage Cc: Subscribed Subject: Re: [Azure/sonic-buildimage] file system full could cause service not running after reboot (#839) @grozovik<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.luolix.top%2Fgrozovik&data=02%7C01%7Cliatg%40mellanox.com%7C34c62d706137490eafac08d4d37c650a%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636365978607571397&sdata=CdtNroM6SXTOsjlKMSw6gz2%2FF10LUW1sLHKK7E%2FRl20%3D&reserved=0>, even when clear the /var/log we should still see the reboot flow. @grozovik<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.luolix.top%2Fgrozovik&data=02%7C01%7Cliatg%40mellanox.com%7C34c62d706137490eafac08d4d37c650a%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636365978607571397&sdata=CdtNroM6SXTOsjlKMSw6gz2%2FF10LUW1sLHKK7E%2FRl20%3D&reserved=0>, can you explain this? I do not fully understand this. " Also, when we install a version it does not recent automatically so it means that this will be removed even before the upgrade which is not correct. I do think that the install procedure on these cases should be manually." @marian-pritsak<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.luolix.top%2Fmarian-pritsak&data=02%7C01%7Cliatg%40mellanox.com%7C34c62d706137490eafac08d4d37c650a%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636365978607571397&sdata=HDjYRsQXrl6ymEqlIBK6kBqbNk%2FDnmDR2taT%2FODpYl4%3D&reserved=0> , swss won't start if the log is full as it tries to open swss/sairedis log file. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.luolix.top%2FAzure%2Fsonic-buildimage%2Fissues%2F839%23issuecomment-317796686&data=02%7C01%7Cliatg%40mellanox.com%7C34c62d706137490eafac08d4d37c650a%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636365978607571397&sdata=15hLz3t47hdKvhGPsX0CoQYSdAeB1d7PFBic3ALioGU%3D&reserved=0>, or mute the thread<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.luolix.top%2Fnotifications%2Funsubscribe-auth%2FAQ8YX6pmBxougYzQbH4arkzQbvyFSs9eks5sRhtigaJpZM4Oh0NB&data=02%7C01%7Cliatg%40mellanox.com%7C34c62d706137490eafac08d4d37c650a%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636365978607571397&sdata=bhxfkjEAUBw%2BAQ3vnTbxWsB3htF%2BGgCwJGu2aNslLN8%3D&reserved=0>.

[lguohan]

seems we are following liat suggestion, nothing further needs to be done.