"Mailserver too busy. Please try again later." when attempting to install or enable SnappyMail in Nextcloud

[dsaderholm]

SnappyMail version: 2.26.1, browser: Firefox, browser version: 110

Expected behavior and actual behavior: I should be able to install it, but I get this error, whether through command line (occ) or the Apps page on the web interface.

Steps to reproduce the problem: Attempt to install SnappyMail

I have tried deleting the app, its config file, and the app's folder. I can install Rainloop just fine. There is nothing wrong with my email server. I got the error when trying to access SnappyMail while it was installed. I removed it and reinstalled it and still get that same error.

[the-djmaze]

Happens when your serverload is more then 95%

snappymail/snappymail/v/0.0.0/include.php

Lines 6 to 12 in 38c4b4e

	if (function_exists('sys_getloadavg')) {
	$load = sys_getloadavg();
	if ($load && $load[0] > 95) {
	header('HTTP/1.1 503 Service Unavailable');
	header('Retry-After: 120');
	exit('Mailserver too busy. Please try again later.');
	}

Can you check what is eating the CPU?

[dsaderholm]

My CPU usage has been below 10% mostly. It only went to about 25% at one point.

[dsaderholm]

I commented out those lines and now SnappyMail works flawlessly. I think SnappyMail could be reading this high CPU usage from Nextcloud. For some reason, I see this on my System tab in settings:

[the-djmaze]

Oh, how many cores do you have?

As 2181 is a very high number, so something goes wrong here.

Maybe i should move the code so that you can have a config setting to set the max load yourself.
As 95 is already for 95 cpu's.
https://en.m.wikipedia.org/wiki/Load_(computing)

[mattv8]

Hey there. I'm the server owner that's hosting @dsaderholm's SnappyMail instance. I watched the CPU for several minutes while trying to log in to SnappyMail, and the issue persisted, so I investigated further. I think that how is SnappyMail determining whether or not the server is 'busy' could be improved by calculating CPU percentage rather than load averages.

The sys_getloadavg() function returns an array containing the average system load values for the last 1, 5, and 15 minutes. There is no straightforward way to convert these values to percentages of total CPU use. The values for $load returned in my case are the following:

<?php
$load = sys_getloadavg();
print_r($load);
Array
(
    [0] => 2268.01
    [1] => 2263.21
    [2] => 2251.98
)

My server has 32 total cores. These numbers are very high for some reason, I do not understand why.

So, I have re-written the CPU load calculation using PHP's exec function and the sysstat package sar. This is more robust and should work on a wide range of Unix-like systems, but obviously won't work on Windows. I'll need help implementing the calculation on Windows environments. Here is how the "system is busy" check could be corrected:

if (function_exists('exec')) {
    $load_percent = get_system_load_percent();
    if ($load_percent > 95.0) {
        header('http/1.1 503 service unavailable');
        header('retry-after: 120');
        exit('mailserver too busy. please try again later.');
    }
} else {
    // fallback behavior if get_system_load_percent() is not available
}

function get_system_load_percent() {
    if (strtoupper(substr(PHP_OS, 0, 3)) === 'WIN') {
        return 0.0; // Return 0 on Windows as sar command is not available
    }

    $cpu_usage = array();
    exec('sar -u 1 1 | tail -1 | awk \'{print $NF}\'', $cpu_usage, $return_var);
    if ($return_var !== 0) {
        return 0.0; // Command execution failed
    }
    if (count($cpu_usage) > 0) {
        $cpu_usage = floatval($cpu_usage[0]);
    } else {
        $cpu_usage = 0.0;
    }
    return round(100.0 - $cpu_usage, 2);
}

The above does introduce some latency to the loading of SnappyMail, about 1 second, in order to calculate the CPU percentage. There's no way around this unless someone can find a cleverer way to calculate the relative CPU usage.

EDIT: I have edited my post to correctly indicate that if ($load && $load[0] > 95) is indeed a good check if the output of sys_getloadavg() returns sane values. My apologies for initially indicating the code is flawed!

[the-djmaze]

Hi @mattv8 good that you try to investigate.

When you read https://en.m.wikipedia.org/wiki/Load_(computing) your numbers should not be higher then 32.
But the page also says that stalled I/O and many others could raise that number.

So basically my 95 is already insane high.
That's why i propose to make it a config setting.
Then move the code to the place where config gets loaded and every admin can tweak it to a sane number for their servers and prevent server overloading.

Security wise this could prevent large attacks when 1000's of requests are made per minute.

[mattv8]

Indeed you are right the high values indicate something else is going on. Interestingly top, vmstat, htop all report sane load averages on my system, yet PHP's sys_getloadavg() is impossibly high. I can't find anything on the internet that might indicate why. I'm sure rebooting my server would be a fix... But uptime!

I agree, providing admins with a configurable flag for this check is the most ideal way to go. Introducing latency by calculating the CPU percentage would defeat the purpose of the check in the first place, lol.