Garbage collector seems to have an impact on webserver responsiveness

[sanderegg]

As discovered today with DK, it seems the interval of the garbage collector has an influence on the fact that the webserver becomes periodically unable to answer the healthcheck within 1 second.

DK used a script to bombard the webserver with healtcheck using curl from the simcore_traefik node, showing that there is a periodic request that returns in more than 1 second. (reference here)

By changing the WEBSERVER_GARBAGE_COLLECTION_INTERVAL_SECONDS variable from 30 seconds to 300 made the error completely disappear.
Now it is set to 90seconds.

But it seems that whatever the garbage collector is doing, that it completely prevents the webserver from doing its work correctly.

[mrnicegyu11]

Ill quickly add that when the garbage collector is running, this blocks all endpoints, and thus affects the /v0/health and /v0/ endpoints specifically as well. If traefik checks these endpoints at this time, it will determine the webserver to be unhealthy and stop forwarding any requests to the container. Instead, traefik will answer every request to the webserver with a 503. With high confidence, this is the origin of the statusping failures