Automatically detect number of cpus from cgroup in Scylla Operator binaries #1612
Conversation
…naries Golang runtime doesn't understand cgroups hence it assumes all cpus are available for running goroutines. Since applications on Kubernetes are running under restricted resources we should detect cgroups settings and align Go runtime to discovered settings. This should lower resource consumption, and allow us to run on even more limited resources.
zimnx
added
kind/feature
scylla-operator-bot
bot
added
area/dependency
|
Did we actually hit any throttling-related issues in the past? |
I don't recall hitting latency issues with the operator itself. As this is well know and we haven't seen an impact, so far we have been waiting for a golang to fix golang/go#33803. I suppose we could as well use the uber lib in the meantime although the impact may be blurry. |
No, we haven't, as we don't have any observability, nor any container running on low resources having a liveness/readiness probe so any slowness is not visible. As part of other PR I will add such low resource container with probes. I hit multiple times issue with such container being started but not even printing starting log message. After I configured Go runtime properly I haven't noticed it again. Can't tell whether I was lucky or it indeed fixed what I saw. This is a known issue, fixing it wouldn't hurt. |
There was a problem hiding this comment.
if it helps you I am fine going ahead with it and reverting this when golang/go#33803 gets fixed
/approve
/lgtm
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: tnozicka, zimnx The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Of course, I just wanted to know why do it now, since I remember we discussed in the past and never followed through. Thanks. |
|
Flake: #1028 (comment) |
Description of your changes:
Golang runtime doesn't understand cgroups hence it assumes all cpus are
available for running goroutines.
Since applications on Kubernetes are running under restricted resources
we should detect cgroups settings and align Go runtime to discovered settings.
This should lower resource consumption, and allow us to run on even more limited resources.