-
Notifications
You must be signed in to change notification settings - Fork 693
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Operator crashes with OOM error because of small limit #1468
Comments
Thanks for the report.
|
I've been seeing the same issue over the past few days. In my case, the increased memory usage seems to be correlated to an increase in the number of nodes being managed by the operator (currently 10). On average, it takes about 30-40 minutes for the pod to be killed due to OOM errors. Increasing the limit seems to have resolved things for me as well. |
@philrhinehart Can you provide more information about the workload? Is it just one Elasticsearch cluster of 10 nodes managed by the operator? What is the Kubernetes cluster utilization? If it is possible to do so without revealing any sensitive information, can you provide the manifest for Elasticsearch as well? |
I am closing this issue for now since our internal testing couldn't reproduce the problem. If anybody else experiences the same problem, please re-open this issue and provide details about the environment. |
my elastic operator kept oom too, removed the requirements it is sitting at 188MB memory, one cluster, one node, https://www.elastic.co/guide/en/cloud-on-k8s/1.0/k8s-quickstart.html |
@masterkain can you provide more details about your environment? Kubernetes version, self-hosted or cloud, other workloads in the cluster, whether this is a fresh install of ECK etc. |
Happening to me too. ECK Version: 1.0.0-beta1
AWS EKS
Anything else I can provide, let me know. |
@mcfearsome there were multiple changes that reduced memory usage that landed in v1.0.0 (especially https://www.elastic.co/guide/en/cloud-on-k8s/1.0-beta/release-highlights-1.0.0-beta1.html#k8s_memory_leak_in_the_eck_process), I would recommend upgrading and/or increasing the memory limit. |
Same problem here. I couldn't deploy the CRD because the resulting pod was continually killed by OOM. Which is strange because I had a 2 nodes (Kubernetes) cluster and the operator had no issues, and on this new cluster that has 7 (kubernetes) nodes and more CPU/RAM it gets killed every 30 sec... I removed the limits/requests section for now and everything seems to be back to normal. |
@Docteur-RS can you provide more details about your Kubernetes environment? Which version of ECK are you using? |
@sebgl I updated the default resources to the following and it did not work either:
I pretty much doubled everything... In the end I just commented the whole "resources" section and it fixed the OOM. |
Could you give us more information about the OS you are using:
|
@barkbay I applyed the limits / requests that failled and this is the logs I got:
2 stacktraces that appears multiple times in
|
Default Kernel for CentOS 7 is Kubernetes and container runtimes are relying on low level Kernel functions (like |
Hum... if I remember correctly we updated the Kernel version because Cilium (our kubernetes' CNI) needed BPF which was not availble in the default Kernel version we had. Though I checked the cluster's version on which it was working correctly: Small difference but maybe it's only what it takes. |
I think you can use Centos 8 if you want to use Cillium on CentOS. |
Just wanted to add some metric points here: ECK 1.0 on GKE - the operator kept OOMkilling itself, on average tries to use ~140Mi of memory, and I'm now trying to stabilize this with 200Mi of memory with guaranteed QoS. |
We increased the default memory limits in #3046 which should be included in the next release, so it can work out of the box in more environments. |
Bug Report
What did you do?
I simply followed the instructions here:
https://www.elastic.co/elasticsearch-kubernetes
What did you expect to see?
To have an elasticsearch node up and running.
What did you see instead? Under which circumstances?
I noticed the elastic-operator to be OOMKilled:
I noticed the memory limit set for the operator is also small (only 100Mb)
https://github.com/elastic/cloud-on-k8s/blob/master/operators/config/operator/all-in-one/operator.template.yaml#L39
Environment
Script version:
https://download.elastic.co/downloads/eck/0.9.0/all-in-one.yaml
Kubernetes 1.13.5
Version information:
https://download.elastic.co/downloads/eck/0.9.0/all-in-one.yaml
EC2 on AWS, (not EKS), using Rancher 2.2.2.
Kubernetes 1.13.5
Other notes:
Interestingly enough I used the same operator on a different cluster on digital ocean, and there it didn't need more that the 100Mb limit.
I now increased the limit on my machine to 500M and it works well. (probably could have done less).
The text was updated successfully, but these errors were encountered: