OOM kill of the pelorus operator controller manager

[mpryc]

Another OOM Kill of the pelorus-operator-controller-manager was spotted.

One approach is to update documentation to ensure users do know how to adjust them dynamically.

It's unclear why this OOM Kill happened, from this diagram we can see it did not reach limit of 512MiB, however maybe node has not enough memory to allocate for this pod ?

[mpryc]

To check the node resource usage/limits:

$ oc describe pod pelorus-operator-controller -n pelorus | grep Node\:
Node:         ip-10-0-164-133.eu-central-1.compute.internal/10.0.164.133

Describe the node (remove /IP from the Node: output):

$ oc describe node ip-10-0-164-133.eu-central-1.compute.internal
[...]
  Resource                    Requests      Limits
  --------                    --------      ------
  cpu                         803m (22%)    1400m (40%)
  memory                      2076Mi (13%)  2036Mi (13%)
  ephemeral-storage           0 (0%)        0 (0%)
  hugepages-1Gi               0 (0%)        0 (0%)
  hugepages-2Mi               0 (0%)        0 (0%)
  attachable-volumes-aws-ebs  0             0
[...]

[mpryc]

After short debug session:

1. OpenShift cluster version: 4.9.51
2. The node has no problem with resources. Plenty to accommodate, so it's definitely not over-committed.
3. Pelorus helm operator manager is OOMKilled almost every minute
4. There were some spikes where pelorus-operator-manager was taking more then 512MB, around 530MB
5. The pelorus-operator-manager is using mostly 200MB, so way below 512MB limit. It has requested 512MB and limit set to the same number, which means it won't overcommit
6. On the shell inside the pelorus-operator-manager pod:

ls /proc/sys/vm/overcommit_*; cat /proc/sys/vm/overcommit_*
0
1
50

7. Wasn't able to adjust / perm denied:

echo -1000 /proc/1/oom_score_adj

[mpryc]

After discussing with @dmesser it may be required to adjust Maximum Concurrent Reconciles:

https://sdk.operatorframework.io/docs/building-operators/helm/reference/advanced_features/max_concurrent_reconciles/

[mpryc]

@milles9393
Please try the following:

oc edit csv -n pelorus pelorus-operator.v0.0.1

Find the containers: section and add the - --max-concurrent-reconciles=, by default it's set to 1:

[...]
                        values:
                        - linux
              containers:
              - args:
                - --health-probe-bind-address=:8081
                - --metrics-bind-address=127.0.0.1:8080
                - --max-concurrent-reconciles=5
                - --leader-elect
                - --leader-election-id=pelorus-operator
                image: quay.io/pelorus/pelorus-operator:0.0.1
                livenessProbe:
[...]

Then save it and we will see if there was any improvement.

[weshayutin]

Reading list for wes:
operator-framework/helm-operator-plugins#198
operator-framework/operator-sdk#6026

[mpryc]

The bug should be closed now:

• We released Pelorus Operator v0.0.2 which is already in the OpenShift Marketplace with the fix (Pelorus Operator v0.0.2 #786).

Note that our documentation on the https://pelorus.readthedocs.io/ is currently up to date and reviewed for the Pelrous Operator scenario.