Training Operator pod failed to start on OCP 4.10.30 with error "memory limit too low"

[vtlrazin]

Hi,
The training operator failed to start in OpenShift cluster v4.10.30 with error:

# oc -n kubeflow get pod
NAME                                 READY   STATUS                 RESTARTS   AGE
training-operator-5cc8cdfdd6-fpthk   0/1     CreateContainerError   0          2m29s

Warning  Failed  113s   kubelet  Error: container create failed: time="2022-09-14T11:35:13Z" level=error msg="runc create failed: unable to start container process: container init was OOM-killed (memory limit too low?)"

System info:
OCP - v4.10.30
Worker node - NVIDIA DGX A100

The proposed solution to resolve the issue to increase the limit in daemonset deployment to:

        resources:
          limits:
            cpu: 500m
            memory: 300Mi

Best regards,
Vitaliy