PyTorch and MPI Operator pulls hardcoded initContainer

[MuhammadZeeshan34]

Problem
While running any pytorch or MPI operator job on on-prem clusters, the worker pods init-container try to pull default image i.e. alpine:3.12 in the case of pytorch and mpioperator/kubectl-delivery in the case of MPI. Since these jobs are running on-prem thus has no access to pull these images from public repositories.

In the previous version of pytorch operator, there was an option to override the default image used for init container.

Also, it can't be provided in the core deployment of training-operator since different operators require different image for initContainers.

Is there any way to override this image for specific operators?

state:
  waiting:
    message: Back-off pulling image "alpine:3.10"
    reason: ImagePullBackOff

In the previous versions when we had separate deployment of each operators we were overriding like this

spec:
  replicas: 1
  selector:
    matchLabels:
      app: pytorch-operator
      app.kubernetes.io/component: pytorch
      app.kubernetes.io/name: pytorch-operator
      kustomize.component: pytorch-operator
      name: pytorch-operator
  template:
    metadata:
      labels:
        app: pytorch-operator
        app.kubernetes.io/component: pytorch
        app.kubernetes.io/name: pytorch-operator
        kustomize.component: pytorch-operator
        name: pytorch-operator
    spec:
      containers:
      - command:
        - /pytorch-operator.v1
        - --alsologtostderr
        - -v=1
        - --monitoring-port=8443
        - --enable-gang-scheduling=true
        - --**init-container-image=<custom-deocker-repo>alpine:3.10**

[andreyvelich]

Thank you for raising this @MuhammadZeeshan34!

For the PyTorchJob and MPIJob you still can use PyTorchInitContainerImage and MPIKubectlDeliveryImage flag for the Training Operator deployment to modify the InitContainer image.

[MuhammadZeeshan34]

Thanks @andreyvelich . It has resolved the problem.