"failed to create containerd task: failed to create shim task: hcs::CreateComputeSystem xxxxxxx--cid--xxxxxxxx : Access is denied.: unknown"

[vinayaksakharkar]

Less than 1% of time, container failed on started. Kind is Batch & runs in parallelism. In parallelism, one of container failed while all other successfully start. In case of failed container, its pod reaches running state & then container throw up starterror with Access is denied.: unknown"

"status": {
    "phase": "Failed",
    "conditions": [
      {
        "type": "Initialized",
        "status": "True",
        "lastProbeTime": null,
        "lastTransitionTime": "2024-01-30T09:03:59Z"
      },
      {
        "type": "Ready",
        "status": "False",
        "lastProbeTime": null,
        "lastTransitionTime": "2024-01-30T09:03:59Z",
        "reason": "PodFailed"
      },
      {
        "type": "ContainersReady",
        "status": "False",
        "lastProbeTime": null,
        "lastTransitionTime": "2024-01-30T09:03:59Z",
        "reason": "PodFailed"
      },
      {
        "type": "PodScheduled",
        "status": "True",
        "lastProbeTime": null,
        "lastTransitionTime": "2024-01-30T09:03:59Z"
      }
    ],
    "hostIP": "ntentional removed IP",
    "podIP": "ntentional removed IP",
    "podIPs": [
      {
        "ip": "ntentional removed IP"
      }
    ],
    "startTime": "2024-01-30T09:03:59Z",
    "containerStatuses": [
      {
        "name": "tsk",
        "state": {
          "terminated": {
            "exitCode": 128,
            "reason": "StartError",
            "message": "failed to create containerd task: failed to create shim task: hcs::CreateComputeSystem xxxxxxx--cid--xxxxxxxx: Access is denied.: unknown",
            "startedAt": "1970-01-01T00:00:00Z",
            "finishedAt": "2024-01-30T09:04:35Z",
            "containerID": "containerd://xxxxxxx--cid--xxxxxxxx"
          }
        },
        "lastState": {},
        "ready": false,
        "restartCount": 0,
        "image": "intentional removed image name",
        "imageID": intentional removed image id,
        "containerID": "containerd://xxxxxxx--cid--xxxxxxxx",
        "started": false
      }
    ],
    "qosClass": "BestEffort"
  }