self-hosted action runner with kubernetes mode on EKS failed at Initialize containers step from Action UI

[joosangkim]

Describe the bug
ghcr.io/actions/actions-runner:2.317.0 self-hosted runner with kubernetes mode on EKS failed at Initialize containers step from Action UI.

To Reproduce
Steps to reproduce the behavior:

1. Generate Runner deploy with the values.yaml

runnerScaleSetName: "arc-runner"
githubConfigUrl: ####MASKED####
githubConfigSecret: ####MASKED####
  
containerMode:
  type: "kubernetes" 
  kubernetesModeWorkVolumeClaim:
    accessModes: ["ReadWriteOnce"]
    storageClassName: "gp2"
    resources:
      requests:
        storage: 2Gi

template:
  spec:
    initContainers: 
    - name: kube-init
      image: ghcr.io/actions/actions-runner:2.317.0
      command: ["sudo", "chown", "-R", "runner:runner", "/home/runner/_work"]
      volumeMounts:
      - name: work
        mountPath: /home/runner/_work
    containers:
    - name: runner
      image: ghcr.io/actions/actions-runner:2.317.0
      command: ["/home/runner/run.sh"]
      env:
        - name: ACTIONS_RUNNER_CONTAINER_HOOKS
          value: /home/runner/k8s/index.js
        - name: ACTIONS_RUNNER_POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: ACTIONS_RUNNER_REQUIRE_JOB_CONTAINER
          value: "false"
      volumeMounts:
        - name: work
          mountPath: /home/runner/_work
    nodeSelector:
      karpenter.sh/capacity-type: spot
    volumes:
      - name: work
        ephemeral:
          volumeClaimTemplate:
            spec:
              accessModes: [ "ReadWriteOnce" ]
              storageClassName: "gp2"
              resources:
                requests:
                  storage: 1Gi

2. Set .github/workflows/test.yaml

name: "[ARC] CI Test "

on:
  pull_request:

jobs:
  CI-Test:
    permissions:
      contents: read
      issues: read
      id-token: write
      checks: write
      pull-requests: write
    runs-on: arc-runner
    services:
      mysql: .......
    steps:
      - uses: actions/checkout@v3
      - name: Setup Python
        run: |
          sudo apt-get update
          sudo apt-get install -y gcc python3-dev

3. See error in Initialize containers step in Action UI.
4. Check runner pod log

[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper] Starting process:
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper]   File name: '/home/runner/externals/node16/bin/node'
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper]   Arguments: '/home/runner/k8s/index.js'
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper]   Working directory: '/home/runner/_work/MASKED'
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper]   Require exit code zero: 'False'
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper]   Encoding web name:  ; code page: ''
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper]   Force kill process on cancellation: 'False'
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper]   Redirected STDIN: 'True'
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper]   Persist current code page: 'False'
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper]   Keep redirected STDIN open: 'False'
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper]   High priority process: 'False'
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper] Failed to update oom_score_adj for PID: 86.
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper] System.UnauthorizedAccessException: Access to the path '/proc/86/oom_score_adj' is denied.
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper]  ---> System.IO.IOException: Permission denied
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper]    --- End of inner exception stack trace ---
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper]    at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan`1 buffer, Int64 fileOffset)
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper]    at System.IO.Strategies.BufferedFileStreamStrategy.FlushWrite()
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper]    at System.IO.Strategies.BufferedFileStreamStrategy.Dispose(Boolean disposing)
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper]    at System.IO.StreamWriter.CloseStreamFromDispose(Boolean disposing)
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper]    at System.IO.StreamWriter.Dispose(Boolean disposing)
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper]    at System.IO.File.WriteAllText(String path, String contents)
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper]    at GitHub.Runner.Sdk.ProcessInvoker.WriteProcessOomScoreAdj(Int32 processId, Int32 oomScoreAdj)
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper] Process started with process id 86, waiting for process exit.
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper] Close STDIN after the first redirect finished.
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper] STDIN stream write finished.
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper] STDOUT/STDERR stream read finished.
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper] STDOUT/STDERR stream read finished.
[WORKER 2024-07-04 17:58:34Z INFO ProcessInvokerWrapper] Finished process 86 with exit code 1, and elapsed time 00:00:00.2572861.
[WORKER 2024-07-04 17:58:34Z ERR  StepsRunner] Caught exception from step: System.Exception: Executing the custom container implementation failed. Please contact your self hosted runner administrator.
[WORKER 2024-07-04 17:58:34Z ERR  StepsRunner]  ---> System.Exception: The hook script at '/home/runner/k8s/index.js' running command 'PrepareJob' did not execute successfully
[WORKER 2024-07-04 17:58:34Z ERR  StepsRunner]    at GitHub.Runner.Worker.Container.ContainerHooks.ContainerHookManager.ExecuteHookScript[T](IExecutionContext context, HookInput input, ActionRunStage stage, String prependPath)
[WORKER 2024-07-04 17:58:34Z ERR  StepsRunner]    --- End of inner exception stack trace ---
[WORKER 2024-07-04 17:58:34Z ERR  StepsRunner]    at GitHub.Runner.Worker.Container.ContainerHooks.ContainerHookManager.ExecuteHookScript[T](IExecutionContext context, HookInput input, ActionRunStage stage, String prependPath)
[WORKER 2024-07-04 17:58:34Z ERR  StepsRunner]    at GitHub.Runner.Worker.Container.ContainerHooks.ContainerHookManager.PrepareJobAsync(IExecutionContext context, List`1 containers)
[WORKER 2024-07-04 17:58:34Z ERR  StepsRunner]    at GitHub.Runner.Worker.ContainerOperationProvider.StartContainersAsync(IExecutionContext executionContext, Object data)
[WORKER 2024-07-04 17:58:34Z ERR  StepsRunner]    at GitHub.Runner.Worker.JobExtensionRunner.RunAsync()
[WORKER 2024-07-04 17:58:34Z ERR  StepsRunner]    at GitHub.Runner.Worker.StepsRunner.RunStepAsync(IStep step, CancellationToken jobCancellationToken)

The problem is "PrepareJob" step failed because of Access to the path '/proc/<PID>/oom_score_adj' is denied. even though I changed a permission on working directory. There were a similar issue already reported as a bug.

Expected behavior
kubernetes mode should work like dind mode.

Runner Version and Platform

Running on EKS
Runner: 2.317.0

What's not working?

Please include error messages and screenshots.

Job Log Output

Outputs from Action UI

##[debug]Evaluating condition for step: 'Initialize containers'
##[debug]Evaluating: success()
##[debug]Evaluating success:
##[debug]=> true
##[debug]Result: true
##[debug]Starting: Initialize containers
##[debug]Register post job cleanup for stopping/deleting containers.
Run '/home/runner/k8s/index.js'
  shell: /home/runner/externals/node16/bin/node {0}
##[debug]/home/runner/externals/node16/bin/node /home/runner/k8s/index.js
Error: Error: Job Container is required.
Error: Process completed with exit code 1.
Error: Executing the custom container implementation failed. Please contact your self hosted runner administrator.
##[debug]System.Exception: Executing the custom container implementation failed. Please contact your self hosted runner administrator.
##[debug] ---> System.Exception: The hook script at '/home/runner/k8s/index.js' running command 'PrepareJob' did not execute successfully
##[debug]   at GitHub.Runner.Worker.Container.ContainerHooks.ContainerHookManager.ExecuteHookScript[T](IExecutionContext context, HookInput input, ActionRunStage stage, String prependPath)
##[debug]   --- End of inner exception stack trace ---
##[debug]   at GitHub.Runner.Worker.Container.ContainerHooks.ContainerHookManager.ExecuteHookScript[T](IExecutionContext context, HookInput input, ActionRunStage stage, String prependPath)
##[debug]   at GitHub.Runner.Worker.Container.ContainerHooks.ContainerHookManager.PrepareJobAsync(IExecutionContext context, List`1 containers)
##[debug]   at GitHub.Runner.Worker.ContainerOperationProvider.StartContainersAsync(IExecutionContext executionContext, Object data)
##[debug]   at GitHub.Runner.Worker.JobExtensionRunner.RunAsync()
##[debug]   at GitHub.Runner.Worker.StepsRunner.RunStepAsync(IStep step, CancellationToken jobCancellationToken)
##[debug]Finishing: Initialize containers

Runner and Worker's Diagnostic Logs

See above

[joosangkim]

I just had few more test crosschecking with dind mode, I should define container: in github action workflow file. Is there any instruction this functional restriction on kubernetes mode?