Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Self host crush - is not authorized to access this resource #3380

Open
noamgreen opened this issue Jul 10, 2024 · 3 comments
Open

Self host crush - is not authorized to access this resource #3380

noamgreen opened this issue Jul 10, 2024 · 3 comments
Labels
bug Something isn't working

Comments

@noamgreen
Copy link

Describe the bug
I start running a job and i geting this error many times and the job is cancelled

System.UnauthorizedAccessException: Access to the path '/proc/554/oom_score_adj' is denied.

test-t82bf-runner-vcgwt runner [WORKER 2024-07-10 13:30:38Z INFO ProcessInvokerWrapper] Starting process:
test-t82bf-runner-vcgwt runner [WORKER 2024-07-10 13:30:38Z INFO ProcessInvokerWrapper]   File name: '/usr/bin/bash'
test-t82bf-runner-vcgwt runner [WORKER 2024-07-10 13:30:38Z INFO ProcessInvokerWrapper]   Arguments: '--noprofile --norc -e -o pipefail /home/runner/_work/_temp/0e164abb-bd20-4274-91b5-72f26c8fae7f.sh'
test-t82bf-runner-vcgwt runner [WORKER 2024-07-10 13:30:38Z INFO ProcessInvokerWrapper]   Working directory: '/home/runner/_work/devops'
test-t82bf-runner-vcgwt runner [WORKER 2024-07-10 13:30:38Z INFO ProcessInvokerWrapper]   Require exit code zero: 'False'
test-t82bf-runner-vcgwt runner [WORKER 2024-07-10 13:30:38Z INFO ProcessInvokerWrapper]   Encoding web name:  ; code page: ''
test-t82bf-runner-vcgwt runner [WORKER 2024-07-10 13:30:38Z INFO ProcessInvokerWrapper]   Force kill process on cancellation: 'False'
test-t82bf-runner-vcgwt runner [WORKER 2024-07-10 13:30:38Z INFO ProcessInvokerWrapper]   Redirected STDIN: 'False'
test-t82bf-runner-vcgwt runner [WORKER 2024-07-10 13:30:38Z INFO ProcessInvokerWrapper]   Persist current code page: 'False'
test-t82bf-runner-vcgwt runner [WORKER 2024-07-10 13:30:38Z INFO ProcessInvokerWrapper]   Keep redirected STDIN open: 'False'
test-t82bf-runner-vcgwt runner [WORKER 2024-07-10 13:30:38Z INFO ProcessInvokerWrapper]   High priority process: 'False'
test-t82bf-runner-vcgwt runner [WORKER 2024-07-10 13:30:38Z INFO ProcessInvokerWrapper] Failed to update oom_score_adj for PID: 554.
test-t82bf-runner-vcgwt runner [WORKER 2024-07-10 13:30:38Z INFO ProcessInvokerWrapper] System.UnauthorizedAccessException: Access to the path '/proc/554/oom_score_adj' is denied.
test-t82bf-runner-vcgwt runner [WORKER 2024-07-10 13:30:38Z INFO ProcessInvokerWrapper]  ---> System.IO.IOException: Permission denied
test-t82bf-runner-vcgwt runner [WORKER 2024-07-10 13:30:38Z INFO ProcessInvokerWrapper]    --- End of inner exception stack trace ---
test-t82bf-runner-vcgwt runner [WORKER 2024-07-10 13:30:38Z INFO ProcessInvokerWrapper]    at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan`1 buffer, Int64 fileOffset)
test-t82bf-runner-tzkbq runner [WORKER 2024-07-10 13:30:38Z INFO JobServerQueue] Try to append 1 batches web console lines for record 'd8be3eeb-808b-5cb1-9c71-884a43769e10', success rate: 1/1.
test-t82bf-runner-vcgwt runner [WORKER 2024-07-10 13:30:38Z INFO ProcessInvokerWrapper]    at System.IO.Strategies.BufferedFileStreamStrategy.FlushWrite()
test-t82bf-runner-vcgwt runner [WORKER 2024-07-10 13:30:38Z INFO ProcessInvokerWrapper]    at System.IO.Strategies.BufferedFileStreamStrategy.Dispose(Boolean disposing)
test-t82bf-runner-vcgwt runner [WORKER 2024-07-10 13:30:38Z INFO ProcessInvokerWrapper]    at System.IO.StreamWriter.CloseStreamFromDispose(Boolean disposing)
test-t82bf-runner-vcgwt runner [WORKER 2024-07-10 13:30:38Z INFO ProcessInvokerWrapper]    at System.IO.StreamWriter.Dispose(Boolean disposing)
test-t82bf-runner-vcgwt runner [WORKER 2024-07-10 13:30:38Z INFO ProcessInvokerWrapper]    at System.IO.File.WriteAllText(String path, String contents)
test-t82bf-runner-vcgwt runner [WORKER 2024-07-10 13:30:38Z INFO ProcessInvokerWrapper]    at GitHub.Runner.Sdk.ProcessInvoker.WriteProcessOomScoreAdj(Int32 processId, Int32 oomScoreAdj)

test-t82bf-runner-bzrls runner [RUNNER 2024-07-10 13:29:47Z WARN GitHubActionsService] GET request to https://pipelinesghubeus26.actions.githubusercontent.com/0TagnS0qaEOWaiMRmM6TEfnPxHtFutTo92gSnylMXf5cdGZrsW/_apis/distributedtask/pools/10/messages?sessionId=7259e804-7eac-4621-ba0e-6bcd61f8a395&lastMessageId=1&status=Busy&runnerVersion=2.317.0&os=Linux&architecture=X64&disableUpdate=true has been cancelled.
test-t82bf-runner-bzrls runner [RUNNER 2024-07-10 13:29:47Z INFO MessageListener] Get next message has been cancelled.
test-t82bf-runner-bzrls runner [RUNNER 2024-07-10 13:29:47Z INFO JobDispatcher] Shutting down JobDispatcher. Make sure all WorkerDispatcher has finished.
test-t82bf-runner-bzrls runner [RUNNER 2024-07-10 13:29:47Z INFO JobDispatcher] Ensure WorkerDispatcher for job 1cfa91de-17dd-561a-f553-3e6471df7df2 run to finish, cancel any running job.
test-t82bf-runner-bzrls runner [RUNNER 2024-07-10 13:29:47Z INFO JobDispatcher] Send job cancellation message to worker for job 1cfa91de-17dd-561a-f553-3e6471df7df2.
test-t82bf-runner-bzrls runner [RUNNER 2024-07-10 13:29:47Z INFO ProcessChannel] Sending message of length 0, with hash 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'
test-t82bf-runner-bzrls runner [WORKER 2024-07-10 13:29:47Z INFO ProcessChannel] Receiving message of length 0, with hash 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'
test-t82bf-runner-bzrls runner [WORKER 2024-07-10 13:29:47Z INFO Worker] Cancellation/Shutdown message received.
test-t82bf-runner-bzrls runner [WORKER 2024-07-10 13:29:47Z INFO HostContext] Runner will be shutdown for UserCancelled
test-t82bf-runner-bzrls runner [WORKER 2024-07-10 13:29:47Z INFO JobServerQueue] Try to append 1 batches web console lines for record '1cfa91de-17dd-561a-f553-3e6471df7df2', success rate: 1/1.
test-t82bf-runner-bzrls runner [WORKER 2024-07-10 13:29:47Z INFO StepsRunner] Cancel current running step.
test-t82bf-runner-bzrls runner [WORKER 2024-07-10 13:29:47Z INFO ProcessInvokerWrapper] Sending SIGINT to process 175.
test-t82bf-runner-bzrls runner [WORKER 2024-07-10 13:29:47Z INFO ProcessInvokerWrapper] Successfully send SIGINT to process 175.
test-t82bf-runner-bzrls runner [WORKER 2024-07-10 13:29:47Z INFO ProcessInvokerWrapper] Waiting for process exit or 7.5 seconds after SIGINT signal fired.
test-t82bf-runner-bzrls runner [WORKER 2024-07-10 13:29:47Z INFO ProcessInvokerWrapper] STDOUT/STDERR stream read finished.
test-t82bf-runner-bzrls runner [WORKER 2024-07-10 13:29:47Z INFO ProcessInvokerWrapper] STDOUT/STDERR stream read finished.
test-t82bf-runner-bzrls runner [WORKER 2024-07-10 13:29:47Z INFO ProcessInvokerWrapper] Process exit successfully.
test-t82bf-runner-bzrls runner [WORKER 2024-07-10 13:29:47Z INFO ProcessInvokerWrapper] Process cancelled successfully through Ctrl+C/SIGINT.
test-t82bf-runner-bzrls runner [WORKER 2024-07-10 13:29:47Z INFO ProcessInvokerWrapper] Process Cancellation finished.
test-t82bf-runner-bzrls runner [WORKER 2024-07-10 13:29:47Z INFO ProcessInvokerWrapper] Finished process 175 with exit code 130, and elapsed time 00:00:48.9883086.
test-t82bf-runner-bzrls runner [WORKER 2024-07-10 13:29:47Z INFO CreateStepSummaryCommand] Step Summary file (/home/runner/_work/_temp/_runner_file_commands/step_summary_6931528a-638b-4f31-8479-e73982adfb14) is empty; skipping attachment upload
test-t82bf-runner-bzrls runner [WORKER 2024-07-10 13:29:47Z ERR  StepsRunner] Caught cancellation exception from step: System.OperationCanceledException: The operation was canceled.
test-t82bf-runner-bzrls runner [WORKER 2024-07-10 13:29:47Z ERR  StepsRunner]    at System.Threading.CancellationToken.ThrowOperationCanceledException()
test-t82bf-runner-bzrls runner [WORKER 2024-07-10 13:29:47Z ERR  StepsRunner]    at GitHub.Runner.Sdk.ProcessInvoker.ExecuteAsync(String workingDirectory, String fileName, String arguments, IDictionary`2 environment, Boolean requireExitCodeZero, Encoding outputEncoding, Boolean killProcessOnCancel, Channel`1 redirectStandardIn, Boolean inheritConsoleHandler, Boolean keepStandardInOpen, Boolean highPriorityProcess, CancellationToken cancellationToken)
test-t82bf-runner-bzrls runner [WORKER 2024-07-10 13:29:47Z ERR  StepsRunner]    at GitHub.Runner.Common.ProcessInvokerWrapper.ExecuteAsync(String workingDirectory, String fileName, String arguments, IDictionary`2 environment, Boolean requireExitCodeZero, Encoding outputEncoding, Boolean killProcessOnCancel, Channel`1 redirectStandardIn, Boolean inheritConsoleHandler, Boolean keepStandardInOpen, Boolean highPriorityProcess, CancellationToken cancellationToken)
test-t82bf-runner-bzrls runner [WORKER 2024-07-10 13:29:47Z ERR  StepsRunner]    at GitHub.Runner.Worker.Handlers.DefaultStepHost.ExecuteAsync(IExecutionContext context, String workingDirectory, String fileName, String arguments, IDictionary`2 environment, Boolean requireExitCodeZero, Encoding outputEncoding, Boolean killProcessOnCancel, Boolean inheritConsoleHandler, String standardInInput, CancellationToken cancellationToken)
test-t82bf-runner-bzrls runner [WORKER 2024-07-10 13:29:47Z ERR  StepsRunner]    at GitHub.Runner.Worker.Handlers.ScriptHandler.RunAsync(ActionRunStage stage)
test-t82bf-runner-bzrls runner [WORKER 2024-07-10 13:29:47Z ERR  StepsRunner]    at GitHub.Runner.Worker.ActionRunner.RunAsync()
test-t82bf-runner-bzrls runner [WORKER 2024-07-10 13:29:47Z ERR  StepsRunner]    at GitHub.Runner.Worker.StepsRunner.RunStepAsync(IStep step, CancellationToken jobCancellationToken)
test-t82bf-runner-bzrls runner [WORKER 2024-07-10 13:29:47Z INFO StepsRunner] Step result: Canceled```

{
"$id": "1",
"innerException": null,
"message": "The user 'System:PublicAccess;aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa' is not authorized to access this resource.",
"typeName": "Microsoft.TeamFoundation.Framework.Server.UnauthorizedRequestException, Microsoft.TeamFoundation.Framework.Server",
"typeKey": "UnauthorizedRequestException",
"errorCode": 0,
"eventId": 3000
}


 i am not sure why this happens  , working on eks + dind 

**To Reproduce**
Steps to reproduce the behavior:
NA 


## Runner Version and Platform
2.317.0
OS of the machine running the runner? OSX/Windows/Linux/...

## What's not working?
job is not finishing 

## Job Log Output
If applicable, include the relevant part of the job / step log output here.  All sensitive information should already be masked out, but please double-check before pasting here.

@noamgreen noamgreen added the bug Something isn't working label Jul 10, 2024
@noamgreen noamgreen changed the title Self host crush Self host crush - is not authorized to access this resource Jul 10, 2024
@noamgreen
Copy link
Author

So i did full checkup .... i can explain the issues but i cant explain the error . long story short the runner is burst the CPU, MEM to a point of OOM and if you use k8s you will star evection process

i think the runner needs to have some mem + cpu limits ( k8s "resources" will not work ) not sure why i see this CPU and MEM issues

@joennlae
Copy link

joennlae commented Aug 3, 2024

I had the same issue. I also saw some other issues in the same direction here, for conservation and also if someone else has this issue. I had to increase the shm-size of the docker container.

    container:
      image: your/image:latest
      options: --shm-size=4gb # otherwise we get a bus error (core dumped), do not ask me how long it took to debug this :-)

Maybe this helps someone :-)

@noamgreen
Copy link
Author

@joennlae i try to fix for me its not helped i think its in the runner code its try to update in /proc/ and its cant

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants