/root/helix/work/correlation/scripts/<hash>/execute.sh: Permission denied

[carlossanlop]

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=733773

Build leg reported

Various (I could not see a pattern)

Pull Request

dotnet/runtime#104578

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "execute.sh: Permission denied",
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

Output examples:

https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-104578-merge-bb499c7f2db04bf19e/System.Text.Json.Tests/1/console.2c72ad54.log?helixlogtype=result

Console log: 'System.Text.Json.Tests' from job bb499c7f-2db0-4bf1-9ef7-8b36ad24faf3 (ubuntu.2204.armarch.open) using docker image mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.17-helix-arm32v7 on a000TVE
running $HELIX_CORRELATION_PAYLOAD/scripts/c67bd92f21d14a719bae3e21249b8fcb/execute.sh in /datadisks/disk1/work/BC340A3D/w/B1B409CE/e max 2700 seconds

Output:
[BEGIN EXECUTION]
+ sudo python -m pip install --disable-pip-version-check -r /root/helix/scripts/runtime_python_requirements.txt
Ignoring cffi: markers 'python_version < "3.8"' don't match your environment
Requirement already satisfied: certifi>=2023.7.22 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 1)) (2024.6.2)
Requirement already satisfied: ndg-httpsclient==0.5.1 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 2)) (0.5.1)
Requirement already satisfied: psutil==5.9.3 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 3)) (5.9.3)
Requirement already satisfied: pyasn1==0.4.5 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 4)) (0.4.5)
Requirement already satisfied: six>=1.11.0 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 5)) (1.16.0)
Requirement already satisfied: cryptography>=39.0.1 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 7)) (42.0.8)
Requirement already satisfied: pyopenssl>=23.0.0 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 8)) (24.1.0)
Requirement already satisfied: asn1crypto==0.24.0 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 10)) (0.24.0)
Requirement already satisfied: idna==2.8 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 11)) (2.8)
Requirement already satisfied: cffi>=1.16.0 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 13)) (1.16.0)
Requirement already satisfied: pycparser==2.19 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 14)) (2.19)
Requirement already satisfied: docker==3.7.2 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 16)) (3.7.2)
Requirement already satisfied: docker-pycreds==0.4.0 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 17)) (0.4.0)
Requirement already satisfied: websocket-client==0.56.0 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 18)) (0.56.0)
Requirement already satisfied: requests<=2.28.1,>=2.21.0 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 20)) (2.28.1)
Requirement already satisfied: urllib3<1.27,>=1.24.3 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 21)) (1.26.19)
Requirement already satisfied: azure-storage-blob==12.5.0 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 23)) (12.5.0)
Requirement already satisfied: azure-core==1.7.0 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 24)) (1.7.0)
Requirement already satisfied: azure-devops==5.0.0b9 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 25)) (5.0.0b9)
Requirement already satisfied: msrest==0.6.10 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 26)) (0.6.10)
Requirement already satisfied: isodate==0.6.0 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 27)) (0.6.0)
Requirement already satisfied: requests-oauthlib==1.2.0 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 28)) (1.2.0)
Requirement already satisfied: oauthlib==3.1.0 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 29)) (3.1.0)
Requirement already satisfied: charset-normalizer<3,>=2 in /usr/lib/python3.10/site-packages (from requests<=2.28.1,>=2.21.0->-r /root/helix/scripts/runtime_python_requirements.txt (line 20)) (2.1.1)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
+ export 'PYTHONPATH=/root/helix/scripts:'
+ cd /root/helix/work/workitem/e
+ mkdir -p /home/helixbot/dotnetbuild/dumps/
+ /root/helix/work/correlation/scripts/c67bd92f21d14a719bae3e21249b8fcb/execute.sh
/root/commands/helix_docker_work.sh: line 7: /root/helix/work/correlation/scripts/c67bd92f21d14a719bae3e21249b8fcb/execute.sh: Permission denied
+ export '_commandExitCode=126'
+ chmod -R 777 /home/helixbot/dotnetbuild/dumps
+ exit 126

[END EXECUTION]
Exit Code:126

https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-104578-merge-bb499c7f2db04bf19e/System.Reflection.Emit.ILGeneration.Tests/1/console.83a562de.log?helixlogtype=result

Console log: 'System.Reflection.Emit.ILGeneration.Tests' from job bb499c7f-2db0-4bf1-9ef7-8b36ad24faf3 (ubuntu.2204.armarch.open) using docker image mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.17-helix-arm32v7 on a000TVE
running $HELIX_CORRELATION_PAYLOAD/scripts/2cf1fa015b6347e1b418539a2fef97a0/execute.sh in /datadisks/disk1/work/BC340A3D/w/AB740946/e max 2700 seconds

Output:
[BEGIN EXECUTION]
+ sudo python -m pip install --disable-pip-version-check -r /root/helix/scripts/runtime_python_requirements.txt
Ignoring cffi: markers 'python_version < "3.8"' don't match your environment
Requirement already satisfied: certifi>=2023.7.22 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 1)) (2024.6.2)
Requirement already satisfied: ndg-httpsclient==0.5.1 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 2)) (0.5.1)
Requirement already satisfied: psutil==5.9.3 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 3)) (5.9.3)
Requirement already satisfied: pyasn1==0.4.5 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 4)) (0.4.5)
Requirement already satisfied: six>=1.11.0 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 5)) (1.16.0)
Requirement already satisfied: cryptography>=39.0.1 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 7)) (42.0.8)
Requirement already satisfied: pyopenssl>=23.0.0 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 8)) (24.1.0)
Requirement already satisfied: asn1crypto==0.24.0 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 10)) (0.24.0)
Requirement already satisfied: idna==2.8 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 11)) (2.8)
Requirement already satisfied: cffi>=1.16.0 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 13)) (1.16.0)
Requirement already satisfied: pycparser==2.19 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 14)) (2.19)
Requirement already satisfied: docker==3.7.2 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 16)) (3.7.2)
Requirement already satisfied: docker-pycreds==0.4.0 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 17)) (0.4.0)
Requirement already satisfied: websocket-client==0.56.0 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 18)) (0.56.0)
Requirement already satisfied: requests<=2.28.1,>=2.21.0 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 20)) (2.28.1)
Requirement already satisfied: urllib3<1.27,>=1.24.3 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 21)) (1.26.19)
Requirement already satisfied: azure-storage-blob==12.5.0 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 23)) (12.5.0)
Requirement already satisfied: azure-core==1.7.0 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 24)) (1.7.0)
Requirement already satisfied: azure-devops==5.0.0b9 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 25)) (5.0.0b9)
Requirement already satisfied: msrest==0.6.10 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 26)) (0.6.10)
Requirement already satisfied: isodate==0.6.0 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 27)) (0.6.0)
Requirement already satisfied: requests-oauthlib==1.2.0 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 28)) (1.2.0)
Requirement already satisfied: oauthlib==3.1.0 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 29)) (3.1.0)
Requirement already satisfied: charset-normalizer<3,>=2 in /usr/lib/python3.10/site-packages (from requests<=2.28.1,>=2.21.0->-r /root/helix/scripts/runtime_python_requirements.txt (line 20)) (2.1.1)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
+ export 'PYTHONPATH=/root/helix/scripts:'
+ cd /root/helix/work/workitem/e
+ mkdir -p /home/helixbot/dotnetbuild/dumps/
+ /root/helix/work/correlation/scripts/2cf1fa015b6347e1b418539a2fef97a0/execute.sh
/root/commands/helix_docker_work.sh: line 7: /root/helix/work/correlation/scripts/2cf1fa015b6347e1b418539a2fef97a0/execute.sh: Permission denied
+ export '_commandExitCode=126'
+ chmod -R 777 /home/helixbot/dotnetbuild/dumps
+ exit 126

[END EXECUTION]
Exit Code:126

https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-104578-merge-bb499c7f2db04bf19e/System.Configuration.ConfigurationManager.Tests/1/console.8a497b96.log?helixlogtype=result

Console log: 'System.Configuration.ConfigurationManager.Tests' from job bb499c7f-2db0-4bf1-9ef7-8b36ad24faf3 (ubuntu.2204.armarch.open) using docker image mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.17-helix-arm32v7 on a000TVE
running $HELIX_CORRELATION_PAYLOAD/scripts/0750deb016234b008a80ce02551e0593/execute.sh in /datadisks/disk1/work/BC340A3D/w/9EE508A6/e max 2700 seconds

Output:
[BEGIN EXECUTION]
+ sudo python -m pip install --disable-pip-version-check -r /root/helix/scripts/runtime_python_requirements.txt
Ignoring cffi: markers 'python_version < "3.8"' don't match your environment
Requirement already satisfied: certifi>=2023.7.22 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 1)) (2024.6.2)
Requirement already satisfied: ndg-httpsclient==0.5.1 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 2)) (0.5.1)
Requirement already satisfied: psutil==5.9.3 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 3)) (5.9.3)
Requirement already satisfied: pyasn1==0.4.5 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 4)) (0.4.5)
Requirement already satisfied: six>=1.11.0 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 5)) (1.16.0)
Requirement already satisfied: cryptography>=39.0.1 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 7)) (42.0.8)
Requirement already satisfied: pyopenssl>=23.0.0 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 8)) (24.1.0)
Requirement already satisfied: asn1crypto==0.24.0 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 10)) (0.24.0)
Requirement already satisfied: idna==2.8 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 11)) (2.8)
Requirement already satisfied: cffi>=1.16.0 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 13)) (1.16.0)
Requirement already satisfied: pycparser==2.19 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 14)) (2.19)
Requirement already satisfied: docker==3.7.2 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 16)) (3.7.2)
Requirement already satisfied: docker-pycreds==0.4.0 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 17)) (0.4.0)
Requirement already satisfied: websocket-client==0.56.0 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 18)) (0.56.0)
Requirement already satisfied: requests<=2.28.1,>=2.21.0 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 20)) (2.28.1)
Requirement already satisfied: urllib3<1.27,>=1.24.3 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 21)) (1.26.19)
Requirement already satisfied: azure-storage-blob==12.5.0 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 23)) (12.5.0)
Requirement already satisfied: azure-core==1.7.0 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 24)) (1.7.0)
Requirement already satisfied: azure-devops==5.0.0b9 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 25)) (5.0.0b9)
Requirement already satisfied: msrest==0.6.10 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 26)) (0.6.10)
Requirement already satisfied: isodate==0.6.0 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 27)) (0.6.0)
Requirement already satisfied: requests-oauthlib==1.2.0 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 28)) (1.2.0)
Requirement already satisfied: oauthlib==3.1.0 in /usr/lib/python3.10/site-packages (from -r /root/helix/scripts/runtime_python_requirements.txt (line 29)) (3.1.0)
Requirement already satisfied: charset-normalizer<3,>=2 in /usr/lib/python3.10/site-packages (from requests<=2.28.1,>=2.21.0->-r /root/helix/scripts/runtime_python_requirements.txt (line 20)) (2.1.1)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
+ export 'PYTHONPATH=/root/helix/scripts:'
+ cd /root/helix/work/workitem/e
+ mkdir -p /home/helixbot/dotnetbuild/dumps/
+ /root/helix/work/correlation/scripts/0750deb016234b008a80ce02551e0593/execute.sh
/root/commands/helix_docker_work.sh: line 7: /root/helix/work/correlation/scripts/0750deb016234b008a80ce02551e0593/execute.sh: Permission denied
+ export '_commandExitCode=126'
+ chmod -R 777 /home/helixbot/dotnetbuild/dumps
+ exit 126

[END EXECUTION]
Exit Code:126

### Known issue validation **Build: 🔎** https://dev.azure.com/dnceng-public/public/_build/results?buildId=733773 **Error message validated:** `[execute.sh: Permission denied`] **Result validation:** ✅ Known issue matched with the provided build. **Validation performed at:** 7/12/2024 8:13:22 PM UTC

Report

Build	Definition	Test	Pull Request
813575	dotnet/runtime	System.Text.Json.SourceGeneration.Roslyn3.11.Unit.Tests.WorkItemExecution
796401	dotnet/runtime	System.IO.Compression.Brotli.Tests.WorkItemExecution	dotnet/runtime#107297

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	1	2

[dougbu]

symptom of Alpine docker image problems that would likely be mitigated if we could ever find time to at least scream test a #1494 fix. any version of this show up outside Docker or in a non-Alpine container @carlossanlop❔