Windows 2022 with Debian locks up running Ansible

[briantist]

First I want to say I don't think this is a problem with the action, but I have run out of troubleshooting steps and hope opening a discussion might help me find an answer.

It's a strange problem: the GitHub runner completely locks up and stops responding (as far as I can tell from actions interface) when running certain Ansible commands in WSL, but only in Debian, and only on Windows 2022.

When I say it completely locks up, what I mean is that all output ceases, the step never finishes. If I cancel the workflow, the cancel operation will also not be responded to, and the cancel request will timeout (at which point some higher level process takes care of forcing it to stop).

timeout-minutes directives in the workflow have no effect.
Using the timeout command in bash/wsl-bash has no effect.

Debug logging has not given me anything useful either.

One interesting thing is that it seems like the commands that lock up the system do run, but the next step is never reached.

The output from those commands can be seen, but you must have the page open to that command to see the output. If you try to go back to it after it has locked up, the output will not be shown. It is also not included in the downloadable logs.

Using the update flag to run uodates in the distro also did not help.

Another interesting point is that the wrapper script changes things a little bit, for example using shell: wsl-bash, it locks up even just running ansible --version, however if I use shell: cmd and run wsl.exe ansible --version it will work (a few other commands work too), but ansible-galaxy collection install ... will lock up with either method.

My workflow: https://github.com/briantist/gha-junk/blob/try/.github/workflows/wsl.yml
Workflow runs: https://github.com/briantist/gha-junk/actions/workflows/wsl.yml

Here's output from `wsl-bash`

##[debug]Evaluating condition for step: 'Can we run something?'
##[debug]Evaluating: success()
##[debug]Evaluating success:
##[debug]=> true
##[debug]Result: true
##[debug]Starting: Can we run something?
##[debug]Loading inputs
##[debug]Loading env
Run pwd
##[debug]Overwrite 'shell' base on job defaults.
##[debug]D:\a\_temp\wsl-shell-wrapper\wsl-bash.BAT D:\a\_temp\1cde6051-13e6-4763-97a6-e8[2](https://github.com/briantist/gha-junk/runs/5095368630?check_suite_focus=true#step:8:2)2ca0f[3](https://github.com/briantist/gha-junk/runs/5095368630?check_suite_focus=true#step:8:3)10d
D:\a\gha-junk\gha-junk>SETLOCAL
D:\a\gha-junk\gha-junk>IF 'D:\a\_temp\1cde6051-13e6-[4](https://github.com/briantist/gha-junk/runs/5095368630?check_suite_focus=true#step:8:4)763-97a6-e822ca0f310d' EQU '' (
REM wsl-shell  
 GOTO INVALID_ARGUMENTS 
)  ELSE IF '' EQU '' (
REM wsl-shell scriptFile  
 SET scriptFile=D:\a\_temp\1cde60[5](https://github.com/briantist/gha-junk/runs/5095368630?check_suite_focus=true#step:8:5)1-13e[6](https://github.com/briantist/gha-junk/runs/5095368630?check_suite_focus=true#step:8:6)-4[7](https://github.com/briantist/gha-junk/runs/5095368630?check_suite_focus=true#step:8:7)63-97a6-e[8](https://github.com/briantist/gha-junk/runs/5095368630?check_suite_focus=true#step:8:8)22ca0f310d 
)  ELSE IF 'D:\a\_temp\1cde6051-13e6-4763-[9](https://github.com/briantist/gha-junk/runs/5095368630?check_suite_focus=true#step:8:9)7a6-e822ca0f310d' NEQ '-u' (
REM wsl-shell user scriptFile  
 GOTO INVALID_ARGUMENTS 
)  ELSE IF '' EQU '' (
REM wsl-shell -u user  
 GOTO INVALID_ARGUMENTS 
)  ELSE IF '' NEQ '' (
REM wsl-shell -u user scriptFile foo  
 GOTO INVALID_ARGUMENTS 
)  ELSE (
REM wsl-shell -u user scriptFile  
 SET wslShellUser=-u   
 SET scriptFile= 
) 
D:\a\gha-junk\gha-junk>IF NOT EXIST D:\a\_temp\1cde6051-13e6-4763-97a6-e822ca0f310d GOTO INVALID_SCRIPT_FILE 
D:\a\gha-junk\gha-junk>GOTO START 
D:\a\gha-junk\gha-junk>FOR /F "usebackq tokens=*" %F IN (`wsl -u root wslpath 'D:\a\_temp\1cde6051-13e6-4763-97a6-e822ca0f310d'`) DO SET wslScriptFile=%F 
D:\a\gha-junk\gha-junk>SET wslScriptFile=/mnt/d/a/_temp/1cde6051-13e6-4763-97a6-e822ca0f310d 
D:\a\gha-junk\gha-junk>wsl -u root sed -i 's/\r$//' '/mnt/d/a/_temp/1cde6051-13e6-4763-97a6-e822ca0f310d' 
D:\a\gha-junk\gha-junk>wsl  bash --noprofile --norc -euo pipefail '/mnt/d/a/_temp/1cde6051-13e6-4763-97a6-e822ca0f310d' 
/mnt/d/a/gha-junk/gha-junk
total 0
drwxrwxrwx 1 root root 512 Feb  7 15:[21](https://github.com/briantist/gha-junk/runs/5095368630?check_suite_focus=true#step:8:21) .
drwxrwxrwx 1 root root 512 Feb  7 15:21 ..
drwxrwxrwx 1 root root 512 Feb  7 15:21 actions
drwxrwxrwx 1 root root 512 Feb  7 15:21 .git
drwxrwxrwx 1 root root 512 Feb  7 15:21 .github
drwxrwxrwx 1 root root 512 Feb  7 15:21 inventory
-rwxrwxrwx 1 root root  18 Feb  7 15:21 test.txt
-rwxrwxrwx 1 root root  [27](https://github.com/briantist/gha-junk/runs/5095368630?check_suite_focus=true#step:8:27) Feb  7 15:21 unrelated.txt
ansible [core 2.12.2]
  config file = None
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.9/dist-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.9.2 (default, Feb [28](https://github.com/briantist/gha-junk/runs/5095368630?check_suite_focus=true#step:8:28) 2021, 17:03:[44](https://github.com/briantist/gha-junk/runs/5095368630?check_suite_focus=true#step:8:44)) [GCC 10.2.1 20210110]
  jinja version = 3.0.3
  libyaml = True

Here's some output using `cmd` and `wsl.exe` (snipped)

ansible [core 2.12.2]
  config file = None
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.9/dist-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.9.2 (default, Feb 28 2021, 17:03:44) [GCC 10.2.1 20210110]
  jinja version = 3.0.3
  libyaml = True
  3178 1644246523.08928: starting run
ansible-galaxy [core 2.12.2]
  config file = None
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.9/dist-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible-galaxy
  python version = 3.9.2 (default, Feb 28 2021, 17:03:44) [GCC 10.2.1 20210110]
  jinja version = 3.0.3
  libyaml = True
No config file found; using defaults
Creating Galaxy API response cache file at '/root/.ansible/galaxy_cache/api.json'
Galaxy cache file at '/root/.ansible/galaxy_cache/api.json' has an invalid version, clearing
  3178 1644246523.47398: Validate TLS certificates for https://galaxy.ansible.com: True
Starting galaxy collection install process
Process install dependency map
Initial connection to galaxy_server: https://galaxy.ansible.com
Created /root/.ansible/galaxy_token
Calling Galaxy at https://galaxy.ansible.com/api/
Found API version 'v1, v2' with Galaxy server default (https://galaxy.ansible.com/api/)
Calling Galaxy at https://galaxy.ansible.com/api/v2/collections/community/windows/
Calling Galaxy at https://galaxy.ansible.com/api/v2/collections/community/windows/versions/?page_size=100
Calling Galaxy at https://galaxy.ansible.com/api/v2/collections/ansible/windows/
Calling Galaxy at https://galaxy.ansible.com/api/v2/collections/ansible/windows/versions/?page_size=100
Calling Galaxy at https://galaxy.ansible.com/api/v2/collections/community/windows/versions/1.9.0/
Calling Galaxy at https://galaxy.ansible.com/api/v2/collections/ansible/windows/
Calling Galaxy at https://galaxy.ansible.com/api/v2/collections/ansible/windows/versions/1.9.0/
Starting collection install process
Fetching a collection tarball for 'community.windows:1.9.0' from Ansible Galaxy
Downloading https://galaxy.ansible.com/download/community-windows-1.9.0.tar.gz to /root/.ansible/tmp/ansible-local-31782tsposea/tmpqpc9vstr/community-windows-1.9.0-o7vr044j
Validating downloaded file hash 63909e16fc055e39266127098f7c11e9da5e6000b231be5bdac74c1758ef9e9b with expected hash 63909e16fc055e39266127098f7c11e9da5e6000b231be5bdac74c1758ef9e9b
Collection 'community.windows:1.9.0' obtained from server default https://galaxy.ansible.com/api/
Installing 'community.windows:1.9.0' to '/root/.ansible/collections/ansible_collections/community/windows'
community.windows:1.9.0 was installed successfully
Fetching a collection tarball for 'ansible.windows:1.9.0' from Ansible Galaxy
Downloading https://galaxy.ansible.com/download/ansible-windows-1.9.0.tar.gz to /root/.ansible/tmp/ansible-local-31782tsposea/tmpqpc9vstr/ansible-windows-1.9.0-jm3900cl
Validating downloaded file hash ffba8ba75cbf72f911f0e289fb0868adf2a8a2edbe3fd03b8768faf[18](https://github.com/briantist/gha-junk/runs/5095135868?check_suite_focus=true#step:8:18)4[23](https://github.com/briantist/gha-junk/runs/5095135868?check_suite_focus=true#step:8:23)79b5 with expected hash ffba8ba75cbf72f911f0e[28](https://github.com/briantist/gha-junk/runs/5095135868?check_suite_focus=true#step:8:28)9fb0868adf2a8a2edbe3fd03b8768faf1842[37](https://github.com/briantist/gha-junk/runs/5095135868?check_suite_focus=true#step:8:37)9b5
Collection 'ansible.windows:1.9.0' obtained from server default https://galaxy.ansible.com//api/
Installing 'ansible.windows:1.9.0' to '/root/.ansible/collections/ansible_collections/ansible/windows'

I'm really stuck at this point, and would appreciate any tips or ideas.

[Vampire]

I think I have seen such hangs before, but unfortunately I don't remember what the problem was or how it was resolved.
Maybe some out-of-memory watchdog kills something or similar?
I guess you have not much options left.
Here some ideas to collect more information:

• run the hanging command through strace
• run the hanging command detached in the background and in the foreground
  • tail -f logs
  • ps aux -H processes
  • ...
• try to reproduce the hang with a self-hosted runner where you then can investigate the situation
• contact GitHub support and let their engineers investigate what is hanging and why

[ssbarnea]

I was able to use tmate action to get onto the machine and all I could find was that I lose access to the machine few seconds after my tests start. That top was last command I run, few seconds after I lost connection.

Apparently machine does not run out of disk or memory, something else happens. Even forcing to run on single cpu does not fix it.

Sadly Github engineers ask for simple reproducing case but I was not able to find a single command that causes the crash. Running pytest with wsl based version of python is not really a simple example. Still, running with --collect-only seems to be working, so it must be some of the tests.

[briantist]

I agree any example with WSL is not exactly a simple repro (it is kind f simple if they will use this action); in my case with Debian + 2022, it is fairly simple to hit though, since almost any ansible command causes the crash.

[ssbarnea]

I raised official ticket with GHA at actions/runner-images#5151 -- i think it is time to get them involved as all our attempts failed. The fact that runner gets stuck and we have zero output makes it more of a critical issue that can be addressed only by them.