Poe/Poetry issue on Windows

[arzamas500]

Environment:

• poethepoet-0.10.0
• poetry-1.1.6

Problem using Poe on Windows.

[tool.poe.tasks]
test = "poetry build"

Expected:
No error.

Actual Result:

  FileNotFoundError

  [WinError 2] The system cannot find the file specified

  at C:\Python39\lib\subprocess.py:1420 in _execute_child
      1416│             sys.audit("subprocess.Popen", executable, args, cwd, env)
      1417│
      1418│             # Start the process
      1419│             try:
    → 1420│                 hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
      1421│                                          # no special security
      1422│                                          None, None,
      1423│                                          int(not close_fds),
      1424│                                          creationflags,

Non-portable workaround.

[tool.poe.tasks]
test = "poetry.bat build"

Not sure if it is a Poe problem, because Poetry has no exe file in distribution, and Popen will fail running just "poetry" from the PATH without bat extension.

 Directory of C:\Users\a_selivanov\.poetry\bin

21.04.2021  20:56    <DIR>          .
21.04.2021  20:56    <DIR>          ..
21.04.2021  20:56               475 poetry
21.04.2021  20:56                59 poetry.bat

[nat-n]

Hi @arzamas500, Indeed if the poetry executable is not on the path by that name to be found when executing a subprocess then I wouldn't expect it to work. Does running poetry from the host shell work? (if so then how? I don't know much about CMD/powershell so I'm genuinely curious).

It's not clear to me that there's anything poe can or should to differently in this situation, but I'll leave this issue open for a bit in case you can help me understand otherwise.

I guess specifying the .bat suffix an acceptable work around?

[arzamas500]

Poetry on the host shell works perfectly, the problem appears with Popen only.
Should work when this one is fixed python-poetry/poetry#3339 The problem is that poetry itself is bat file.

[nat-n]

@arzamas500 well that's a relief. I'll close this issue then if the root cause has been located elsewhere.

[i3v]

Just in case anyone else would run into this, here's a somewhat simplified case:

With poetry version 1.2.2, python Python 3.10.7 and poethepoet version 0.16.5 on Windows 10 and the following lines in pyproject.toml:

run-demo1 = {shell = "echo 111"}
demo1 = ["run-demo1"]

run-demo2 = "echo 222"
demo2 = ["run-demo2"]

, I found that the poetry run poe demo1 works OK, but the poetry run poe demo2 fails:

PS C:\Users\username\WorkingCopies\some_project> poetry run poe demo1  
Poe => echo 111
111
PS C:\Users\username\WorkingCopies\some_project> poetry run poe demo2
Poe => echo 222
Traceback (most recent call last):
  File "C:\Users\username\.pyenv\pyenv-win\versions\3.10.7\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\username\.pyenv\pyenv-win\versions\3.10.7\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\username\AppData\Local\pypoetry\Cache\virtualenvs\some-project-RiwcqItH-py3.10\lib\site-packages\poethepoet\__init__.py", line 33, in main
    result = app(cli_args=sys.argv[1:])
    return self.run_task() or 0
  File "C:\Users\username\AppData\Local\pypoetry\Cache\virtualenvs\some-project-RiwcqItH-py3.10\lib\site-packages\poethepoet\app.py", line 98, in run_task
    return self.task.run(context=context, extra_args=self.ui["task"][1:])
  File "C:\Users\username\AppData\Local\pypoetry\Cache\virtualenvs\some-project-RiwcqItH-py3.10\lib\site-packages\poethepoet\task\base.py", line 239, in run
    return self._handle_run(
  File "C:\Users\username\AppData\Local\pypoetry\Cache\virtualenvs\some-project-RiwcqItH-py3.10\lib\site-packages\poethepoet\task\sequence.py", line 84, in _handle_run
    task_result = subtask.run(
  File "C:\Users\username\AppData\Local\pypoetry\Cache\virtualenvs\some-project-RiwcqItH-py3.10\lib\site-packages\poethepoet\task\base.py", line 239, in run
    return self._handle_run(
  File "C:\Users\username\AppData\Local\pypoetry\Cache\virtualenvs\some-project-RiwcqItH-py3.10\lib\site-packages\poethepoet\task\ref.py", line 33, in _handle_run
    return task.run(context=context, extra_args=extra_args, parent_env=env)
  File "C:\Users\username\AppData\Local\pypoetry\Cache\virtualenvs\some-project-RiwcqItH-py3.10\lib\site-packages\poethepoet\task\base.py", line 239, in run
    return self._handle_run(
  File "C:\Users\username\AppData\Local\pypoetry\Cache\virtualenvs\some-project-RiwcqItH-py3.10\lib\site-packages\poethepoet\task\cmd.py", line 44, in _handle_run
    return context.get_executor(self.invocation, env, self.options).execute(
  File "C:\Users\username\AppData\Local\pypoetry\Cache\virtualenvs\some-project-RiwcqItH-py3.10\lib\site-packages\poethepoet\executor\poetry.py", line 33, in execute
    return self._execute_cmd(
  File "C:\Users\username\AppData\Local\pypoetry\Cache\virtualenvs\some-project-RiwcqItH-py3.10\lib\site-packages\poethepoet\executor\base.py", line 156, in _execute_cmd
    return self._exec_via_subproc(cmd, input=input, env=env, shell=shell)
  File "C:\Users\username\AppData\Local\pypoetry\Cache\virtualenvs\some-project-RiwcqItH-py3.10\lib\site-packages\poethepoet\executor\base.py", line 208, in _exec_via_subproc
    proc = Popen(cmd, **popen_kwargs)
  File "C:\Users\username\.pyenv\pyenv-win\versions\3.10.7\lib\subprocess.py", line 969, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "C:\Users\username\.pyenv\pyenv-win\versions\3.10.7\lib\subprocess.py", line 1438, in _execute_child
    hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
FileNotFoundError: [WinError 2] The system cannot find the file specified

Effectively the same thing happens with plain Popen:

PS C:\Users\username\WorkingCopies\some_project> python -c "from subprocess import Popen; Popen('echo 111',bufsize=-1,shell=False)"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\username\.pyenv\pyenv-win\versions\3.10.7\lib\subprocess.py", line 969, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "C:\Users\username\.pyenv\pyenv-win\versions\3.10.7\lib\subprocess.py", line 1438, in _execute_child
    hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
FileNotFoundError: [WinError 2] The system cannot find the file specified

PS C:\Users\username\WorkingCopies\some_project> python -c "from subprocess import Popen; Popen('echo 111',bufsize=-1,shell=True)" 
111

It is possible to make both demo1 and demo2 work by changing this to popen_kwargs: MutableMapping[str, Any] = {"shell": True}, but I guess that the correct way is to just use explicit shell version, like in the poetry run poe demo1 case.

[nat-n]

Hi @i3v,

Not a windows/powershell expert here but this looks like expected behavior to me given that AFAIK echo is not a valid executable in windows. i.e. Popen(["echo", "hello"]) works on linux etc but not on windows.

So since run-demo2 is a cmd task (because that's the default task type normally), it runs the indicated executable as a subtask without a shell.

By contrast demo1 works because you specify that the task is a shell task, which by default means that it will go find a shell executable like git-bash or whatever on your system and use that to execute the command, and in the shell you have the echo command.

This might defy expectations if you're used to how npm works for instance, where it just uses a shell to execute the whatever command line you give it, though IMO that approach is overall less powerful and less predictable across environments.

[i3v]

Hi, @nat-n ,
Yep, AFAIU you're absolutely correct in your analysis. I was not trying to say that something is broken.

The only slightly questionable point is that (posix) shell based task in the readme might look like this option won't work on Windows, even though this could be a better option in some cases.

My only intention was to provide a readable example for someone who would search for an issue that contains [WinError 2] The system cannot find the file specified, just like me yesterday.