Tox fails silently when parsing logs containing non-ascii characters

[ni-fgenois]

Dear Tox maintainers,

I opened this ticket a while back and it seems it was closed, but it is still not fixed : #1506

I switched package managers since then and we are now using poetry instead of pipenv, but they now both face essentially the same problem.

Please note that this only occurs on Windows, and only if the --parallel flag is set. I'm

Essentially, the commands that are executed by the tox environment generate a log file with non-UTF8 characters (ANSI, in this case containing the •/\u2022/BULLET/0x95 character), and then, the parallel log parser crashes during the parsing of the logs.

Whenever such a non-UTF8 log is parsed, it discards the real result of the tox environment execution and returns a success. This means that the tox environment succeeds even if it shouldn't (e.g. failing tests). This causes our Windows CI builds to fail silently.

I have already tried setting the LANG and LC_ALL locale everywhere I could (globally, in [tox], in[testenv]), but no luck.

My tox environment looks like :

[testenv]
whitelist_externals = poetry
passenv = POETRY_*
commands =
    poetry install -v -n --no-root --no-ansi
    coverage run --parallel-mode ...

The output of poetry (in PROJECT/.tox/py38/log/py38-*.log) looks something like :

action: py38, msg: parallel py38
...
py38 installed: appdirs @ ...
py38 run-test-pre: PYTHONHASHSEED='272'
py38 run-test: commands[0] | poetry install -v -n --no-root --no-ansi
Using virtualenv: C:\code\nib\.tox\py38
Installing dependencies from lock file

Finding the necessary packages for the current system

Package operations: 0 installs, 0 updates, 0 removals, 76 skipped

  • Removing importlib-metadata (4.6.4): Skipped for the following reason: Not currently installed
  • Installing mccabe (0.6.1): Skipped for the following reason: Already installed
  • Installing pycodestyle (2.7.0): Skipped for the following reason: Already installed
  • Installing pyflakes (2.3.1): Skipped for the following reason: Already installed
  • Installing colorama (0.4.4): Skipped for the following reason: Already installed
  • Installing flake8 (3.9.2): Skipped for the following reason: Already installed
  • Installing pyparsing (2.4.7): Skipped for the following reason: Already installed
  • Installing smmap (4.0.0): Skipped for the following reason: Already installed
  • Installing astpretty (2.1.0): Skipped for the following reason: Already installed
  • Installing atomicwrites (1.4.0): Skipped for the following reason: Already installed
  • Installing certifi (2021.5.30): Skipped for the following reason: Already installed
...

Here is an example of the stacktrace :

C:\Python39\python.exe -m tox -p auto -e py38
✔ OK py38 in 27.88 seconds
Exception in thread Thread-11:
Traceback (most recent call last):
  File "C:\Python39\lib\threading.py", line 950, in _bootstrap_inner
    self.run()
  File "C:\Python39\lib\threading.py", line 888, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Python39\lib\site-packages\tox\session\commands\run\parallel.py", line 57, in run_in_thread
    output = action.popen(
  File "C:\Python39\lib\site-packages\tox\action.py", line 128, in popen
    lines = out_path.read_text("UTF-8").split("\n")
  File "C:\Python39\lib\site-packages\py\_path\common.py", line 171, in read_text
    return f.read()
  File "C:\Python39\lib\codecs.py", line 322, in decode
___________________________________ summary ___________________________________
  py38: commands succeeded
  congratulations :)
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x95 in position 14133: invalid start byte

Process finished with exit code 0

Please help :)

[ni-fgenois]

A colleague pointed me towards this thread : #1550

And it seem that the proposed workaround works;

[testenv]
...
setenv =
    PYTHONIOENCODING=utf-8

This extra configuration should not be required, but at least we have a workaround

[gaborbernat]

This extra configuration should not be required, but at least we have a workaround

tox does not try to alter your environment. You'd have this issue even if you'd run that command outside of tox. That being said perhaps this has been fixed by our rewrite tox v 4. Please try that and see if it helps.