parallel output does not respect --skip-missing-interpreters

[rpkilby]

Standard tox usage reports when a testenv was skipped due to a missing interpreter. This is included in both the live output, as well as the report summary at the end.

tox output

(.venv) Sherpa:tox-factor bagel$ tox --skip-missing-interpreters
GLOB sdist-make: /Users/bagel/Documents/projects/tox-factor/setup.py
py27 develop-inst-noop: /Users/bagel/Documents/projects/tox-factor
py27 installed: DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.,coverage==4.5.3,filelock==3.0.10,funcsigs==1.0.2,mock==2.0.0,pbr==5.1.3,pluggy==0.9.0,py==1.8.0,six==1.12.0,toml==0.10.0,tox==3.9.0,-e git+git@github.com:rpkilby/tox-factor.git@a3a7e5d512d0548e0499ee393b03b01649d8de0a#egg=tox_factor,virtualenv==16.4.3
py27 run-test-pre: PYTHONHASHSEED='3399770719'
py27 run-test: commands[0] | coverage run -m unittest discover
.......................................
----------------------------------------------------------------------
Ran 39 tests in 2.202s

OK
py34 create: /Users/bagel/Documents/projects/tox-factor/.tox/py34
SKIPPED: InterpreterNotFound: python3.4
py35 create: /Users/bagel/Documents/projects/tox-factor/.tox/py35
SKIPPED: InterpreterNotFound: python3.5
py36 create: /Users/bagel/Documents/projects/tox-factor/.tox/py36
SKIPPED: InterpreterNotFound: python3.6
py37 develop-inst-noop: /Users/bagel/Documents/projects/tox-factor
py37 installed: coverage==4.5.3,filelock==3.0.10,mock==2.0.0,pbr==5.1.3,pluggy==0.9.0,py==1.8.0,six==1.12.0,toml==0.10.0,tox==3.9.0,-e git+git@github.com:rpkilby/tox-factor.git@a3a7e5d512d0548e0499ee393b03b01649d8de0a#egg=tox_factor,virtualenv==16.4.3
py37 run-test-pre: PYTHONHASHSEED='3399770719'
py37 run-test: commands[0] | coverage run -m unittest discover
.......................................
----------------------------------------------------------------------
Ran 39 tests in 1.954s

OK
pypy create: /Users/bagel/Documents/projects/tox-factor/.tox/pypy
SKIPPED: InterpreterNotFound: pypy
isort develop-inst-noop: /Users/bagel/Documents/projects/tox-factor
isort installed: filelock==3.0.10,isort==4.3.17,pluggy==0.9.0,py==1.8.0,six==1.12.0,toml==0.10.0,tox==3.9.0,-e git+git@github.com:rpkilby/tox-factor.git@a3a7e5d512d0548e0499ee393b03b01649d8de0a#egg=tox_factor,virtualenv==16.4.3
isort run-test-pre: PYTHONHASHSEED='3399770719'
isort run-test: commands[0] | isort --check-only --recursive src
lint develop-inst-noop: /Users/bagel/Documents/projects/tox-factor
lint installed: attrs==19.1.0,darglint==0.5.4,entrypoints==0.3,filelock==3.0.10,flake8==3.7.7,flake8-bugbear==19.3.0,flake8-commas==2.0.0,flake8-quotes==2.0.0,mccabe==0.6.1,pluggy==0.9.0,py==1.8.0,pycodestyle==2.5.0,pyflakes==2.1.1,six==1.12.0,toml==0.10.0,tox==3.9.0,-e git+git@github.com:rpkilby/tox-factor.git@a3a7e5d512d0548e0499ee393b03b01649d8de0a#egg=tox_factor,virtualenv==16.4.3
lint run-test-pre: PYTHONHASHSEED='3399770719'
lint run-test: commands[0] | flake8 src
dist create: /Users/bagel/Documents/projects/tox-factor/.tox/dist
dist installdeps: coverage, mock
dist inst: /Users/bagel/Documents/projects/tox-factor/.tox/.tmp/package/1/tox-factor-0.0.2.zip
dist installed: coverage==4.5.3,filelock==3.0.10,mock==2.0.0,pbr==5.1.3,pluggy==0.9.0,py==1.8.0,six==1.12.0,toml==0.10.0,tox==3.9.0,tox-factor==0.0.2,virtualenv==16.4.3
dist run-test-pre: PYTHONHASHSEED='3399770719'
dist run-test: commands[0] | python -m unittest discover
.......................................
----------------------------------------------------------------------
Ran 39 tests in 2.038s

OK
_____________________________________________________ summary ______________________________________________________
  py27: commands succeeded
SKIPPED:  py34: InterpreterNotFound: python3.4
SKIPPED:  py35: InterpreterNotFound: python3.5
SKIPPED:  py36: InterpreterNotFound: python3.6
  py37: commands succeeded
SKIPPED:  pypy: InterpreterNotFound: pypy
  isort: commands succeeded
  lint: commands succeeded
  dist: commands succeeded
  congratulations :)

In comparison, tox in parallel mode simply shows success. It would be great if parallel mode also reported when a build was skipped due to a missing interpreter.

[Rotonen]

The same is true for platform mismatches.

I have a tox.ini like:

[tox]
envlist = {py36,py37}-{darwin,linux,win32}


[testenv]
platform =
    darwin: darwin
    linux: linux
    win32: win32

deps =
    -rrequirements-noarch.txt
    darwin: -rrequirements-darwin.txt
    win32: -rrequirements-win32.txt
    -rrequirements-tests.txt

commands =
    python setup.py test

The relevant part of a tox -q -p auto run on Linux:

✔ OK py36-darwin in 0.295 second
✔ OK py36-win32 in 0.287 second
✔ OK py37-darwin in 0.287 second
✔ OK py37-linux in 12.25 seconds
✔ OK py36-linux in 13.175 seconds
✔ OK py37-win32 in 0.27 second

Errors are also handled as expected:
✖ FAIL py36-darwin in 3.982 seconds

I'd expect something like - SKIPPED <reason> py37-win32 in 0.27 second as the status of a skipped platform.

Running just plain tox on darwin gives me helpful information lines like:
SKIPPED: py36-linux: platform mismatch ('darwin' does not match 'linux')

[stinovlas]

This is a deal breaker for me on whether to use tox parallel mode. It would be great to utilize its performance, but reporting success on skipped environments is really not OK.

I haven't digged in a tox code yet, but if someone can point me to where I should look, I might put together a pull request to resolve this.

[stinovlas]

I just bumped into #1394, and it seems that this issue might be resolved during the rewrite leading to tox 4. It's probably good idea to wait for the tox 4 changes first.

[gaborbernat]

Did you try with --skip-missing-interpreters false flag passed in?

[stinovlas]

Did you try with --skip-missing-interpreters false flag passed in?

Well yes, but that results in ERROR result and my pipeline fails. I'd like for parallel mode to behave in the same way as sequential and even parallel-live behaves. That is, it should list skipped environments as SKIPPED, as long as I have skip_missing_interpreters set to True.

[rpkilby]

This would just be a partial workaround, but you could at least parallelize testenv creation, while still
running the test suite in sequence (and thus getting the desired output).

$ tox -p auto --notest --skip-missing-interpreters
$ tox --skip-missing-interpreters

I did a quick test run with Django REST Framework on Python 3.7 - here are the results.

Sequential:

$ time tox --skip-missing-interpreters
...

real	3m42.750s
user	3m13.440s
sys	0m32.634s

Parallelized venv creation/sequential test run:

time sh -c 'tox -p auto --notest --skip-missing-interpreters && tox --skip-missing-interpreters'
...

real	2m37.989s
user	4m15.337s
sys	1m0.262s

[Rotonen]

In case people have a desire to go for uglier workarounds, xargs does have a parallel flag.

[stinovlas]

@rpkilby That's actually pretty good idea, at least for projects where testenv creation makes significant part of the test time. Thank you, I'll definitely try it out!

[gaborbernat]

This should have been fixed now with tox 4.