Run tests in a parallel matrix

[bonjourmauko]

Hello!

(Working) prototype for solving #1025 and #1035, poke #1027, please chime in! 😄

⚠ This is not a proposal but a demonstration of the different tools, so we can discuss and choose what make more sense in terms both of performance and readability to the community.

Trying it out

$ make test.matrix
...

real	2m13.152s
user	10m12.086s
sys	1m48.311s

New features

• Run openfisca test tests in parallel with make, nox, and invoke
• Run tests against Python 3.7+
• Run tests against NumPy 1.18+

Notes

• We don't have control over the way subprocesses are launched and run, so reporting could be a bit confusing
• Performance gains are relative to the number of cores in your machine
• OpenFisca is not threadsafe, so each subprocess will run a single thread
• If one day we solve concurrency problems, ... that could go faster.

TODO

• Run each test suite with invoke
• Run the matrix of tests with nox
• Parallelize with make
• Parallelize with pytest-parallel
• Parallelize with pytest-split
• Run .py tests in parallel
• Run .yaml tests in parallel
• Provide decent reporting
• What to do in CI Switch from CircleCI to GitHub Actions #1030
• Benchmark

[MattiSG]

Thanks @maukoquiroga! The promise of such a speed improvement is exciting, and I would be very appreciative of a refactor / overhaul of the current test runner 😃

However, I must say that the changeset is pretty daunting, and seems to have impacts way beyond the test runner, with some core functions having their visibility changed… Could we maybe convene of a video discussion with other interested @openfisca/international-contrib contributors, in order to understand the goals, side effects and side benefits? 🙂

In terms of pure performance and parallelisation, it might also be that #1030 allows us to use https://github.com/nektos/act as suggested by @benoit-cty, and we could have local parallelisation with the exact same setup as CI, which I would be very excited about 😃

[bonjourmauko]

However, I must say that the changeset is pretty daunting, and seems to have impacts way beyond the test runner, with some core functions having their visibility changed…

Some of them were absolutely required, in which case I've tried to avoid a breaking change by exposing changes through __init__.py files, some maybe just because I just was rushing to make it work (hence the draft status).

Indeed a partial refactoring of the test runner was imposed in order to make this work, which could definitely go in a PR on its own, with proper deprecations, and agreed upon breaking changes.

In terms of pure performance and parallelisation, it might also be that #1030 allows us to use https://github.com/nektos/act as suggested by @benoit-cty, and we could have local parallelisation with the exact same setup as CI, which I would be very excited about 😃

❤️

Could we maybe convene of a video discussion with other interested @openfisca/international-contrib contributors, in order to understand the goals, side effects and side benefits? 🙂

Definitely.

[bonjourmauko]

Closing for now.