How do you set up your development environment?

[robocioaca]

So far, my development process while fixing an issue has been:

1. change the source file in the dev tree;
2. install the vaex library again in my venv (using the new source): pip install -e .
3. try to come up with suitable tests in ipython over there and go through them.

This is clearly artisanal and quite haphazard. (🙄 understatement ) 😄

How do you organize yourselves while doing development work in vaex? How do you write tests and batch run them (I see the tests folder, however I can't yet grok how to launch them in bulk...) Is there any CI infrastructure I might clone on my machine and run as I wish? How should I prepare for using the makefile (right now I get errors...)?

Please excuse the obvious greenhorn confusion. Any advice will be welcome, thanks!

[JovanVeljanoski]

Hi @robocioaca

This is a very good question indeed. There is some documentation about this here. It shows steps how to set up dev version of vaex, and what is our workflow for contributing. Perhaps it can be more verbose, so please take a look, and if something is not clear I can try and elaborate on it further.

Just FYI, I personally (and I think @maartenbreddels too) use (mini)conda as a python environment manager, and mamba for installation (since it is so fast). You are not limited to this, but I use miniconda almost exclusively for years now and am so happy with it that i've spent 0 time looking and trying other similar things.

Regarding running tests, most of the (moden) tests are in the tests folder, and for those we use the pytest library. So to run all tests you can do (from the root of the vaex ) pytest tests/. To run a specific test file you can just do pytest /tests/agg_test.py for example.

When you submit a PR, we have CI based on github actions that will run all the tests. The total suite of tests includes those in the tests folder, most of the documentation notebooks and some older tests using nose, but you probably should not care about those.

I hope this helps a bit. Feel free to ask more questions here if needed!

Thanks!

[robocioaca]

Thank you, @JovanVeljanoski ! I've installed pytest and got around to running batch testing by myself. Now I get how to use individual tests, as well. 😎

This is the first time I hear of mamba, I'll dig deeper and see how it works.

Have a great weekend,
Alex