An intermittent venv corruption problem with running transforms

[shahrokhDaijavad]

Search before asking

• I searched the issues and found no similar issues.

Component

Transforms/Other

What happened + What you expected to happen

The problem is an intermittent corruption of venv for an unknown reason while running any test transform. By intermittent, of course, it means that it is not reproducible all the time, but when it happens (and it does have a temporary work-around solution), it is very frustrating. Here is what happens:

After a clean clone of the repo, you go to a transform like noop and do a make venv. Then, you go to a subdirectory like the python subdirectory and try to test using make run-local-sample. You will see errors like this:

make run-local-sample
/Applications/Xcode.app/Contents/Developer/usr/bin/make RUN_FILE=noop_local.py .transforms.run-src-file
source venv/bin/activate;
cd src;
python noop_local.py
Traceback (most recent call last):
File “/Users/shahrokhdaijavad/Documents/GitHub/data-prep-kit-testing/data-prep-kit/transforms/universal/noop/python/src/noop_local.py”, line 15, in
from data_processing.data_access import DataAccessLocal
ModuleNotFoundError: No module named ‘data_processing’
make[1]: *** [.defaults.run-src-file] Error 1
make: *** [.transforms.run-local-sample] Error 2

Now, if you clean the corrupted venv using: make clean venv run-local-sample and then run the make run-local-sample again, it works without any error!

Reproduction script

Please see above.

Anything else

This happens intermittently.

OS

MacOS (limited support)

Python

3.11.x

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[shahrokhDaijavad]

@daw3rd I verified your observation that make help is the culprit in corrupting the venv. So, this is now a reproducible bug in the following way:
A) make env B) make help and C) make run-local-sample => You will get the above error
A) make env and B) make run-local-sample => You do not get an error and it runs fine
A) make env B) make help C) make clean venv run-local-sample D) make run-local-sample => You do not get an error and it runs fine.

[daw3rd]

I've narrowed this a bit further.

cd transforms/noop/python
make venv
du -s venv # shows a certain size...
make run-local-sample  # ... and this works
make -n test    # Should have no effect on local file system
du -s venv # shows a different size!...
make run-local-sample # ...and so this fails.

It seems to be specific to the test target as none of the other targets (lexigraphically before test) did not cause the problem.

[daw3rd]

This seems to be the culprit, which is fixed if I add the echo to the pip uninstall below

.PHONY: .defaults.install-python-lib-src-venv
.defaults.install-python-lib-src-venv::
        @# Help: Install Python data processing library source into existing venv
        @echo Installing Python data processing library source to existing venv
        @source venv/bin/activate;                                      \
        pip install pytest;                                             \       
        echo pip uninstall -y data-prep-toolkit;                                \
        $(MAKE) PYTHON_PROJECT_DIR=$(DPK_PYTHON_LIB_DIR) .defaults.install-src-venv;    \
        echo Installed source from Python processing library for `which $(PYTHON)`

This is very odd though since make -n .defaults.install-python-lib-src-venv says it will Print the commands that would be executed, but do not execute them. yet it seems to be executing the pip uninstall

Calling all make gurus: @revit13 @deanwampler

To reproduce (in transforms/noop/python):

rm -rf venv
make venv
du -s venv # shows a certain size...
make -n .defaults.install-python-lib-src-venv    # Should have no effect on local file system
du -s venv # shows a different size!...

[daw3rd]

Fixed in PR #302 which as been merged.