macOS Big Sur 11.2.1 - installing virtual environment and dependencies compiler issues

[twerkmeister]

Rasa version: master branch

pyenv version: 1.2.22

Operating system (windows, osx, ...): macOS Big Sur 11.2.1

Issue: For me there were a couple of compiler problems in getting the virtual env and dependencies for development running.

1. pyenv install only worked for certain versions: pyenv install 3.7.6 as shown in the readme won't work.pyenv install 3.7.9 turned out to work in the end among a few others

> pyenv install 3.7.6
python-build: use openssl@1.1 from homebrew
python-build: use readline from homebrew
Downloading Python-3.7.6.tar.xz...
-> https://www.python.org/ftp/python/3.7.6/Python-3.7.6.tar.xz
Installing Python-3.7.6...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk

BUILD FAILED (OS X 11.2.1 using python-build 20180424)

Inspect or clean up the working tree at /var/folders/g8/frp_8d7s2639spgcxqyd3kpw0000gn/T/python-build.20210215142044.19884
Results logged to /var/folders/g8/frp_8d7s2639spgcxqyd3kpw0000gn/T/python-build.20210215142044.19884.log

Last 10 log lines:
/usr/local/include/pthread.h:197:34: note: expanded from macro '_PTHREAD_SWIFT_IMPORTER_NULLABILITY_COMPAT'
        defined(SWIFT_CLASS_EXTRA) && (!defined(SWIFT_SDK_OVERLAY_PTHREAD_EPOCH) || (SWIFT_SDK_OVERLAY_PTHREAD_EPOCH < 1))
                                        ^
197 warnings generated.
197 warnings generated.
198 warnings generated.
197 warnings generated.
197 warnings generated.
197 warnings generated.
197 warnings generated.

proposed improvement: Possibly update Readme to use 3.7.9 right away, or add a note for macOS users.

2. dependency grpcio will run into clang errors during installation when executing make install. I tried a bunch of stuff. In the end a simple environment variable helped:
   export SYSTEM_VERSION_COMPAT=1

proposed improvement: Add a note for macOS users.

I don't know if this was only related to my machine but it was quite some effort to get this to run. In between I tried all sorts of things from updating xcode, command line tools, brew, using specific brew libraries instead of the std macOS libraries when installing things... In the end the above two things apparently made the difference.

[sara-tagger]

Thanks for raising this issue, @yennycheung will get back to you about it soon✨

Please also check out the docs and the forum in case your issue was raised there too 🤗

[erohmensing]

Sorry about that @twerkmeister, I'll make tagbot ignore you, and welcome back! :)

[twerkmeister]

No worries @erohmensing! Good to be back :)

[kossy]

don't know if this was only related to my machine but it was quite some effort to get this to run. In between I tried all sorts of things from updating xcode, command line tools, brew, using specific brew libraries instead of the std macOS libraries when installing things... In the end the above two things apparently made the difference.

After sorting the current issues with pyenv 1.2.23 on Big Sur. We also have had issues too instaling grpcio too on Macos 11.2.2 export SYSTEM_VERSION_COMPAT=1 fixed the issue, hopefully that clears up it not being isolated to you machine.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[ErickGiffoni]

Hey, I am experiencing problems like that here too when trying to do pyenv install 3.7.6.

In my case I get the error :

[...]
./Modules/posixmodule.c:8436:15: error: implicit declaration of function 'sendfile' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
        ret = sendfile(in, out, offset, &sbytes, &sf, flags);
              ^
1 error generated.
make: *** [Modules/posixmodule.o] Error 1
make: *** Waiting for unfinished jobs....
1 warning generated.

I am using macOS Big Sur 11.5 , pyenv 1.2.21

Can someone help ?

[ErickGiffoni]

Hey, I am experiencing problems like that here too when trying to do pyenv install 3.7.6.

In my case I get the error :

[...]
./Modules/posixmodule.c:8436:15: error: implicit declaration of function 'sendfile' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
        ret = sendfile(in, out, offset, &sbytes, &sf, flags);
              ^
1 error generated.
make: *** [Modules/posixmodule.o] Error 1
make: *** Waiting for unfinished jobs....
1 warning generated.

I am using macOS Big Sur 11.5 , pyenv 1.2.21

Can someone help ?

Yes, I am replying to my own comment because I may have found a work around the problem.

First of all, I found a similar issue describing a similar problem here. Then I was able to fix the issue with:

pyenv install --patch 3.7.6 < <(curl -sSL https://github.com/python/cpython/commit/8ea6353.patch)

But the universe conspired against me and I got another error:

Last 10 log lines:
  File "/private/var/folders/c8/05hjylz57llf63n1bhhv9n6w0000gn/T/python-build.20210804100934.72978/Python-3.7.6/Lib/ensurepip/__main__.py", line 5, in <module>
    sys.exit(ensurepip._main())
  File "/private/var/folders/c8/05hjylz57llf63n1bhhv9n6w0000gn/T/python-build.20210804100934.72978/Python-3.7.6/Lib/ensurepip/__init__.py", line 204, in _main
    default_pip=args.default_pip,
  File "/private/var/folders/c8/05hjylz57llf63n1bhhv9n6w0000gn/T/python-build.20210804100934.72978/Python-3.7.6/Lib/ensurepip/__init__.py", line 117, in _bootstrap
    return _run_pip(args + [p[0] for p in _PROJECTS], additional_paths)
  File "/private/var/folders/c8/05hjylz57llf63n1bhhv9n6w0000gn/T/python-build.20210804100934.72978/Python-3.7.6/Lib/ensurepip/__init__.py", line 27, in _run_pip
    import pip._internal
**zipimport.ZipImportError: can't decompress data; zlib not available**
make: *** [install] Error 1

So, to solve that I had to do some things, like:

brew install zlib
export LDFLAGS="-L/usr/local/opt/zlib/lib"
export CPPFLAGS="-I/usr/local/opt/zlib/include"
export PKG_CONFIG_PATH="/usr/local/opt/zlib/lib/pkgconfig"
ln -s /usr/local/Cellar/zlib/1.2.11/include/* /usr/local/include/
ln -s /usr/local/Cellar/zlib/1.2.11/lib/* /usr/local/lib/
brew reinstall python

and then again

pyenv install --patch 3.7.6 < <(curl -sSL https://github.com/python/cpython/commit/8ea6353.patch)

After all those struggles... my final comments

For all macOS users,

1. You may want to guarantee creating your venv with the right python version
   • In my case, python3 -m venv .venv, as the README suggests
   • Then, activate the venv
2. Now if you run make install it should work well !!!

[ErickGiffoni]

Hey Rasa team, after I got some many problems and could solve them,
I thought I should document the steps for other macOS users.

It may solve this issue. I'll be opening a pull request soon. Thank you.

[twerkmeister]

Hey @ErickGiffoni. Thanks for your efforts - Did you also have those problems when using python 3.7.9 which I described would work? Is there a reason you had to stick to 3.7.6?

[ErickGiffoni]

Hey @ErickGiffoni. Thanks for your efforts - Did you also have those problems when using python 3.7.9 which I described would work? Is there a reason you had to stick to 3.7.6?

Hey @twerkmeister Thomas,

I stick to Python 3.7.6 only because it was the README suggestion. I was having so much trouble and finding so many problems that using 3.7.9 didn’t even come to my mind... but the good thing is that the efforts worked in the end.

If you want, we could add to the pull request I created using Python 3.7.9 as your suggestion.