Add Mac M1 Compatibility

[pierreia]

Hi,

Like the most part of Python librairies, SentencePiece won't install on Mac M1 architecture... "A revolution in data science" they said... what a joke, every data science library is a real pain to install!
Do you plan to make a compatible version of SentencePiece?

Thank you!

[taku910]

What kind of error did you see when installing sentencepiece on Mac M1? At least you might want to elaborate the steps you tried, otherwise, it is hard to provide the workaround at this moment.

[pierreia]

I use PyCharm, with a virtual environment where Tensorflow Mac OS and Torch are installed. Since the architecture of M1 is "arm64", there is a problem while building SentencePiece wheel. Here is my the end of my error logs, when I type pip install sentencepiece :
env: pkg-config: No such file or directory Failed to find sentencepiece pkg-config ---------------------------------------- ERROR: Command errored out with exit status 1: /Users/user/tensorflow_macos/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/tmp/pip-install-1svp5b95/sentencepiece_507a7b7ee8884aad99ee6348ea812b50/setup.py'"'"'; __file__='"'"'/private/tmp/pip-install-1svp5b95/sentencepiece_507a7b7ee8884aad99ee6348ea812b50/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/tmp/pip-record-o2bcirdh/install-record.txt --single-version-externally-managed --compile --install-headers /Users/user/tensorflow_macos/include/site/python3.8/sentencepiece Check the logs for full command output.

I got the same problem with Pytorch, which I was able to install since this answer: pytorch/pytorch#48145

[taku910]

Seems that you are trying to build/install old package. What happens if you download the latest source package from here? Then try the following command to build the wheel locally.

% pip -v install sentencepiece-0.1.94.tar.gz

[pierreia]

Thank you for your very fast answer. I manually build the .whl package, and I have been able to install sentence piece. However, I got an error while trying to import sentencepiece:

`

import sentencepiece
Traceback (most recent call last):
File "", line 1, in
File "/Users/user/tensorflow_macos/lib/python3.8/site-packages/sentencepiece/init.py", line 13, in
from . import _sentencepiece
ImportError: dlopen(/Users/user/tensorflow_macos/lib/python3.8/site-packages/sentencepiece/_sentencepiece.cpython-38-darwin.so, 2): Symbol not found: __ZN13sentencepiece4util6StatusD1Ev
Referenced from: /Users/user/tensorflow_macos/lib/python3.8/site-packages/sentencepiece/_sentencepiece.cpython-38-darwin.so
Expected in: flat namespace
in /Users/user/tensorflow_macos/lib/python3.8/site-packages/sentencepiece/_sentencepiece.cpython-38-darwin.so
`

[pierreia]

issue solved: I resintalled sentence piece through this:
pip uninstall sentencepiece

pip install --no-cache-dir sentencepiece

I don't know how, but it seems that a broken version of sentencepiece was cached. It now smoothly works on my Mac m1!

[millnerryan]

I'm still facing issues, even with @pierreia 's suggestion.

Building wheel for sentencepiece (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /Users/rmillner/miniforge3/envs/sentiment/bin/python3.9 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/zq/x2xccyn16ddbl6dzkr4lkzjm0000gn/T/pip-install-lb831ekl/sentencepiece_d30662501d75442aab80fcfc3f65ceeb/setup.py'"'"'; __file__='"'"'/private/var/folders/zq/x2xccyn16ddbl6dzkr4lkzjm0000gn/T/pip-install-lb831ekl/sentencepiece_d30662501d75442aab80fcfc3f65ceeb/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /private/var/folders/zq/x2xccyn16ddbl6dzkr4lkzjm0000gn/T/pip-wheel-2y4k66lp
       cwd: /private/var/folders/zq/x2xccyn16ddbl6dzkr4lkzjm0000gn/T/pip-install-lb831ekl/sentencepiece_d30662501d75442aab80fcfc3f65ceeb/
  Complete output (43 lines):
  /Users/rmillner/miniforge3/envs/sentiment/lib/python3.9/site-packages/setuptools/dist.py:717: UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead
    warnings.warn(
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.macosx-11.0-arm64-3.9
  creating build/lib.macosx-11.0-arm64-3.9/sentencepiece
  copying src/sentencepiece/__init__.py -> build/lib.macosx-11.0-arm64-3.9/sentencepiece
  copying src/sentencepiece/sentencepiece_model_pb2.py -> build/lib.macosx-11.0-arm64-3.9/sentencepiece
  copying src/sentencepiece/sentencepiece_pb2.py -> build/lib.macosx-11.0-arm64-3.9/sentencepiece
  running build_ext
  Package sentencepiece was not found in the pkg-config search path.
  Perhaps you should add the directory containing `sentencepiece.pc'
  to the PKG_CONFIG_PATH environment variable
  No package 'sentencepiece' found
  Cloning into 'sentencepiece'...
  Note: switching to 'd8711f55d9b2cb9c77a00adcc18108482b29b675'.
  
  You are in 'detached HEAD' state. You can look around, make experimental
  changes and commit them, and you can discard any commits you make in this
  state without impacting any branches by switching back to a branch.
  
  If you want to create a new branch to retain commits you create, you may
  do so (now or later) by using -c with the switch command. Example:
  
    git switch -c <new-branch-name>
  
  Or undo this operation with:
  
    git switch -
  
  Turn off this advice by setting config variable advice.detachedHead to false
  
  ./build_bundled.sh: line 15: cmake: command not found
  ./build_bundled.sh: line 16: nproc: command not found
  make: *** No targets specified and no makefile found.  Stop.
  make: *** No rule to make target `install'.  Stop.
  Package sentencepiece was not found in the pkg-config search path.
  Perhaps you should add the directory containing `sentencepiece.pc'
  to the PKG_CONFIG_PATH environment variable
  No package 'sentencepiece' found
  Failed to find sentencepiece pkg-config
  ----------------------------------------
  ERROR: Failed building wheel for sentencepiece
  Running setup.py clean for sentencepiece
Failed to build sentencepiece

[mdvandergon]

@millnerryan I ran into this issue today and got the same error you posted. I have a M1 Macbook, too. I was able to fix it, first by installing cmake:

brew install cmake

And then by installing from the sentencepiece-*.tar.gz source package as recommended by @taku910. As of today, after downloading the file the command was:

pip -v install  sentencepiece-0.1.96.tar.gz

After these two steps I was able to import sentencepiece. I hope that also works for you. Take care.

[simhallq]

MacBook Pro M1 -21 using conda

$ arch -arm64 brew install cmake

$ pip install --no-cache-dir sentencepiece

did the trick for me

[atoultaro]

@simhallq Thanks! Your answer solved the problem for me.

[stevhliu]

Hi! I am also encountering some issues with installing sentencepiece on my M1 in a virtual conda environment. I've tried both solutions from @mdvandergon and @simhallq and unfortunately neither did the trick. The error I get at the end is:

[ninjalu]

Hello. I have the same problem as @stevhliu above. Can't install with any of the solutions listed here. Really quite frustrating :I

[ninjalu]

@stevhliu

Finally resolved
brew install pkgconfig
worked for me, after a long session of trials and errors. Hopefully this will help others with the same problem

[loyio]

MacBook Pro M1 -21 using conda

$ arch -arm64 brew install cmake

$ pip install --no-cache-dir sentencepiece

did the trick for me

Thanks, It's work

[mrtipale]

I think, both
$ arch -arm64 brew install cmake and brew install cmake does same thing and worked for me to fix the issue. Thanks for details!

[entzyeung]

MacBook Pro M1 -21 using conda

$ arch -arm64 brew install cmake

$ pip install --no-cache-dir sentencepiece

did the trick for me

Thanks, It's work

this works for me, M1 Mac

[srulik-ben-david-hs]

what works for me a lot of the time is install some previous version of problematic packages, and thats it.
so for sentencepiece:
pip3 install sentencepiece==0.1.95
(instead of 0.1.96)

[ashesfall]

Installing cmake beforehand worked for me. Looking at the README, this is already part of the instructions (that I didn't read)

[g-vallejo]

I tried @mdvandergon 's solution: 1) installing cmake and 2) downloading and installing the latest file from sources:

But still got problems. I tried then @ninjalu 's solution of installing also pkgconfig and that worked! 🥳.
Here my steps:

$ brew install cmake
$ wget https://files.pythonhosted.org/packages/aa/71/bb7d64dcd80a6506146397bca7310d5a8684f0f9ef035f03affb657f1aec/sentencepiece-0.1.96.tar.gz
$ brew install pkgconfig
$ pip -v install  sentencepiece-0.1.96.tar.gz

Thanks everyone involved in making live with M1 a bit easier!

[linhaosheng]

brew install sentencepiece
brew install pkg-config
pip install sentencepiece

it work for me

[shaun-scale]

I was going crazy, thank you @linhaosheng !

brew install sentencepiece

seems to have done it...

[chandrakalagowda]

what works for me a lot of the time is install some previous version of problematic packages, and thats it. so for sentencepiece: pip3 install sentencepiece==0.1.95 (instead of 0.1.96)

This worked for me. Using Mac 10.13.6

[kerbachi]

Installing cmake fixed the error on M2

brew install cmake