Apple M1 installation and usage

[panosl]

Executing pip install SoundFile on an M1, installs just fine using generic wheel:

  Using cached SoundFile-0.10.3.post1-py2.py3-none-any.whl (21 kB)

But trying to import soundfile shows the libsndfile as missing:

Python 3.9.4 (default, Apr 22 2021, 12:00:49)
[Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import soundfile
Traceback (most recent call last):
  File "...venv/lib/python3.9/site-packages/soundfile.py", line 142, in <module>
    raise OSError('sndfile library not found')
OSError: sndfile library not found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "...venv/lib/python3.9/site-packages/soundfile.py", line 162, in <module>
    _snd = _ffi.dlopen(_os.path.join(
OSError: cannot load library '...venv/lib/python3.9/site-packages/_soundfile_data/libsndfile.dylib': dlopen(...venv/lib/python3.9/site-packages/_soundfile_data/libsndfile.dylib, 2): image not found
>>>

Installing the libsndfile through brew, is not picked up either.

[panosl]

Turns out the fix is, copy the dylib installed from Homebrew into the expected directory and it picks it up.

So, my assumption is that this just a packaging issue. I could open a PR if you can tell me how you usually solve those issues.

[bastibe]

I guess we'll need a new dylib for M1. I am still waiting for the next release of libsndfile, which will include MP3 support. Once that is available, I'll rebuild libsndfile for all platforms, and will try to include an M1 version if that's possible from an Intel Mac (I don't have M1 hardware).

But really, the source version of soundfile should just pick up any Homebrew-installed libsndfile. Do you know why it didn't do that in your case? Probably either a library search path issue, or a file naming issue that we could easily fix as an interim solution.

[panosl]

The problem is that ctypes.util.find_library searches predefined paths defined within dyld.py.

Latest releases of MacOS no longer seem to honor DYLD_LIBRARY_PATH et all.

Homebrew which is the de facto installer on MacOS uses /opt/homebrew/lib for the M1 architecture instead of the /usr/local/lib which was used in Intel Macs.

So it's sort of an impasse. We either need to:

• handle it at the python-soundfile level, with code specific for M1
• Suggest a fix at cpython to allow loading from homebrew's lib directory
• Suggest a fix to homebrew to link to a different directory?

That is, until a new release is made, so we can just include libsndfile.dylib directly in the installer.

[bastibe]

Well, that is troubling.

Do you know how other libraries are handling this? I wouldn't mind having a workaround specific for M1 in python-soundfile, at least until this situation is resolved. Would you like to create a pull request?

[panosl]

I've opened PR #311. It seems to resolve things until the next build of libsndfile were we can have the actual library included in the distribution of python-soundfile.

[bastibe]

Thank you very much for your contribution! I'll try to release a bugfix release for this in the next few days.

[chuma9615]

Any news on this release? I'd really like to use this package on my M1 machine 😬

[bastibe]

I hope I'll find time to do the release this week. I'd like to include the related changes in #322 as well.

[bastibe]

I have started working on the release. Man, it's been a while since I've used a Mac, and I've forgotten just how annoying the command line experience is on there. Right now it's stuck installing homebrew (did this always take so long?), because the included curl is woefully out-of-date and can't even download files any more...

At any rate, I would like to ship binary wheels that include M1 support, but the only Mac I have access to is an older Intel machine. If anyone could provide me with a recently-compiled version of libsndfile in https://github.com/bastibe/libsndfile-binaries/, I'd love to include them in the wheel! There's a build-script for macOS included in the repository that should work out of the box (just update the version numbers at the top to the most recent ones). Hopefully.

[chuma9615]

I can do that, let me know how I can ship you the binaries.

[bastibe]

Brilliant, thank you! Open a pull request on https://github.com/bastibe/libsndfile-binaries with the new library, and I'll happily merge it!

I don't know about *.dylib conventions. It might be possible to have a universal libsndfile.dylib that works on Intel and ARM, or it might be necessary to ship two separate dylibs (like for Windows). As of five minutes ago, the build script for macOS should be up-to-date, and worked fine on Intel Big Sur machine.

[bastibe]

I'm afraid that's about all I can do this week. My FOSS-time is now pretty much used up. I'll try to continue next week, and hopefully build a beta version at least. Once that is confirmed working, I'll push it to PyPi.

python-soundfile has become too big to rush it onto PyPi without testing on all systems. Thank you everyone for your patience, and your contributions!

[chuma9615]

I just opened a PR here, hope it helps!

[bastibe]

Thank you so very much @chuma9615! I'll have a look later this week, when I continue work on this.

[bastibe]

I'll continue the discussion on creating a new release in #325. Any help would be greatly appreciated! It's been a while since we last built a release for soundfile.

[stefan-balke]

Quick workaround for everybody.

Add:
export DYLD_LIBRARY_PATH="/opt/homebrew/lib:$DYLD_LIBRARY_PATH"

to your ~/.zshrc.

[bastibe]

Just so you know, this is dangerous advice. It forces all shell-executed software to link preferentially against homebrew libraries. If you must use DYLD_LIBRARY_PATH, do it in a per-execution basis, e.g. env DYLD_LIBRARY_PATH="/opt/homebrew/lib:$DYLD_LIBRARY_PATH" python myscript.py.

[stefan-balke]

True, but I suspect it for 90% of the users as safe :-)

[bastibe]

I emphatically do not! In fact I have run into problems with DYLD_LIBRARY_PATH and its various permutations quite often. C libraries are eternally fickle when it comes to linking. Some even depend on a specific version of glibc, or a GUI library, or a libpython. They might specify their requirements quite clearly, but DYLD_LIBRARY_PATH intentionally overrides those. It's dangerous stuff.

[stefan-balke]

Maybe true for Linux but on Mac 90% of the libs go through hombrew anyways.
Actually 99%...but as I said, I agree and people should have this in mind.

[chazarnik]

Hello, is there any update on this issue? I still get the '_soundfile_data/libsndfile.dylib' ' no such file error, as mentioned initially by @panosl .

Surprisingly, creating a directory named '_soundfile_data' into the miniforge3 'site_packages' dir doesn't work either.

[task4233]

It worked on my M1 Mac with this comment.
ohmtech-rdi/eurorack-blocks#444 (comment)

thx.

[bastibe]

The upcoming release should fix this issue, by bundling an appropriate version of libsndfile with the soundfile wheel.

[mattiasu96]

The upcoming release should fix this issue, by bundling an appropriate version of libsndfile with the soundfile wheel.

When this is coming?

[bastibe]

v0.11.0 has been released a few days ago

[asapsmc]

I installed soundfile v0.11.0 and I still ran into the same problem (in a conda env running on Mac M1)

[bastibe]

Then conda is probably delivering the wrong package. pip install soundfile should download and install the correct version (the binary wheel). You also might want to uninstall any soundfile- or libsndfile- related package from conda.

[asapsmc]

I did pip install soundfile and it installed v0.11.0. I did not uninstall the related packages.

[bastibe]

You probably have another, system-installed libsndfile. Soundfile always tries your system-installed libraries before it loads its own.

[AlienKevin]

I'm having trouble using soundfile in miniconda3 on m1 mac.

soundfile seems to have trouble finding the dylib file:

OSError: cannot load library '/Users/kevin/miniconda3/envs/py39/lib/python3.9/site-packages/_soundfile_data/libsndfile.dylib': dlopen(/Users/kevin/miniconda3/envs/py39/lib/python3.9/site-packages/_soundfile_data/libsndfile.dylib, 0x0002): tried: '/Users/kevin/miniconda3/envs/py39/lib/python3.9/site-packages/_soundfile_data/libsndfile.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/kevin/miniconda3/envs/py39/lib/python3.9/site-packages/_soundfile_data/libsndfile.dylib' (no such file), '/Users/kevin/miniconda3/envs/py39/lib/python3.9/site-packages/_soundfile_data/libsndfile.dylib' (no such file).  Additionally, ctypes.util.find_library() did not manage to locate a library called '/Users/kevin/miniconda3/envs/py39/lib/python3.9/site-packages/_soundfile_data/libsndfile.dylib'

The reason is probably that the dylib file is named libsndfile_arm64.dylib, rather than libsndfile.dylib......

Workaround

The current workaround is to make a copy of libsndfile_arm64.dylib under /Users/USER_NAME/miniconda3/envs/py39/lib/python3.9/site-packages/_soundfile_data/ and name the copy libsndfile.dylib. Alternatively, I think a symlink will also work.

[bastibe]

Don't use conda. They break things frequently for many users, and there's nothing we developers can do about it. Conda used to be useful back when wheels weren't yet available everywhere. But no longer.

Soundfile is published on PyPi as a wheel. Install that, and you'll be good.

[AlienKevin]

Ok, I see. Python package management/dependency resolution seems like quite a mess to me. I'm only using conda because tensorflow needs to be installed through conda on M1 macs to have GPU support. Otherwise, I would be happy to stick with PyPi.

[bastibe]

You can mix and match. Install only what you must with conda, and get what you can from PyPi.