Python crashes in unpacked environment on osx-arm64

[jtilly]

Checklist

• I added a descriptive title
• I searched open reports and couldn't find a duplicate

What happened?

When I run conda pack on my M1, Python just crashes in the unpacked environment:

# conda-pack appears to be broken on an m1
cd /tmp/
conda create -y -p ./original python conda-pack
conda activate ./original
python -v  # this works!

conda-pack --format "tar.gz"

mkdir unpacked
tar -xUzf original.tar.gz -C unpacked

conda activate ./unpacked
conda-unpack
python -v # this crashes

Output from the last step:

import _frozen_importlib # frozen
import _imp # builtin
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import '_warnings' # <class '_frozen_importlib.BuiltinImporter'>
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
import '_io' # <class '_frozen_importlib.BuiltinImporter'>
import 'marshal' # <class '_frozen_importlib.BuiltinImporter'>
import 'posix' # <class '_frozen_importlib.BuiltinImporter'>
import '_frozen_importlib_external' # <class '_frozen_importlib.FrozenImporter'>
# installing zipimport hook
import 'time' # <class '_frozen_importlib.BuiltinImporter'>
import 'zipimport' # <class '_frozen_importlib.FrozenImporter'>
# installed zipimport hook
# /private/tmp/unpacked/lib/python3.10/encodings/__pycache__/__init__.cpython-310.pyc matches /private/tmp/unpacked/lib/python3.10/encodings/__init__.py
# code object from '/private/tmp/unpacked/lib/python3.10/encodings/__pycache__/__init__.cpython-310.pyc'
# /private/tmp/unpacked/lib/python3.10/__pycache__/codecs.cpython-310.pyc matches /private/tmp/unpacked/lib/python3.10/codecs.py
# code object from '/private/tmp/unpacked/lib/python3.10/__pycache__/codecs.cpython-310.pyc'
import '_codecs' # <class '_frozen_importlib.BuiltinImporter'>
import 'codecs' # <_frozen_importlib_external.SourceFileLoader object at 0x1012a35b0>
# /private/tmp/unpacked/lib/python3.10/encodings/__pycache__/aliases.cpython-310.pyc matches /private/tmp/unpacked/lib/python3.10/encodings/aliases.py
# code object from '/private/tmp/unpacked/lib/python3.10/encodings/__pycache__/aliases.cpython-310.pyc'
import 'encodings.aliases' # <_frozen_importlib_external.SourceFileLoader object at 0x1012f4b80>
import 'encodings' # <_frozen_importlib_external.SourceFileLoader object at 0x1012a3430>
# /private/tmp/unpacked/lib/python3.10/encodings/__pycache__/utf_8.cpython-310.pyc matches /private/tmp/unpacked/lib/python3.10/encodings/utf_8.py
# code object from '/private/tmp/unpacked/lib/python3.10/encodings/__pycache__/utf_8.cpython-310.pyc'
import 'encodings.utf_8' # <_frozen_importlib_external.SourceFileLoader object at 0x1012a3310>
import '_signal' # <class '_frozen_importlib.BuiltinImporter'>
# /private/tmp/unpacked/lib/python3.10/__pycache__/io.cpython-310.pyc matches /private/tmp/unpacked/lib/python3.10/io.py
# code object from '/private/tmp/unpacked/lib/python3.10/__pycache__/io.cpython-310.pyc'
# /private/tmp/unpacked/lib/python3.10/__pycache__/abc.cpython-310.pyc matches /private/tmp/unpacked/lib/python3.10/abc.py
# code object from '/private/tmp/unpacked/lib/python3.10/__pycache__/abc.cpython-310.pyc'
import '_abc' # <class '_frozen_importlib.BuiltinImporter'>
import 'abc' # <_frozen_importlib_external.SourceFileLoader object at 0x1012f4eb0>
import 'io' # <_frozen_importlib_external.SourceFileLoader object at 0x1012f4ca0>
# /private/tmp/unpacked/lib/python3.10/__pycache__/site.cpython-310.pyc matches /private/tmp/unpacked/lib/python3.10/site.py
# code object from '/private/tmp/unpacked/lib/python3.10/__pycache__/site.cpython-310.pyc'
# /private/tmp/unpacked/lib/python3.10/__pycache__/os.cpython-310.pyc matches /private/tmp/unpacked/lib/python3.10/os.py
# code object from '/private/tmp/unpacked/lib/python3.10/__pycache__/os.cpython-310.pyc'
# /private/tmp/unpacked/lib/python3.10/__pycache__/stat.cpython-310.pyc matches /private/tmp/unpacked/lib/python3.10/stat.py
# code object from '/private/tmp/unpacked/lib/python3.10/__pycache__/stat.cpython-310.pyc'
import '_stat' # <class '_frozen_importlib.BuiltinImporter'>
import 'stat' # <_frozen_importlib_external.SourceFileLoader object at 0x1012f70d0>
# /private/tmp/unpacked/lib/python3.10/__pycache__/_collections_abc.cpython-310.pyc matches /private/tmp/unpacked/lib/python3.10/_collections_abc.py
# code object from '/private/tmp/unpacked/lib/python3.10/__pycache__/_collections_abc.cpython-310.pyc'
import '_collections_abc' # <_frozen_importlib_external.SourceFileLoader object at 0x1012f7430>
# /private/tmp/unpacked/lib/python3.10/__pycache__/posixpath.cpython-310.pyc matches /private/tmp/unpacked/lib/python3.10/posixpath.py
# code object from '/private/tmp/unpacked/lib/python3.10/__pycache__/posixpath.cpython-310.pyc'
# /private/tmp/unpacked/lib/python3.10/__pycache__/genericpath.cpython-310.pyc matches /private/tmp/unpacked/lib/python3.10/genericpath.py
# code object from '/private/tmp/unpacked/lib/python3.10/__pycache__/genericpath.cpython-310.pyc'
import 'genericpath' # <_frozen_importlib_external.SourceFileLoader object at 0x10132dc60>
import 'posixpath' # <_frozen_importlib_external.SourceFileLoader object at 0x1012f74c0>
import 'os' # <_frozen_importlib_external.SourceFileLoader object at 0x1012f5cf0>
# /private/tmp/unpacked/lib/python3.10/__pycache__/_sitebuiltins.cpython-310.pyc matches /private/tmp/unpacked/lib/python3.10/_sitebuiltins.py
# code object from '/private/tmp/unpacked/lib/python3.10/__pycache__/_sitebuiltins.cpython-310.pyc'
import '_sitebuiltins' # <_frozen_importlib_external.SourceFileLoader object at 0x1012f6dd0>
Processing user site-packages
Processing global site-packages
Adding directory: '/private/tmp/unpacked/lib/python3.10/site-packages'
Processing .pth file: '/private/tmp/unpacked/lib/python3.10/site-packages/distutils-precedence.pth'
import 'site' # <_frozen_importlib_external.SourceFileLoader object at 0x1012f56c0>
Python 3.10.6 | packaged by conda-forge | (main, Aug 22 2022, 20:38:29) [Clang 13.0.1 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
[1]    72869 killed     python -v

So things appear to crash when importing the site module.

Conda Info

active environment : /private/tmp/original
    active env location : /private/tmp/original
            shell level : 1
       user config file : /Users/jtilly/.condarc
 populated config files : /Users/jtilly/.condarc
          conda version : 4.14.0
    conda-build version : 3.22.0
         python version : 3.10.6.final.0
       virtual packages : __osx=12.3=0
                          __unix=0=0
                          __archspec=1=arm64
       base environment : /opt/conda  (writable)
      conda av data dir : /opt/conda/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/osx-arm64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /opt/conda/pkgs
                          /Users/jtilly/.conda/pkgs
       envs directories : /opt/conda/envs
                          /Users/jtilly/.conda/envs
               platform : osx-arm64
             user-agent : conda/4.14.0 requests/2.28.1 CPython/3.10.6 Darwin/21.4.0 OSX/12.3
                UID:GID : 501:20
             netrc file : None
           offline mode : False

Conda Config

channels:
  - conda-forge

Conda list

# packages in environment at /private/tmp/original:
#
# Name                    Version                   Build  Channel
bzip2                     1.0.8                h3422bc3_4    conda-forge
ca-certificates           2022.9.14            h4653dfc_0    conda-forge
conda-pack                0.7.0              pyh6c4a22f_0    conda-forge
libffi                    3.4.2                h3422bc3_5    conda-forge
libsqlite                 3.39.3               h76d750c_0    conda-forge
libzlib                   1.2.12               h03a7124_3    conda-forge
ncurses                   6.3                  h07bb92c_1    conda-forge
openssl                   3.0.5                h03a7124_2    conda-forge
pip                       22.2.2             pyhd8ed1ab_0    conda-forge
python                    3.10.6          hae75cb6_0_cpython    conda-forge
python_abi                3.10                    2_cp310    conda-forge
readline                  8.1.2                h46ed386_0    conda-forge
setuptools                59.8.0          py310hbe9552e_1    conda-forge
tk                        8.6.12               he1e0b03_0    conda-forge
tzdata                    2022c                h191b570_0    conda-forge
wheel                     0.37.1             pyhd8ed1ab_0    conda-forge
xz                        5.2.6                h57fd34a_0    conda-forge

Additional Context

I'm running into this issue regardless of the Python version. I'm not running into this issue on linux, linux-aarch64 or windows.

[xhochy]

This is due to the enforced code signing on Apple Silicon. conda-pack is modifying the binaries but not updating the code signatures. Thus all relocated binaries will be killed by the Kernel. We need to call codesign or ldid on the newly relocated binaries in the conda-unpack script.

[vvzen]

Just checking on this issue: was there progress on it or is there something that contributors willing to help could tackle? I'm not entirely clear on how the code signing procedure would work, would you try to move the original signature to the new binary? I imagine that probably it would not be allowed, security-wise, so I guess it means that you would sign the binaries again on the machine of the user running conda-pack ?
Thanks!

[xhochy]

I imagine that probably it would not be allowed, security-wise, so I guess it means that you would sign the binaries again on the machine of the user running conda-pack ?

Yes, you will need to run ldid -S <binary> on any file where a prefix replacement has happened.

[xhochy]

Fix is available in #257