ModuleNotFoundError: No module named '_sysconfigdata__emscripten_wasm32-emscripten'

[josephrocca]

Is there any obvious reason why buildpkg would be giving the error in the title? I had a build process "working" for JAX (up to this point), but I've recently re-cloned the Pyodide repo and started fresh (replicating the edits and meta.yaml config that I'd made previously), but now I'm running into this, which occurs very early in the build process:

ERROR: An error occurred during the fetch of repository 'local_execution_config_python':
   Traceback (most recent call last):
	File "/src/.docker_home/.cache/bazel/_bazel_joe/7e454c473e82025e727812785b12ccc7/external/org_tensorflow/third_party/py/python_configure.bzl", line 215, column 41, in _create_local_python_repository
		python_include = _get_python_include(repository_ctx, python_bin)
	File "/src/.docker_home/.cache/bazel/_bazel_joe/7e454c473e82025e727812785b12ccc7/external/org_tensorflow/third_party/py/python_configure.bzl", line 152, column 21, in _get_python_include
		result = execute(
	File "/src/.docker_home/.cache/bazel/_bazel_joe/7e454c473e82025e727812785b12ccc7/external/org_tensorflow/third_party/remote_config/common.bzl", line 230, column 13, in execute
		fail(
Error in fail: Problem getting python include path.
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/local/lib/python3.10/distutils/sysconfig.py", line 53, in <module>
    _config_vars = get_config_vars()
  File "/usr/local/lib/python3.10/sysconfig.py", line 615, in get_config_vars
    _init_posix(_CONFIG_VARS)
  File "/usr/local/lib/python3.10/sysconfig.py", line 477, in _init_posix
    _temp = __import__(name, globals(), locals(), ['build_time_vars'], 0)
ModuleNotFoundError: No module named '_sysconfigdata__emscripten_wasm32-emscripten'
Is the Python binary path set up right? (See ./configure or PYTHON_BIN_PATH.) Is distutils installed?

I've confirmed that these environment variables are made available to Bazel from the pypabuild.py caller environment: PYODIDE_ROOT, TARGETINSTALLDIR, PYODIDE_PACKAGE_ABI, PYODIDE_JOBS, PKG_CONFIG_PATH, PYMICRO, PYMAJOR, HOSTSITEPACKAGES, PYMINOR, SIDE_MODULE_LDFLAGS, PWD, SYSCONFIG_NAME, PKG_BUILD_DIR, PYO3_CONFIG_FILE, CARGO_BUILD_TARGET, HOME, PYODIDE, CARGO_HOME, RUSTFLAGS, PYTHONINCLUDE, WASM_LIBRARY_DIR, CARGO_TARGET_WASM32_UNKNOWN_EMSCRIPTEN_LINKER, PYTHONPATH, NUMPY_LIB, SHLVL, EMSDK_NODE, WASM_PKG_CONFIG_PATH, SIDE_MODULE_CFLAGS, LC_CTYPE, PKG_VERSION, UNISOLATED_PACKAGES, PATH, HOSTINSTALLDIR, EMSDK, CPYTHONBUILD, EM_CONFIG, STDLIB_MODULE_CFLAGS, _, LD, CXX, CC, GFORTRAN, CARGO, AR, GCC, PYWASMCROSS_ARGS, _PYTHON_HOST_PLATFORM, _PYTHON_SYSCONFIGDATA_NAME

I'll keep trying to debug this, but just wondering if this hints at some obvious mistake that I've made, or if this could be due to a very recent commit (I'm git cloneing the main branch) which would explain why it seemed to not be a problem for me previously?

[hoodmane]

What's the value of PYTHONPATH?

[josephrocca]

As in, a bug with Pyodide, or something related to my code? I've minimally replicated the PYTHONPATH behavior that I'm seeing like this:

git clone https://github.com/pyodide/pyodide
cd pyodide

Now add these 3 lines here in pywasmcross.py:

print("#########@@@@@@@@@@@@@@@@@@@@@@@@@@@@")
print(env)
print("#########@@@@@@@@@@@@@@@@@@@@@@@@@@@@")

And then run this:

make
pip install ./pyodide-build
cd packages/numpy
python -m pyodide_build buildpkg meta.yaml

And it prints this:

#########@@@@@@@@@@@@@@@@@@@@@@@@@@@@
{'PYODIDE_ROOT': '/src', 'TARGETINSTALLDIR': '/src/cpython/installs/python-3.10.2', 'PYODIDE_PACKAGE_ABI': '1', 'PYODIDE_JOBS': '4', 'PKG_CONFIG_PATH': '/src/packages/.libs/lib/pkgconfig', 'PYMICRO': '2', 'PYMAJOR': '3', 'HOSTSITEPACKAGES': '/src/packages/.artifacts/lib/python3.10/site-packages', 'PYMINOR': '10', 'SIDE_MODULE_LDFLAGS': '-O2 -g0  -s MODULARIZE=1 -s LINKABLE=1 -s EXPORT_ALL=1 -s WASM=1 -std=c++14 -s LZ4=1 -L /src/cpython/installs/python-3.10.2/lib/ -s WASM_BIGINT  -s SIDE_MODULE=1', 'PWD': '/src/packages/numpy', 'SYSCONFIG_NAME': '_sysconfigdata__emscripten_wasm32-emscripten', 'PYO3_CONFIG_FILE': '/src/tools/pyo3_config.ini', 'CARGO_BUILD_TARGET': 'wasm32-unknown-emscripten', 'HOME': '/src/.docker_home', 'PYODIDE': '1', 'CARGO_HOME': '/src/.docker_home/.cargo', 'RUSTFLAGS': '-C relocation-model=pic -C link-arg=-sSIDE_MODULE=2 -C link-arg=-sWASM_BIGINT', 'PYTHONINCLUDE': '/src/cpython/installs/python-3.10.2/include/python3.10', 'WASM_LIBRARY_DIR': '/src/packages/.libs', 'CARGO_TARGET_WASM32_UNKNOWN_EMSCRIPTEN_LINKER': '/src/tools/rust_emcc_wrapper.py', 'PYTHONPATH': '/src/packages/.artifacts/lib/python3.10/site-packages', 'NUMPY_LIB': '/src/packages/.artifacts/lib/python3.10/site-packages/numpy/', 'SHLVL': '0', 'EMSDK_NODE': '/src/emsdk/emsdk/node/14.18.2_64bit/bin/node', 'WASM_PKG_CONFIG_PATH': '/src/packages/.libs/lib/pkgconfig', 'SIDE_MODULE_CFLAGS': '-O2 -g0 -fPIC', 'UNISOLATED_PACKAGES': '["scipy", "pycparser", "cffi", "numpy"]', 'PATH': '/src/tools/symlinks:/src/emsdk/emsdk:/src/emsdk/emsdk/upstream/emscripten:/src/emsdk/emsdk/node/14.18.2_64bit/bin:/src/node_modules/.bin/:/usr/local/bin:/usr/local/sbin:/usr/sbin:/usr/bin:/sbin:/bin:/src/packages/.artifacts/bin/', 'HOSTINSTALLDIR': '/src/packages/.artifacts', 'EMSDK': '/src/emsdk/emsdk', 'CPYTHONBUILD': '/src/cpython/build/Python-3.10.2/', 'EM_CONFIG': '/src/emsdk/emsdk/.emscripten', 'STDLIB_MODULE_CFLAGS': '-O2 -g0 -fPIC  -I Include/ -I .', '_': '/usr/local/bin/python', 'LC_CTYPE': 'C.UTF-8', 'PKG_VERSION': '1.22.3', 'PKG_BUILD_DIR': '/src/packages/numpy/build/numpy-1.22.3', 'GCC': 'gcc', 'LD': 'ld', 'GFORTRAN': 'gfortran', 'CXX': 'c++', 'CC': 'cc', 'CARGO': 'cargo', 'AR': 'ar', 'PYWASMCROSS_ARGS': '{"pkgname": "numpy", "cflags": "-Wno-return-type\\n -O2 -g0 -fPIC", "cxxflags": " ", "ldflags": " -O2 -g0  -s MODULARIZE=1 -s LINKABLE=1 -s EXPORT_ALL=1 -s WASM=1 -std=c++14 -s LZ4=1 -L /src/cpython/installs/python-3.10.2/lib/ -s WASM_BIGINT  -s SIDE_MODULE=1", "target_install_dir": "/src/cpython/installs/python-3.10.2", "replace_libs": "", "builddir": "/src/packages/numpy/build/numpy-1.22.3", "PYTHONPATH": ["/src/packages/numpy", "/usr/local/lib/python310.zip", "/usr/local/lib/python3.10", "/usr/local/lib/python3.10/lib-dynload", "/src/.docker_home/.local/lib/python3.10/site-packages", "/usr/local/lib/python3.10/site-packages", "/src/cpython/installs/python-3.10.2/sysconfigdata"], "orig__name__": "pyodide_build.pywasmcross"}', '_PYTHON_HOST_PLATFORM': 'emscripten_3_1_13_wasm32', '_PYTHON_SYSCONFIGDATA_NAME': '_sysconfigdata__emscripten_wasm32-emscripten'}
#########@@@@@@@@@@@@@@@@@@@@@@@@@@@@

So the value of PYTHONPATH is also /src/packages/.artifacts/lib/python3.10/site-packages in this case.

If this is a Pyodide problem (i.e. not related to my code), then should I wait for someone in the core Pyodide team to look into this? Or is there a workaround that could be used in the meantime?

[josephrocca]

It occurred to me that maybe you mean that the bug is purely related to the PYTHONPATH env variable, and so the obvious temporary workaround would just be to manually set it to /src/cpython/installs/python-3.10.2/sysconfigdata/. I tried doing that above with replace_env(build_env): in pypabuild.py, but I get this:

ERROR: /src/packages/jax/build/jax-0.3.13/jaxlib/BUILD:100:17: Compiling jaxlib/cpu_feature_guard.c failed: undeclared inclusion(s) in rule '//jaxlib:cpu_feature_guard.so':
this rule is missing dependency declarations for the following files included by 'jaxlib/cpu_feature_guard.c':
  '/src/cpython/installs/python-3.10.2/include/python3.10/Python.h'
  '/src/cpython/installs/python-3.10.2/include/python3.10/patchlevel.h'
  '/src/cpython/installs/python-3.10.2/include/python3.10/pyconfig.h'
  '/src/cpython/installs/python-3.10.2/include/python3.10/pymacconfig.h'
  '/src/cpython/installs/python-3.10.2/include/python3.10/pyport.h'
  '/src/cpython/installs/python-3.10.2/include/python3.10/exports.h'
  '/src/cpython/installs/python-3.10.2/include/python3.10/pymacro.h'
  '/src/cpython/installs/python-3.10.2/include/python3.10/pymath.h'
  ...
  ...
  ...

Previously this wasn't an issue, so I'm guessing it's related to the same bug.

[hoodmane]

It's a Pyodide problem that would need to be fixed as part of adding the inside the cross build environment script. The fix is pretty simple though.

[josephrocca]

Ah okay cool - any ETA on a fix?

[hoodmane]

I'll try to do it today. The main blocker is naming the meta.yaml keys still. But we can just say that they are unstable still.