pytorch 0.4.1 c++/cuda extensions missing torch library symbols

[sdrobert]

Hello,

I've been fumbling with my own pytorch extensions for the past few days using the latest pkgs/main distribution of pytorch. It appears as though pytorch's c++ extension mechanism has been broken with this version. To reproduce, place the following meta.yaml file into the test/cpp_extensions directory of the pytorch GitHub repo (I also checked out tags/v0.4.1):

package:
  name: torch_test_cpp_extension
  version: 0.0.0

source:
  path: .

requirements:
  host:
    - pytorch 0.4.1
    - python {{ python }}
    - numpy
  build:
    - {{ compiler('cxx') }}
  run:
    - {{ pin_compatible('pytorch', exact=True) }}

build:
  script: python setup.py install --single-version-externally-managed --record=record.txt

test:
  commands:
    - python -c "import torch; import torch_test_cpp_extension.cpp as cpp_extension"
    # for 0.4.0 there's no wrapper module: python -c "import torch; import torch_test_cpp_extension"

Building with conda 4.5.10 gives the following error:

+ python -c 'import torch; import torch_test_cpp_extension.cpp as cpp_extension'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: /u/sdrobert/.pyenv/versions/miniconda3-latest/conda-bld/torch_test_cpp_extension_1534377734852/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/python3.6/site-packages/torch_test_cpp_extension/cpp.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN2at5ErrorC1ENS_14SourceLocationESs
Tests failed for torch_test_cpp_extension-0.0.0-ha8d69ae_0.tar.bz2 - moving package to /u/sdrobert/.pyenv/versions/miniconda3-latest/conda-bld/broken
WARNING:conda_build.build:Tests failed for torch_test_cpp_extension-0.0.0-ha8d69ae_0.tar.bz2 - moving package to /u/sdrobert/.pyenv/versions/miniconda3-latest/conda-bld/broken
removing: torch_test_cpp_extension-0.0.0-ha8d69ae_0.tar.bz2
TESTS FAILED: torch_test_cpp_extension-0.0.0-ha8d69ae_0.tar.bz2

Change the pytorch version to 0.4.0 (you have to check out tags/v0.4.0 for this) or use pytorch 0.4.1 from pytorch's channel and the recipe successfully finishes.

@jjhelmus In your latest commit (fc96334), you excluded the cpp_extensions tests directly. Am I right to assume that this issue is known to you lot?

Thanks for your time,
Sean

[pathaine]

I am having the same issue.

[jjhelmus]

The issue here is that PyTorch has logic which injects a compiler flag to set the C++ ABI when building an extension. Unfortunately the wrong ABI is being select. Specifically -D_GLIBCXX_USE_CXX11_ABI=0 is addedwhich selects the older non-C++11 ABI but the Anaconda PyTorch package is build with the newer C++11 ABI and extension must also use this ABI.

This logic should be patches out of the Anaconda build of PyTorch since it is not needed.

Until then removing the line which adds this flag inside the build will produce working extension.

For example:

package:
  name: torch_test_cpp_extension
  version: 0.0.0

source:
  path: .

requirements:
  host:
    - pytorch 0.4.1
    - python {{ python }}
    - numpy
    - setuptools
  build:
    - {{ compiler('cxx') }}
  run:
    - {{ pin_compatible('pytorch', exact=True) }}

build:
  script:
    - "sed -i '/self._add_gnu_abi/d' ${SP_DIR}/torch/utils/cpp_extension.py "
    - python setup.py install --single-version-externally-managed --record=record.txt

test:
  commands:
    - python -c "import torch; import torch_test_cpp_extension.cpp as cpp_extension"
    # for 0.4.0 there's no wrapper module: python -c "import torch; import torch_test_cpp_extension"