Need some way to lock the version of boost to what we build against #71

greglandrum · 2018-07-16T14:06:25Z

Boost version numbers end up in the library names, so using DLLs from newer boost versions does not work.

This is connected to this rdkit issue.
rdkit/rdkit#1957

greglandrum · 2018-07-16T14:08:37Z

@rvianello : the immediately obvious way to do this would be to just directly match the version numbers in the host and run sections.
Can you think of a better solution?

rvianello · 2018-07-16T14:32:09Z

Since a conda_build_config.yamlfile is already present for the rdkit recipe, maybe we could try using the pinning mechanism that is documented to work for build variants (it shouldn't necessarily imply building multiple rdkit packages).

The section with title "Pinning at the variant level" at https://conda.io/docs/user-guide/tasks/build-packages/variants.html provides an example that specifically addresses the boost case. The following section ("Pinning at the recipe level") illustrates how to obtain a similar result in the meta.yaml file alone.

markussitzmann · 2018-08-06T07:19:08Z

Does this already work? I merged all changes @greglandrum and @rvianello did in their forks and build in a fresh container, it build RDKit 2018.03.3, but at the end it is missing boost-1.65.1 and the container somehow has 1.67 installed:

`The following NEW packages will be INSTALLED:

blas:            1.0-mkl                         
bzip2:           1.0.6-h14c3975_5                
ca-certificates: 2018.03.07-0                    
cairo:           1.14.12-h8948797_3              
certifi:         2018.4.16-py36_0                
fontconfig:      2.13.0-h9420a91_0               
freetype:        2.9.1-h8a8886c_0                
glib:            2.56.1-h000015b_0               
icu:             58.2-h9c2bf20_1                 
intel-openmp:    2018.0.3-0                      
jpeg:            9b-h024ee3a_2                   
libboost:        1.67.0-h46d08c1_4               
libedit:         3.1.20170329-h6b74fdf_2         
libffi:          3.2.1-hd88cf55_4                
libgcc-ng:       7.2.0-hdf63c60_3                
libgfortran-ng:  7.2.0-hdf63c60_3                
libpng:          1.6.34-hb9fc6fc_0               
libstdcxx-ng:    7.2.0-hdf63c60_3                
libtiff:         4.0.9-he85c1e1_1                
libuuid:         1.0.3-h1bed415_2                
libxcb:          1.13-h1bed415_1                 
libxml2:         2.9.8-h26e45fe_1                
mkl:             2018.0.3-1                      
mkl_fft:         1.0.4-py36h4414c95_1            
mkl_random:      1.0.1-py36h4414c95_1            
ncurses:         6.1-hf484d3e_0                  
numpy:           1.15.0-py36h1b885b7_0           
numpy-base:      1.15.0-py36h3dfced4_0           
olefile:         0.45.1-py36_0                   
openssl:         1.0.2o-h14c3975_1               
pandas:          0.23.3-py36h04863e7_0           
pcre:            8.42-h439df22_0                 
pillow:          5.2.0-py36heded4f4_0            
pip:             10.0.1-py36_0                   
pixman:          0.34.0-hceecf20_3               
py-boost:        1.67.0-py36h04863e7_4           
python:          3.6.6-hc3d631a_0                
python-dateutil: 2.7.3-py36_0                    
pytz:            2018.5-py36_0                   
rdkit:           2018.03.3.0-py36h6bb024c_1 local
readline:        7.0-ha6073c6_4                  
setuptools:      39.2.0-py36_0                   
six:             1.11.0-py36_1                   
sqlite:          3.24.0-h84994c4_0               
tk:              8.6.7-hc745277_3                
wheel:           0.31.1-py36_0                   
xz:              5.2.4-h14c3975_4                
zlib:            1.2.11-ha838bed_2

Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
import: 'rdkit'
Traceback (most recent call last):
File "/opt/conda/conda-bld/rdkit_1533517910182/test_tmp/run_test.py", line 2, in
import rdkit
File "/opt/conda/conda-bld/rdkit_1533517910182/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/lib/python3.6/site-packages/rdkit/init.py", line 2, in
from .rdBase import rdkitVersion as version
ImportError: libboost_python3.so.1.65.1: cannot open shared object file: No such file or directory
Tests failed for rdkit-2018.03.3.0-py36h6bb024c_1.tar.bz2 - moving package to /opt/conda/conda-bld/broken
WARNING:conda_build.build:Tests failed for rdkit-2018.03.3.0-py36h6bb024c_1.tar.bz2 - moving package to /opt/conda/conda-bld/broken
removing: rdkit-2018.03.3.0-py36h6bb024c_1.tar.bz2
TESTS FAILED: rdkit-2018.03.3.0-py36h6bb024c_1.tar.bz2
ERROR: Service 'rdkit-conda-compile' failed to build: The command '/bin/sh -c CONDA_PY=36 conda install jinja2 conda-build --yes && CONDA_PY=36 conda build boost --quiet --no-anaconda-upload && CONDA_PY=36 conda build nox --quiet --no-anaconda-upload && CONDA_PY=36 conda build cairo_nox --quiet --no-anaconda-upload && CONDA_PY=36 conda build cairocffi --quiet --no-anaconda-upload && CONDA_PY=36 conda build eigen --quiet --no-anaconda-upload && CONDA_PY=36 CONDA_NPY=112 conda build rdkit --quiet --no-anaconda-upload' returned a non-zero code: 1`

greglandrum · 2018-08-06T09:15:16Z

@markussitzmann : when I do a fresh build using the development branch of this recipe everything works fine. Here's what gets installed for me on a centos6 container:

(py36_rdkit) [root@414215a97f78 conda-rdkit]# conda install --use-local rdkit
Solving environment: done

## Package Plan ##

  environment location: /opt/conda/envs/py36_rdkit

  added / updated specs: 
    - rdkit


The following NEW packages will be INSTALLED:

    blas:            1.0-mkl                         
    bzip2:           1.0.6-h14c3975_5                
    ca-certificates: 2018.03.07-0                    
    cairo:           1.14.12-h8948797_3              
    certifi:         2018.4.16-py36_0                
    fontconfig:      2.13.0-h9420a91_0               
    freetype:        2.9.1-h8a8886c_0                
    glib:            2.56.1-h000015b_0               
    icu:             58.2-h9c2bf20_1                 
    intel-openmp:    2018.0.3-0                      
    jpeg:            9b-h024ee3a_2                   
    libboost:        1.65.1-habcd387_4               
    libedit:         3.1.20170329-h6b74fdf_2         
    libffi:          3.2.1-hd88cf55_4                
    libgcc-ng:       7.2.0-hdf63c60_3                
    libgfortran-ng:  7.2.0-hdf63c60_3                
    libpng:          1.6.34-hb9fc6fc_0               
    libstdcxx-ng:    7.2.0-hdf63c60_3                
    libtiff:         4.0.9-he85c1e1_1                
    libuuid:         1.0.3-h1bed415_2                
    libxcb:          1.13-h1bed415_1                 
    libxml2:         2.9.8-h26e45fe_1                
    mkl:             2018.0.3-1                      
    mkl_fft:         1.0.4-py36h4414c95_1            
    mkl_random:      1.0.1-py36h4414c95_1            
    ncurses:         6.1-hf484d3e_0                  
    numpy:           1.15.0-py36h1b885b7_0           
    numpy-base:      1.15.0-py36h3dfced4_0           
    olefile:         0.45.1-py36_0                   
    openssl:         1.0.2o-h14c3975_1               
    pandas:          0.23.3-py36h04863e7_0           
    pcre:            8.42-h439df22_0                 
    pillow:          5.2.0-py36heded4f4_0            
    pip:             10.0.1-py36_0                   
    pixman:          0.34.0-hceecf20_3               
    py-boost:        1.65.1-py36hf484d3e_4           
    python:          3.6.6-hc3d631a_0                
    python-dateutil: 2.7.3-py36_0                    
    pytz:            2018.5-py36_0                   
    rdkit:           2018.03.3.0-py36h71b666b_1 local
    readline:        7.0-ha6073c6_4                  
    setuptools:      39.2.0-py36_0                   
    six:             1.11.0-py36_1                   (py36_rdkit) [root@414215a97f78 conda-rdkit]# conda install --use-local rdkit
Solving environment: done

## Package Plan ##

  environment location: /opt/conda/envs/py36_rdkit

  added / updated specs: 
    - rdkit


The following NEW packages will be INSTALLED:

    blas:            1.0-mkl                         
    bzip2:           1.0.6-h14c3975_5                
    ca-certificates: 2018.03.07-0                    
    cairo:           1.14.12-h8948797_3              
    certifi:         2018.4.16-py36_0                
    fontconfig:      2.13.0-h9420a91_0               
    freetype:        2.9.1-h8a8886c_0                
    glib:            2.56.1-h000015b_0               
    icu:             58.2-h9c2bf20_1                 
    intel-openmp:    2018.0.3-0                      
    jpeg:            9b-h024ee3a_2                   
    libboost:        1.65.1-habcd387_4               
    libedit:         3.1.20170329-h6b74fdf_2         
    libffi:          3.2.1-hd88cf55_4                
    libgcc-ng:       7.2.0-hdf63c60_3                
    libgfortran-ng:  7.2.0-hdf63c60_3                
    libpng:          1.6.34-hb9fc6fc_0               
    libstdcxx-ng:    7.2.0-hdf63c60_3                
    libtiff:         4.0.9-he85c1e1_1                
    libuuid:         1.0.3-h1bed415_2                
    libxcb:          1.13-h1bed415_1                 
    libxml2:         2.9.8-h26e45fe_1                
    mkl:             2018.0.3-1                      
    mkl_fft:         1.0.4-py36h4414c95_1            
    mkl_random:      1.0.1-py36h4414c95_1            
    ncurses:         6.1-hf484d3e_0                  
    numpy:           1.15.0-py36h1b885b7_0           
    numpy-base:      1.15.0-py36h3dfced4_0           
    olefile:         0.45.1-py36_0                   
    openssl:         1.0.2o-h14c3975_1               
    pandas:          0.23.3-py36h04863e7_0           
    pcre:            8.42-h439df22_0                 
    pillow:          5.2.0-py36heded4f4_0            
    pip:             10.0.1-py36_0                   
    pixman:          0.34.0-hceecf20_3               
    py-boost:        1.65.1-py36hf484d3e_4           
    python:          3.6.6-hc3d631a_0                
    python-dateutil: 2.7.3-py36_0                    
    pytz:            2018.5-py36_0                   
    rdkit:           2018.03.3.0-py36h71b666b_1 local
    readline:        7.0-ha6073c6_4                  
    setuptools:      39.2.0-py36_0                   
    six:             1.11.0-py36_1                   
    sqlite:          3.24.0-h84994c4_0               
    tk:              8.6.7-hc745277_3                
    wheel:           0.31.1-py36_0                   
    xz:              5.2.4-h14c3975_4                
    zlib:            1.2.11-ha838bed_2               

Proceed ([y]/n)? 

    sqlite:          3.24.0-h84994c4_0               
    tk:              8.6.7-hc745277_3                
    wheel:           0.31.1-py36_0                   
    xz:              5.2.4-h14c3975_4                
    zlib:            1.2.11-ha838bed_2               

Proceed ([y]/n)?

markussitzmann · 2018-08-06T11:24:32Z

@greglandrum Thanks for the testing and reply - although I hate it :-) ... I definitely have put boost-1.65.1 everywhere and diff'ed with yours and Riccardo's branches/forks. But somehow it ends up with boost-1.67.

I can retry only tonight. On which distro did you build? Centos? I build on debian:stretch

greglandrum · 2018-08-06T11:42:23Z

@markussitzmann : It also didn't work for me when I tried to force boost 1.65.1. The current branch is set to use 1.65, which grabs 1.65.1 by default.

I'm using centos, but that shouldn't make a difference.

Can you point me to the code you're using to do the build?

markussitzmann · 2018-08-06T12:12:02Z

While looking for the code I might have found the error I made but as I said I can only try tonight again (did the fixes in my repository but checked out the wrong branch during the Docker build - and my attempts were revolving around this 1.65 vs 1.65.1 thingy). Thanks again - I only will come back if I still struggle ;-).

markussitzmann · 2018-08-06T20:14:46Z

@greglandrum: hmm, when I run the development branch of this recipe here, the build works now, however now I receive boost 1.63 as built boost package and then starting rdkit fails because it is missing 1.65.1 ... a bit mysterious.

markussitzmann · 2018-08-06T20:20:41Z

I run this Dockerfile:

https://github.com/chembience/chembience/blob/master/context/base/rdkit-conda-compile/Dockerfile

after replacing line 44-46 with the standard conda-rdkit repository and branch development:

RUN git clone https://github.com/rdkit/conda-rdkit.git
WORKDIR conda-rdkit
RUN git checkout development

greglandrum · 2018-08-07T03:56:38Z

Unless you need it for something else, you can probably skip the boost build now (line 51 in your Dockerfile); that will use the boost build recipe from the RDKit channel, which will use 1.63. You almost certainly don't need to build eigen either.

I don't see anything else obviously wrong in the dockerfile.

When you get failures: do they happen during the build or later when you try and use the contents of the image?

greglandrum · 2018-08-07T05:56:30Z

FWIW: these two containers are what I am using to do conda RDKit builds:
https://github.com/rdkit/rdkit_containers/tree/centos6_cleanup/docker/centos6_modern_bare
https://github.com/rdkit/rdkit_containers/tree/centos6_cleanup/docker/centos6_modernconda

markussitzmann · 2018-08-07T08:43:55Z

@greglandrum Thanks for your hints. The builds fail in different places depending what I try. Sometimes during conda is packaging, sometimes when I start RDKit which then misses the right boost version (this happens with a subsequent image which copies all compiled packages from the image I build with the Dockerfile above).

Yep, and I need to give my Dockerfiles another clean up, they have collected old, overcautious or unnecessary stuff.

I am also considering to just switch to your already available anaconda packages; the reason for compiling RDKit myself was that very early last year, when I started to turn my preliminary works into something real, the anaconda RDKit packages still were on Python 3.5 and I had decided to go for Python 3.6 (BTW, Python 3.7 is out now for a few months already :-))

greglandrum · 2018-08-07T08:52:59Z

Yeah, assuming that there is an anaconda distrib that supports it by then, I will do Python 3.7 builds (and probably retire 3.5) for the 2018.09 release.

szcc · 2018-11-28T14:42:30Z

I have exactly same problem :ImportError: libboost_python3.so.1.65.1: cannot open shared object file: No such file or directory
Tests failed for rdkit-2018.03.3.0-py36h6bb024c_1.tar.bz2 - moving package to /opt/conda/conda-bld/broken.
Is there any solution for this?

greglandrum added the bug label Jul 16, 2018

greglandrum mentioned this issue Jul 18, 2018

RDKit import failure caused by Boost incompatibility. rdkit/rdkit#1963

Closed

rvianello mentioned this issue Jul 23, 2018

pin the runtime dependency from the boost libraries #72

Merged

greglandrum pushed a commit that referenced this issue Jul 24, 2018

pin the runtime dependency from the boost libraries (#71) (#72)

b8a949f

markussitzmann mentioned this issue Aug 6, 2018

fix unix boost version to 1.65.1 #66

Closed

rvianello mentioned this issue Dec 28, 2018

py-boost runtime should be = not >= #75

Open

lohedges mentioned this issue Jan 27, 2022

Python improvements and enhancements for the 2022.1.0 release michellab/BioSimSpace#273

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Need some way to lock the version of boost to what we build against #71

Need some way to lock the version of boost to what we build against #71

greglandrum commented Jul 16, 2018

greglandrum commented Jul 16, 2018

rvianello commented Jul 16, 2018

markussitzmann commented Aug 6, 2018

greglandrum commented Aug 6, 2018

markussitzmann commented Aug 6, 2018

greglandrum commented Aug 6, 2018

markussitzmann commented Aug 6, 2018

markussitzmann commented Aug 6, 2018

markussitzmann commented Aug 6, 2018

greglandrum commented Aug 7, 2018

greglandrum commented Aug 7, 2018

markussitzmann commented Aug 7, 2018

greglandrum commented Aug 7, 2018

szcc commented Nov 28, 2018

Need some way to lock the version of boost to what we build against #71

Need some way to lock the version of boost to what we build against #71

Comments

greglandrum commented Jul 16, 2018

greglandrum commented Jul 16, 2018

rvianello commented Jul 16, 2018

markussitzmann commented Aug 6, 2018

greglandrum commented Aug 6, 2018

markussitzmann commented Aug 6, 2018

greglandrum commented Aug 6, 2018

markussitzmann commented Aug 6, 2018

markussitzmann commented Aug 6, 2018

markussitzmann commented Aug 6, 2018

greglandrum commented Aug 7, 2018

greglandrum commented Aug 7, 2018

markussitzmann commented Aug 7, 2018

greglandrum commented Aug 7, 2018

szcc commented Nov 28, 2018