Conflicts using conda-forge and bioconda with strict channel priority

[alimanfoo]

I would like to use conda with the --strict-channel-priority option, because this helps to ensure packages come from the desired channels.

I have a problem regarding channel ordering which means I cannot install certain combinations of packages into the same environment, when using strict channel priority.

For example, if I use channel priority conda-forge > biopython, then I can install biopython and python 3.6 together, no problem:

$ conda create --yes --dry-run --strict-channel-priority --override-channels --channel conda-forge --channel bioconda --channel defaults --name foobar python=3.6 biopython
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/aliman/malariagen/binder/conda/envs/foobar

  added / updated specs:
    - biopython
    - python=3.6

The following NEW packages will be INSTALLED:

  _libgcc_mutex      pkgs/main/linux-64::_libgcc_mutex-0.1-main
  biopython          conda-forge/linux-64::biopython-1.74-py36h516909a_0
  ca-certificates    conda-forge/linux-64::ca-certificates-2019.6.16-hecc5488_0
  certifi            conda-forge/linux-64::certifi-2019.6.16-py36_1
  libblas            conda-forge/linux-64::libblas-3.8.0-12_openblas
  libcblas           conda-forge/linux-64::libcblas-3.8.0-12_openblas
  libffi             conda-forge/linux-64::libffi-3.2.1-he1b5a44_1006
  libgcc-ng          pkgs/main/linux-64::libgcc-ng-9.1.0-hdf63c60_0
  libgfortran-ng     pkgs/main/linux-64::libgfortran-ng-7.3.0-hdf63c60_0
  liblapack          conda-forge/linux-64::liblapack-3.8.0-12_openblas
  libopenblas        conda-forge/linux-64::libopenblas-0.3.7-h6e990d7_1
  libstdcxx-ng       pkgs/main/linux-64::libstdcxx-ng-9.1.0-hdf63c60_0
  ncurses            conda-forge/linux-64::ncurses-6.1-hf484d3e_1002
  numpy              conda-forge/linux-64::numpy-1.17.1-py36h95a1406_0
  openssl            conda-forge/linux-64::openssl-1.1.1c-h516909a_0
  pip                conda-forge/linux-64::pip-19.2.3-py36_0
  python             conda-forge/linux-64::python-3.6.7-h357f687_1005
  readline           conda-forge/linux-64::readline-8.0-hf8c457e_0
  setuptools         conda-forge/linux-64::setuptools-41.2.0-py36_0
  sqlite             conda-forge/linux-64::sqlite-3.29.0-hcee41ef_1
  tk                 conda-forge/linux-64::tk-8.6.9-hed695b0_1002
  wheel              conda-forge/linux-64::wheel-0.33.6-py36_0
  xz                 conda-forge/linux-64::xz-5.2.4-h14c3975_1001
  zlib               conda-forge/linux-64::zlib-1.2.11-h516909a_1005

However, if I try to use channel priority bioconda > conda-forge, I get a conflict:

$ conda create --yes --dry-run --strict-channel-priority --override-channels --channel bioconda --channel conda-forge --channel defaults --name foobar python=3.6 biopython
Collecting package metadata (current_repodata.json): done
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: | 
Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
failed                                                                                                                                                                       
                                                                                                                                                                             
UnsatisfiableError: The following specifications were found to be incompatible with each other:                                                                              

Package wheel conflicts for:
python=3.6 -> pip -> wheel
biopython -> python=2.7 -> pip -> wheel
Package ca-certificates conflicts for:
python=3.6 -> openssl=1.0 -> ca-certificates
biopython -> python=2.7 -> openssl[version='>=1.1.1a,<1.1.2a'] -> ca-certificates
Package setuptools conflicts for:
python=3.6 -> pip -> setuptools
biopython -> python=2.7 -> pip -> setuptools
Package python conflicts for:
python=3.6
biopython -> reportlab -> pillow[version='>=2.4.0'] -> olefile -> python
Package pip conflicts for:
python=3.6 -> pip
biopython -> python=2.7 -> pip
Package certifi conflicts for:
python=3.6 -> pip -> setuptools -> certifi[version='>=2016.09']
biopython -> python=2.7 -> pip -> setuptools -> certifi[version='>=2016.09']
Package msgpack-python conflicts for:
biopython -> numpy=1.10 -> msgpack-python
Package mmtf-python conflicts for:
biopython -> mmtf-python
Note that strict channel priority may have removed packages required for satisfiability.

In this particular case I realise that the problem occurs because biopython is available both channels, but is out of date in bioconda. It's solved in this case by channel priority conda-forge > bioconda. However, I think there is a more general problem, because there are other problems when I try to use conda-forge > bioconda. E.g., if I try to install bcftools with channel priority conda-forge > biopython then I get a conflict, e.g.:

$ conda create --yes --dry-run --strict-channel-priority --override-channels --channel conda-forge --channel bioconda --channel defaults --name foobar bcftools=1.9
Collecting package metadata (current_repodata.json): done
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: - 
Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
failed                               

UnsatisfiableError: The following specifications were found to be incompatible with each other:

Package bzip2 conflicts for:
bcftools=1.9 -> bzip2[version='>=1.0.6,<2.0a0|>=1.0.8,<2.0a0']
Package libgcc-ng conflicts for:
bcftools=1.9 -> libgcc-ng[version='>=7.3.0']
Package curl conflicts for:
bcftools=1.9 -> curl[version='>=7.59.0,<8.0a0|>=7.64.0,<8.0a0|>=7.64.1,<8.0a0']
Package gsl conflicts for:
bcftools=1.9 -> gsl[version='>=2.5,<2.6.0a0']
Package zlib conflicts for:
bcftools=1.9 -> zlib[version='>=1.2.11,<1.3.0a0']
Package xz conflicts for:
bcftools=1.9 -> xz[version='>=5.2.4,<5.3.0a0']
Package libdeflate conflicts for:
bcftools=1.9 -> libdeflate[version='>=1.0,<1.1.0a0']
Note that strict channel priority may have removed packages required for satisfiability.

This I particularly do not understand, because I am only asking to install a single package (bcftool), and so am surprised there can even occur a conflict/unsatisfiability.

Any help or advice here much appreciated.

[alimanfoo]

Just to add that if I use channel priority bioconda > conda-forge, and pin the biopython version number to ensure it comes from conda-forge, I then still get conflicts:

$ conda create --yes --dry-run --strict-channel-priority --override-channels --channel bioconda --channel conda-forge --channel defaults --name foobar python=3.6 biopython==1.74
Collecting package metadata (current_repodata.json): done
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: - 
Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
failed                                                                                                                                                                       

UnsatisfiableError: The following specifications were found to be incompatible with each other:

Package xz conflicts for:
python=3.6 -> xz[version='5.2.*|>=5.2.3,<5.3.0a0|>=5.2.4,<5.3.0a0']
Package readline conflicts for:
python=3.6 -> readline[version='6.2.*|7.0|>=7.0,<8.0a0|>=8.0,<9.0a0']
Package tk conflicts for:
python=3.6 -> tk[version='8.5.*|8.6.*|>=8.6.8,<8.7.0a0|>=8.6.9,<8.7.0a0']
Package zlib conflicts for:
python=3.6 -> zlib[version='1.2.*|1.2.11|1.2.8|>=1.2.11,<1.3.0a0']
Package libffi conflicts for:
python=3.6 -> libffi[version='>=3.2.1,<3.3.0a0']
Package sqlite conflicts for:
python=3.6 -> sqlite[version='3.13.*|3.20.*|>=3.24.0,<4.0a0|>=3.25.2,<4.0a0|>=3.25.3,<4.0a0|>=3.26.0,<4.0a0|>=3.28.0,<4.0a0']
Package libgcc-ng conflicts for:
python=3.6 -> libgcc-ng[version='>=4.9|>=7.3.0']
Package libstdcxx-ng conflicts for:
python=3.6 -> libstdcxx-ng[version='>=4.9|>=7.3.0']
Package pip conflicts for:
python=3.6 -> pip
Package openssl conflicts for:
python=3.6 -> openssl[version='1.0.*|>=1.0.2o,<1.0.3a|>=1.0.2p,<1.0.3a|>=1.1.1a,<1.1.2a']
Package bzip2 conflicts for:
python=3.6 -> bzip2[version='>=1.0.6,<2.0a0']
Package ncurses conflicts for:
python=3.6 -> ncurses[version='5.9|5.9.*|>=6.1,<6.2.0a0']
Note that strict channel priority may have removed packages required for satisfiability.

[alimanfoo]

$ conda --version
conda 4.7.11

[bgruening]

@alimanfoo I can at least narrow down the failure. You should always use conda-forge > bioconda.

[bgruening]

And for me this works :(

bag@bag:~/projects/code/bioconda-recipes/recipes$ conda create --yes --dry-run --strict-channel-priority --override-channels --channel conda-forge --channel bioconda --channel defaults --name foobar bcftools=1.9
Collecting package metadata: done
Solving environment: | WARNING conda.common.logic:get_sat_solver_cls(278): Could not run SAT solver through interface 'pycryptosat'.
done


==> WARNING: A newer version of conda exists. <==
  current version: 4.6.14
  latest version: 4.7.12

Please update conda by running

    $ conda update -n base conda



## Package Plan ##

  environment location: /home/bag/miniconda3/envs/foobar

  added / updated specs:
    - bcftools=1.9


The following NEW packages will be INSTALLED:

  _libgcc_mutex      pkgs/main/linux-64::_libgcc_mutex-0.1-main
  bcftools           bioconda/linux-64::bcftools-1.9-h68d8f2e_7
  bzip2              conda-forge/linux-64::bzip2-1.0.8-h516909a_1
  ca-certificates    conda-forge/linux-64::ca-certificates-2019.9.11-hecc5488_0
  curl               conda-forge/linux-64::curl-7.65.3-hf8cf82a_0
  gsl                conda-forge/linux-64::gsl-2.5-h294904e_1
  htslib             bioconda/linux-64::htslib-1.9-h4da6232_3
  krb5               conda-forge/linux-64::krb5-1.16.3-h05b26f9_1001
  libblas            conda-forge/linux-64::libblas-3.8.0-12_openblas
  libcblas           conda-forge/linux-64::libcblas-3.8.0-12_openblas
  libcurl            conda-forge/linux-64::libcurl-7.65.3-hda55be3_0
  libdeflate         conda-forge/linux-64::libdeflate-1.3-h516909a_0
  libedit            conda-forge/linux-64::libedit-3.1.20170329-hf8c457e_1001
  libgcc-ng          pkgs/main/linux-64::libgcc-ng-9.1.0-hdf63c60_0
  libgfortran-ng     pkgs/main/linux-64::libgfortran-ng-7.3.0-hdf63c60_0
  libopenblas        conda-forge/linux-64::libopenblas-0.3.7-h6e990d7_1
  libssh2            conda-forge/linux-64::libssh2-1.8.2-h22169c7_2
  libstdcxx-ng       pkgs/main/linux-64::libstdcxx-ng-9.1.0-hdf63c60_0
  ncurses            conda-forge/linux-64::ncurses-6.1-hf484d3e_1002
  openssl            conda-forge/linux-64::openssl-1.1.1c-h516909a_0
  perl               conda-forge/linux-64::perl-5.26.2-h516909a_1006
  tk                 conda-forge/linux-64::tk-8.6.9-hed695b0_1003
  xz                 conda-forge/linux-64::xz-5.2.4-h14c3975_1001
  zlib               conda-forge/linux-64::zlib-1.2.11-h516909a_1006



DryRunExit: Dry run. Exiting.

But I also have still conda 4.6.14.
Anyone else can test this?

[simonbray]

With the newer conda I get:

--strict-channel-priority --override-channels --channel conda-forge --channel bioconda --channel defaults --name foobar bcftools=1.9
Collecting package metadata (current_repodata.json): done
Solving environment: done


==> WARNING: A newer version of conda exists. <==
  current version: 4.7.11
  latest version: 4.7.12

Please update conda by running

    $ conda update -n base conda



## Package Plan ##

  environment location: /home/simon/miniconda3/envs/foobar

  added / updated specs:
    - bcftools=1.9


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    _libgcc_mutex-0.1          |             main           3 KB
    bcftools-1.9               |       h68d8f2e_7         672 KB  bioconda
    htslib-1.9                 |       h4da6232_3         1.2 MB  bioconda
    libdeflate-1.3             |       h516909a_0          60 KB  conda-forge
    libgcc-ng-9.1.0            |       hdf63c60_0         5.1 MB
    libgfortran-ng-7.3.0       |       hdf63c60_0        1006 KB
    libstdcxx-ng-9.1.0         |       hdf63c60_0         3.1 MB
    perl-5.26.2                |    h516909a_1006        15.4 MB  conda-forge
    ------------------------------------------------------------
                                           Total:        26.5 MB

The following NEW packages will be INSTALLED:

  _libgcc_mutex      pkgs/main/linux-64::_libgcc_mutex-0.1-main
  bcftools           bioconda/linux-64::bcftools-1.9-h68d8f2e_7
  bzip2              conda-forge/linux-64::bzip2-1.0.8-h516909a_1
  ca-certificates    conda-forge/linux-64::ca-certificates-2019.9.11-hecc5488_0
  curl               conda-forge/linux-64::curl-7.65.3-hf8cf82a_0
  gsl                conda-forge/linux-64::gsl-2.5-h294904e_1
  htslib             bioconda/linux-64::htslib-1.9-h4da6232_3
  krb5               conda-forge/linux-64::krb5-1.16.3-h05b26f9_1001
  libblas            conda-forge/linux-64::libblas-3.8.0-12_openblas
  libcblas           conda-forge/linux-64::libcblas-3.8.0-12_openblas
  libcurl            conda-forge/linux-64::libcurl-7.65.3-hda55be3_0
  libdeflate         conda-forge/linux-64::libdeflate-1.3-h516909a_0
  libedit            conda-forge/linux-64::libedit-3.1.20170329-hf8c457e_1001
  libgcc-ng          pkgs/main/linux-64::libgcc-ng-9.1.0-hdf63c60_0
  libgfortran-ng     pkgs/main/linux-64::libgfortran-ng-7.3.0-hdf63c60_0
  libopenblas        conda-forge/linux-64::libopenblas-0.3.7-h6e990d7_1
  libssh2            conda-forge/linux-64::libssh2-1.8.2-h22169c7_2
  libstdcxx-ng       pkgs/main/linux-64::libstdcxx-ng-9.1.0-hdf63c60_0
  ncurses            conda-forge/linux-64::ncurses-6.1-hf484d3e_1002
  openssl            conda-forge/linux-64::openssl-1.1.1c-h516909a_0
  perl               conda-forge/linux-64::perl-5.26.2-h516909a_1006
  tk                 conda-forge/linux-64::tk-8.6.9-hed695b0_1003
  xz                 conda-forge/linux-64::xz-5.2.4-h14c3975_1001
  zlib               conda-forge/linux-64::zlib-1.2.11-h516909a_1006



DryRunExit: Dry run. Exiting.

[bgruening]

Thanks @simonbray for testing.
@alimanfoo any change to can retest?

[alimanfoo]

Thanks @bgruening and @simonbray for following this up, sorry I fell of the radar here. This looks fixed to me too so will close.