add mpi4py and its friends

[leofang]

I took the recipes from their conda-forge counterparts. Can be built with the Docker image nsls2/debian-with-miniconda. This will benefit the ptychography code and, interestingly, was requested by an HXN user earlier today, for supposedly different purpose.

A little background just for the record: MPI is a de facto standard for inter-process communication, tailored for scientific computation, and vendors are free to provide their own MPI implementations that conform the MPI standard (v3.1 is the latest). Open MPI and MPICH are two of the most popular, open-sourced, implementations, and both are supported in conda-forge. Since the APIs are standardized, in principle one could swap the underlying MPI library with another without rebuilding the top-level applications (such as mpi4py here). In practice, however, not all vendors guarantee ABI compatibility, and so we need to build mpi4py for each supported MPI implementation.

The reason for supporting both MPI libraries is that one sometimes performs better than the other, or does not crash in certain occasions while the other does, so we need more than one implementation for easy verification and test.

As for why we need the metapackage mpi, my answer shifts to "I do not know". I originally thought mpi4py depends on mpi which in turn depends on the vendors, but I was wrong. Now I'm just following the current practice, which was summarized in the end of a two-year-long discussion conda-forge/staged-recipes#1501 (comment).

UPDATE: the metapackage mpi is to ensure mutual exclusivity of MPI implementations; that is, Open MPI and MPICH cannot coexist. See conda-forge/staged-recipes#1501 (comment).

I keep the dependencies on c, cxx and fortran compilers on build time for both MPIs (as they were in conda-forge) because on a second thought I don't think it'd hurt. Most use cases in our environment would probably not use mpicc, mpic++, or mpifort, but simply link to the MPI shared library. But if anything wrong with the runtime libraries is reported, this should be revisited.

One thing we could play with is to build a version of h5py on top of mpi4py, see conda-forge/h5py-feedstock. But I do not know if this could benefit anything we're doing. (Long ago I tested loading images from Databroker GPFS with the mpio driver but no speedup was observed --- possibly one process is enough to saturate the IO bandwidth?)

[leofang]

btw, when testing building I had to update conda-build to avoid conda.exceptions.InvalidMatchSpec: Invalid spec 'mpi 1.0 openmpi': Cannot match on field(s): {'namespace'} (this also happened for mpich).

[licode]

Thanks for the work. Do you think it will be better if we use "mpicc" to compile other than c? We just need to install open mpi or mpich2 in docker file.

[leofang]

@licode I'm not sure if I understand your question correctly. You still need a C compiler in the toolchain to back mpicc. It's just a wrapper compiler that understands the MPI semantics, and the normal C codes are still compiled by gcc. And if you meant you want to compile mpi4py using a particular mpicc built from, say, Open MPI, I would not suggest it, because it would make the mpi4py-MPICH build problematic.

[licode]

I see. Vote for switch between open MPI and mpich. which conda-build version do you use?

[leofang]

I think it's 3.17.8.

[mrakitin]

In nsls2/debian-with-miniconda we have conda-build 3.16.2.

[leofang]

I'm guessing we need conda-build>=3.17.0, see https://github.com/conda/conda-build/blob/master/CHANGELOG.txt:

avoid discarding build string during pin_run_as_build and ensure_valid_spec #3264

[leofang]

Yes I was right -- the bug fix to conda-build was introduced for exactly the same attempt (building BLAS/MPI related packages that need to resolve build strings), see conda/conda-build/pull/3264

[leofang]

@licode @mrakitin I updated the build requirement so that building with nsls2/debian-with-miniconda no longer requires any intervention. This PR can be merged now. Thanks.

[mrakitin]

Thanks for your work, @leofang! Looks good to me now.

[mrakitin]

It's good the new version fixes the known bug.

[mrakitin]

Good point! I tried to rebuild the Docker image for nsls2/debian-with-miniconda with the newer version of conda-build, but it failed on some strange error. Will retry it later today. Anyway, your addition will fix it for now.

[leofang]

No problem. Is the docker image new as of now? I just tried building this commit with it and it works fine.

[mrakitin]

No, it wasn't updated to have conda-build >=3.17.0, however it's a good practice to have this dependency explicitly. I'll try to rebuild our image so that we have the updated version of conda-build out-of-the-box.