Rely on "run exports" to install zlib at runtime

[jdblischak]

I recently had a colleague get confused about "run exports" after reading the documentation page on migrating to conda build v3. I think the explanation of how to handle the zlib requirement is confusing.

Currently the text contains this explanation:

bioconda-docs/source/contributor/cb3.rst

Lines 203 to 209 in 6366416

	The second major advance in conda-build 3 is the the concept of "run exports".
	The idea here is to specify that any time a dependency (``zlib``, in our running example)
	is used as a build dependency, it should also be automatically be installed as
	a run dependency without having to explicitly add it as such in the recipe.
	This specification is done in the ``zlib`` recipe itself (which is hosted by
	conda-forge), so in general bioconda collaborators can just add ``zlib`` as
	a build dependency.

The first thing I noticed is that zlib shouldn't be added to build, since it doesn't have "strong run exports". Instead it should be added to host.

Then in the example recipe, zlib is still added in run, which contradicts the paragraph about run exports

bioconda-docs/source/contributor/cb3.rst

Lines 246 to 251 in 6366416

	host:
	- python
	- zlib
	run:
	- python
	- zlib

I suspect that part of the confusion is that zlib is itself a complex example of "run exports", which I will explain below. For documentation purposes, it might be better to simply include a more straight-forward example, and then tackle zlib since it is such a common requirement for bioconda recipes.

Here is why I think zlib is a complex example to demonstrate the concept of "run exports"

• This documentation was written way back in 2018 (bioconda/bioconda-utils@d67288b)
• In 2018, the zlib "run exports" was correct, and thus zlib should not have appeared in the run requirements in this example (https://github.com/conda-forge/zlib-feedstock/blob/b8a90e7d70a09d6672b559ac3bd027cc1f3019fa/recipe/meta.yaml#L17)
• However, the zlib-feedstock recipe has undergone some rearrangements, and at some point the "run exports" was broken and had to be added back in 2021 (Fix run_exports conda-forge/zlib-feedstock#53)
• The "run exports" doesn't follow the canonical pattern of exporting the same package at runtime. Instead it exports libzlib (https://github.com/conda-forge/zlib-feedstock/blob/649df646085a23f9f580418c1ca01249e151857b/recipe/meta.yaml#L77)
• It's my understanding that only exporting libzlib should be sufficient for software that properly links against libzlib. However, some software out in the wild still requires the files provided by zlib at runtime. Therefore, there are some cases where you would need to put zlib in both host and run. See this Issue for the details: Something broken with zlib vs. libzlib files (or run-export) conda-forge/zlib-feedstock#65

Lastly, I'd note that there are many existing bioconda recipes that include zlib in the run requirements (https://github.com/search?q=repo%3Abioconda%2Fbioconda-recipes+zlib+language%3AYAML&type=code&l=YAML). If it's a bioconda convention to include zlib in run (perhaps to avoid the complexity I noted above), then I definitely think a different example should be chosen to demonstrate the concept of "run exports"

[jdblischak]

Lastly, I'd note that there are many existing bioconda recipes that include zlib in the run requirements

Note that 3 weeks ago, @jmarshall removed zlib (and other shared libraries that use run exports) from the htslib recipe in PR bioconda/bioconda-recipes#44774. I think this is the correct approach and similar recipe updates should be encouraged

a different example should be chosen to demonstrate the concept of "run exports"

And I still think a different or additional example would be useful. The zlib "run exports" continues to behave unexpectedly at times, as noted in bioconda/bioconda-recipes#44774 (comment)

[jdblischak]

Bump. I think these minor changes would clarify the current bioconda documentation on run exports. Any potential larger changes can wait for a future PR

[jdblischak]

Closing because the file source/contributor/cb3.rst was deleted in the recently merged #16 ☹️

[daler]

It was migrated to https://bioconda.github.io/faqs.html#what-are-the-host-and-build-sections-of-a-recipe (and here in the source), is there anything that should be changed in that section?

[jdblischak]

It was migrated to https://bioconda.github.io/faqs.html#what-are-the-host-and-build-sections-of-a-recipe (and here in the source), is there anything that should be changed in that section?

I like the explanation of build/host/run in that section. The only missing aspect I would add is that the shared libraries in host don't need to be duplicated in run because of "run exports".

However, the current guidance for writing recipes still recommends explicitly adding zlib to the run requirements even though this shouldn't be necessary in most cases due to the zlib recipe's use of run exports. Furthermore, one the example recipes, samtools, no longer includes the unnecessary zlib. Also, the distinction between build and host isn't as clear here as it is in the FAQ entry you shared.

bioconda-docs/source/contributor/guidelines.rst

Lines 419 to 444 in d8b5591

	Conda distinguishes between dependencies needed for building (the ``build`` section),
	and dependencies needed during build time (the ``host`` section).
	For example, the following
	.. code:: yaml
	requirements:
	build:
	- {{ compiler('c') }}
	host:
	- zlib
	run:
	- zlib
	specifies that a recipe needs the C compiler to build, and zlib present during
	building and running.
	For two examples see:
	- example requiring ``autoconf``: `srprism
	<https://github.com/bioconda/bioconda-recipes/tree/master/recipes/srprism>`_
	- simple example: `samtools
	<https://github.com/bioconda/bioconda-recipes/tree/master/recipes/samtools>`_
	If the package uses ``zlib``, then please see the :ref:`troubleshooting section on zlib <zlib>`.

If you're interested, I'm happy to send PR(s) to update either of the above points.

[daler]

Ah, yes, PR/PRs would be wonderful -- thanks