CCTools Packaging: Pip vs Conda vs Tarball etc

[svandenhaute]

Many HPCs put strong limits on inode consumption. File systems become super slow when large numbers of small files are present. This is precisely the case when using conda-like environments. At the moment, this is necessary for a smooth cctools installation.

Is it an option to reconsider packaging cctools with pip? WQ/TaskVine are almost-essential additions to e.g. Parsl, but I'd like to avoid having to convert all my environments into conda environments with tens of thousands of unnecessary duplicate library files.

[svandenhaute]

essentially -- #2091

[dthain]

Hello Sander, thanks for reaching out to us. The main issue here is that pip is really designed around Python software, and CCTools consists of a fair amount of C code and other non-Python dependencies. After working with pip for some time, we decided that it just wasn't able to reliably deploy the right environment needed to deploy cctools. Conda is the only method we have found that encompasses multi-language software builds. (But in fairness, we have found situations where Conda causes its own problems.)

That said, we are willing to work with you to see if there are some other ways to make it work. For example, we do publish plain tarball builds of cctools that are much easier on the filesystem. I think these could be made to work with Parsl with minimal extra effort:
https://github.com/cooperative-computing-lab/cctools/releases

If you let us know a little more about your constraints, I think we can come up with a solution.

[svandenhaute]

I think you're right, but I'm a little unfamiliar with how I'm supposed to use them.
Are they compatible with arbitrary Python versions (let's assume cpython, >= 3.9)? Which system libraries do they rely on?

Based on the Parsl script here, unpacking them suffices, but I'm a little in the dark as to the compatibility with respect to the 'host' OS and Python version.

[svandenhaute]

Nvm, found the answer in the docs.

[dthain]

@svandenhaute if you don't mind, I'm going to move this over to a Discussion thread so we can discuss some options, and others may benefit from the answers as well.

[svandenhaute]

I've tried the main tarballs but noticed that they need the header files of readline and openssl, and these are not automatically discoverable on most HPC systems given that they're often implemented as separate modules. Am I missing something, or is this indeed the case?

[dthain]

I'm not entirely clear what procedure you are following here. Are you building cctools from a source tarball, or installing a binary tarball? Go ahead and post your exact steps and the trouble that you are encountering.

[svandenhaute]

Sorry, I should have added more details.

So I'm trying to use the binary tarballs of cctools, in particular the 7.10 ubuntu 20.04 x86_64 version. I'm running pop-os (essentially ubuntu) 22.04.
After untarring and adapting PATH and PYTHONPATH to the corresponding folders in the extracted folder, testing the package with e.g. work_queue_worker --version gives me that I'm missing libssl.so.1.1? It's a python 3.10 installation, which I think should be compatible with the python3.8 libraries in the binary.

Now that I think of it, the readline missing header actually happened when compiling cctools from source, not when trying to use the binary tarballs. (If compiling somehow makes things easier, that's also an option, but it doesn't look like it)

[svandenhaute]

#3414 actually answers my question I think... In HPC environments, libssl would also not be automatically available. Is it really a critical dependency?

[dthain]

SSL support is optional at build time. But it is used in a variety of places, particularly for integration with the global catalog server, and is an available option for securing the connection between manager and workers in both TaskVine and Work Queue. So our binary builds have it enabled by default.

If you are committed to using the binary tarball, it may be sufficient to obtain libssl.so from the corresponding ubuntu 20.04, drop it into the lib directory of the cctools install, and set LD_LIBRARY_PATH appropriately.

Or it might be more straightforward to build cctools from source with configure and make, and then it will make use of whatever libraries and other options are available in the native environment. If you don't have libopenssl present, configure should just proceed without it.

[svandenhaute]

Given that I'd like to work in as many scenarios as possible, I'll just go for an automated micromamba setup. The whole thing does contain too many files (on the order to 10k) so that's actually fine.