Update official documentation to list recommended container base images

[stevekm]

As per the Slack discussion here I discovered that Nextflow containers meant for usage on AWS Batch should not be built on an Alpine Linux base image;

#1865

some more details here;

aws/aws-cli#4971

Technically this is not a Nextflow issue but instead an AWS Batch issue it seems with the relation between AWS and Alpine Linux, however, considering that we already have some official docs for Nextflow about Container usage;

https://nextflow.io/docs/latest/container.html

maybe we can update these docs to include a couple recommended base images to use for packaging tools and scripts inside Docker containers to run in a Nextflow pipeline.

In my case, I finally settled on Ubuntu 22.04 ( https://hub.docker.com/_/ubuntu/tags ) as a base image for my custom tools since it seems to work with everything out of the box (has all required reporting tools along with ps and bash).

Having a quick note recommending this or some other container that "just works" would be really helpful!

[bentsherman]

I think the differentiator is whether the base OS image uses glibc or musl. Alpine uses musl, but I believe the AWS CLI expects glibc, so they don't work together. If you use Fusion instead of the AWS CLI, then alpine should work because Fusion doesn't depend on either.

[pditommaso]

Worth to add a note in any case

[bentsherman]

There is a note here about requirements for containers in general (i.e. bash and ps). Would be good to either augment that note or add a complementary note to the AWS page about the AWS CLI requirements.

[stevekm]

the part that really confused me for a long time was that aws CLI is not actually installed into the containers. Its part of the AMI which is used for the host environment where the container runs (I think?), so it is really not intuitive to consider that the AWS binary installed outside the container would cease to function like this. Unless my understanding of how the AWS Batch jobs get executed for Nextflow is not accurate?

Worth noting that there's a mention of this error here;
https://apeltzer.github.io/post/01-aws-nfcore/

I have configured nextflow according to your suggestion but every job fails with Essential container in task exited - how can I fix that?

Open CloudWatch and look at the log for the failed task. If the error is aws command not found or bash: /home/ec2-user/miniconda/bin/aws: No such file or directory’ then make sure aws is installed and there are no typos in your nextflow.config.

though this did not resolve the issue either

[bentsherman]

I am working on a PR to clarify these requirements. However the issue between the AWS CLI and Alpine seems to be resolved: aws/aws-cli#4971 (comment)

So it might be TMI to have a note that "AWS CLI and Alpine don't work together in these very specific cases". Also I am hesitant to recommended a particular image when really most base images will work