AWS configuration - can't find awscli #1064
Comments
|
Hi :) maybe try removing the latter. |
|
Hmm I'm confused because the instructions say to add this line: And there's definitely no file named I'll try removing the local executor and see if that helps |
|
Maybe also check your |
Notice that the file is |
|
Closing because there's no more feedback. |
|
Hello @olgabot I am also encountering same issue with awscli but failed to solve this problem. If you have solved this problem can you please guide me through the steps you have taken to solve this issue. |
|
Doesn't work for me as well. I configured a custom AMI with pre-installed AWSCLI due to official instructions. Event with
when I login to EC2 machine created by AWS batch, I can see that |
|
@Volodymyr128 Did you find a resolution for this? I have the same problem. |
|
Bump. Same issue. |
|
I'm not sure whether @olgabot and I experienced the same issue but we got the same error message and I ended up here. Here's how I solved my issue. BackgroundWhen the Problem 1 - The container cannot execute because it cannot find the AWS binary.This manifested for me in Batch Job CloudWatch logs like the following.
The problem here is the process within the container is looking for a binary at You can check for unexpected mounts and volumes by examining the AWS Batch Jobs that are created when you try and run Nextflow. I got past this by using the Amazon Linux 2022 EC2 Optimized AMI for my Batch Compute Environment and specifying the AWS CLI path in the ec2 instance in the Problem 2 - Odd compatibility errors
Unfortunately passing I fixed this by installing the AWS CLI in its own directory structure and passing that. Other Problems
Solution
# Download CLI installer
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
# Download unzip utility
yum install unzip
# Unzip the installer into a temporary directory
unzip awscliv2.zip -d _tmp_aws_cli_installer
# cd into the installer directory. I'm not sure this is necessary, but I ran into some weird symlink issues without this.
# I was probably just tired
cd _tmp_aws_cli_installer
# Run the install. Keep the binary and its dependencies in /nextflow_tools/bin
./aws/install -i /nextflow_tools/bin/aws_cli_files -b /nextflow_tools/bin/
# Check that the installation worked
/nextflow_tools/bin/aws
# Remove the tmp install folder
cd && rm -r _tmp_aws_cli_installer
ConclusionThe EDIT: Edit for clarity. |
|
@Volodymyr128 how do you "login to EC2 machine created by AWS batch"? I can easily ssh into the AMI I indicated in the "Image ID override" section of my compute environment but how to confirm these are the same machines:
|
|
MAKE SURE YOU RUN |
@HarryMWinters Have you by chance gotten this solution to work with a docker container running on batch? I've built the following docker image following your solution because I also keep getting # use the upstream nextflow container as a base image
ARG VERSION=latest
FROM nextflow/nextflow:${VERSION} AS build
FROM amazonlinux:2023 AS final
COPY --from=build /usr/local/bin/nextflow /usr/bin/nextflow
RUN yum update -y --allowerasing \
&& yum install -y --allowerasing \
curl \
hostname \
java \
unzip \
&& yum clean -y all
RUN rm -rf /var/cache/yum
RUN mkdir -p /nextflow_tools/bin
RUN curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
RUN yum install unzip
RUN unzip awscliv2.zip -d _tmp_aws_cli_installer
RUN cd _tmp_aws_cli_installer \
&& ./aws/install -i /nextflow_tools/bin/aws_cli_files -b /nextflow_tools/bin/ \
&& /nextflow_tools/bin/aws --version \
&& cd - && rm -r _tmp_aws_cli_installer
ENV JAVA_HOME /usr/lib/jvm/jre-openjdk/
# # invoke nextflow once to download dependencies
RUN nextflow -version
# # install a custom entrypoint script that handles being run within an AWS Batch Job
COPY nextflow.aws.sh /opt/bin/nextflow.aws.sh
RUN chmod +x /opt/bin/nextflow.aws.sh
WORKDIR /opt/work
ENTRYPOINT ["/opt/bin/nextflow.aws.sh"]I'm trying to run the very simple This is my batch command I'm running: aws batch submit-job \
--job-name nf-hello \
--job-queue low-priority-ec2 \
--job-definition nextflow-dev \
--container-overrides command=nextflow-io/hello \
--region us-east-2 \
--scheduling-priority-override 1 \
--share-identifier defaultI've also tried setting both |
|
Hi @ahummel25, the one thing I did that was the most helpful was a BUNCH of Hope that helps. Good luck! |
|
I see this is an old issue, but it sounds similar to #1865 aws/aws-cli#4971 where the container used inside the AWS Batch job was not compatible with the |
Hey there @ahummel25, It has been some time since I worked on this, so please forgive any lapses in memory. It's also worth mentioning that I continued encountering issues with Nextflow (core and pipelines), which led me to use Redun and code the pipeline myself. Obviously, YMMV. First, I think my solution was for AWS Batch and I don't think I needed to make a custom docker image. Just a custom Amazon Machine Image. All of the installation and path manipulation steps relate to making a custom Amazon Machine Image, which is then used when spinning up EC2 instances for the Batch Compute Environment. Batch uses the EC2 instances as docker hosts, and Nextflow mounts the AWS-CLI directory from the host into the docker container. Second, I agree with @olgabot. IMO, you can never have too many Questions:
TL;DRI made a custom Amazon Machine Image (not a Docker image) to get past my error. |
Bug report
Expected behavior and actual behavior
I expected to see the same output for the
tutorial.rfrun in AWS batch as it is in the tutorial description and as I was able to run locally.Instead, the CloudWatch logs from AWS batch show that the file
/home/ec2-user/miniconda/bin/awsdoes not exist:Steps to reproduce the problem
Here is a GitHub repo with the tutorial code and the setup. I use
make runto run this command:I followed the AWS custom AMI instructions to install the
awscli, and the executable is definitely there:Here is the nextflow config:
Program output
Nextflow log
Environment
N E X T F L O W ~ version 19.01.0Additional context
My hunch is that the
No such file or directoryerror is happening because the file doesn't exist in the docker imageubuntuwhich is running the workflow, but I thought that shouldn't matter, that the docker image and the AMI can be independent. Am I misunderstanding something?The text was updated successfully, but these errors were encountered: