test hic fail on aws

[jarekgeneg]

When we perform test hic on aws it's fail with file can't be read from s3

OS/Platform

• OS: Ubuntu 18.04.6 on AWS
• Conda version: none
• Pipeline version: try on dev and 1.11.2
• Caper version: 2.1.3

Caper configuration file
backend=aws
no-server-heartbeat=True
max-concurrent-workflows=300
max-concurrent-tasks=1000
local-out-dir=/opt/caper/local_out_dir
local-loc-dir=/opt/caper/local_loc_dir
aws-batch-arn=arn:aws:batch:eu-west-2:my_id:job-queue/caper-queue
aws-region=eu-west-2
aws-out-dir=s3://caper-hic
aws-loc-dir=s3://caper-hic/.caper_tmp
cromwell=https://storage.googleapis.com/caper-data/cromwell/cromwell-65-d16af26-SNAP.jar
db=postgresql
postgresql-db-ip=xxxx
postgresql-db-port=5432
postgresql-db-user=xxxxx
postgresql-db-password=xxxxx
postgresql-db-name=xxxxxx

Input JSON file
standard test file: tests/functional/json/test_hic.json

Error log
Started troubleshooting workflow: id=756ed754-2f3d-46e4-8323-194f21bf11fb, status=Failed
Found failures JSON object.
[
{
"causedBy": [
{
"causedBy": [
{
"causedBy": [
{
"message": "s3://s3.amazonaws.com/caper-hic/hic/756ed754-2f3d-46e4-8323-194f21bf11fb/call-get_ligation_site_regex/get_ligation_site_regex-rc.txt",
"causedBy": []
}
],
"message": "Could not read from s3://caper-hic/hic/756ed754-2f3d-46e4-8323-194f21bf11fb/call-get_ligation_site_regex/get_ligation_site_regex-rc.txt: s3://s3.amazonaws.com/caper-hic/hic/756ed754-2f3d-46e4-8323-194f21bf11fb/call-get_ligation_site_regex/get_ligation_site_regex-rc.txt"
}
],
"message": "[Attempted 1 time(s)] - IOException: Could not read from s3://caper-hic/hic/756ed754-2f3d-46e4-8323-194f21bf11fb/call-get_ligation_site_regex/get_ligation_site_regex-rc.txt: s3://s3.amazonaws.com/caper-hic/hic/756ed754-2f3d-46e4-8323-194f21bf11fb/call-get_ligation_site_regex/get_ligation_site_regex-rc.txt"
},
{
"causedBy": [
{
"causedBy": [
{
"message": "s3://s3.amazonaws.com/caper-hic/hic/756ed754-2f3d-46e4-8323-194f21bf11fb/call-normalize_assembly_name/normalize_assembly_name-rc.txt",
"causedBy": []
}
],
"message": "Could not read from s3://caper-hic/hic/756ed754-2f3d-46e4-8323-194f21bf11fb/call-normalize_assembly_name/normalize_assembly_name-rc.txt: s3://s3.amazonaws.com/caper-hic/hic/756ed754-2f3d-46e4-8323-194f21bf11fb/call-normalize_assembly_name/normalize_assembly_name-rc.txt"
}
],
"message": "[Attempted 1 time(s)] - IOException: Could not read from s3://caper-hic/hic/756ed754-2f3d-46e4-8323-194f21bf11fb/call-normalize_assembly_name/normalize_assembly_name-rc.txt: s3://s3.amazonaws.com/caper-hic/hic/756ed754-2f3d-46e4-8323-194f21bf11fb/call-normalize_assembly_name/normalize_assembly_name-rc.txt"
}
],
"message": "Workflow failed"
}
]

• Recursively finding failures in calls (tasks)...

==== NAME=hic.normalize_assembly_name, STATUS=Failed, PARENT=
SHARD_IDX=-1, RC=None, JOB_ID=74e5e04d-1af7-4a18-97e7-555fc23f58bd
START=2022-03-15T10:05:48.022Z, END=2022-03-15T10:09:36.113Z
STDOUT=s3://caper-hic/hic/756ed754-2f3d-46e4-8323-194f21bf11fb/call-normalize_assembly_name/normalize_assembly_name-stdout.log
STDERR=s3://caper-hic/hic/756ed754-2f3d-46e4-8323-194f21bf11fb/call-normalize_assembly_name/normalize_assembly_name-stderr.log

==== NAME=hic.get_ligation_site_regex, STATUS=Failed, PARENT=
SHARD_IDX=-1, RC=None, JOB_ID=32b91045-cf5f-4bca-8f07-4d266d13f97c
START=2022-03-15T10:05:48.585Z, END=2022-03-15T10:09:26.285Z
STDOUT=s3://caper-hic/hic/756ed754-2f3d-46e4-8323-194f21bf11fb/call-get_ligation_site_regex/get_ligation_site_regex-stdout.log
STDERR=s3://caper-hic/hic/756ed754-2f3d-46e4-8323-194f21bf11fb/call-get_ligation_site_regex/get_ligation_site_regex-stderr.log

Machine was provisioned with caper aws create env script

[paul-sud]

Usually such an error indicates that the job didn't even start executing on Batch. Do you have the backend log available? Looking at your Caper config, you have the following value:

aws-batch-arn=arn:aws:batch:eu-west-2:my_id:job-queue/caper-queue

The my_id is a bit odd to me, I think usually you'd put your AWS account number there.

[jarekgeneg]

Caper config - yes, there is account name or queue number - i just hide it
AWS Batch start virtual machine and just fail with error

And aout backenf log ? where I can find it ?

[jarekgeneg]

Log from backed:

cromwell_encodedcc_hic-pipeline_1_11_23f2a62da3521400809715244c3eeb9fcfce048da/default/dce5fe3c972e4392a3777d8b608e9d7d:

2022-03-15T10:08:57.007Z
/bin/bash: /var/scratch/fetch_and_run.sh: Is a directory
@ingestionTime
1647338937082
@log
076634410064:/aws/batch/job
@logstream
cromwell_encodedcc_hic-pipeline_1_11_23f2a62da3521400809715244c3eeb9fcfce048da/default/dce5fe3c972e4392a3777d8b608e9d7d
@message
/bin/bash: /var/scratch/fetch_and_run.sh: Is a directory
@timestamp
1647338937007

[paul-sud]

The backend log should be a file named something like get_ligation_site_regex.log. If you look at the workflow metadata JSON it will point you to the right file on S3, in the JSON it should be under the key calls > hic.get_ligation_site_regex > backendLogs > log. This file will contain more details about anything that went wrong on the provisioned machine before the task started executing, for instance any issue with localizing input data.

[jarekgeneg]

Question:

in metadata.json :

{
"causedBy": [{
"causedBy": [{
"message": "s3://s3.amazonaws.com/caper-hic/hic/bd74874b-ebed-4b73-8058-f484ac8695bc/call-normalize_assembly_name/normalize_assembly_name-rc.txt",
"causedBy": []
}],
"message": "Could not read from s3://caper-hic/hic/bd74874b-ebed-4b73-8058-f484ac8695bc/call-normalize_assembly_name/normalize_assembly_name-rc.txt: s3://s3.amazonaws.com/caper-hic/hic/bd74874b-ebed-4b73-8058-f484ac8695bc/call-normalize_assembly_name/normalize_assembly_name-rc.txt"
}

is uri s3://s3.amazonaws.com/caper-hic/.... is valid ?
When we search AWS docs (https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-bucket-intro.html#accessing-a-bucket-using-S3-format) AWS don't recommend this type uri which is used in hic pipeline.

On AWS we give full access to S3 for AWS BATCH and AWS EC2; while pipeline starts, directories and files are created on s3 volume

Anyway, I can't find any backed log -> it's maybe script which is run to create this files can't move files to S3 ? the same script creates stdout and stderr for process (i think ), and also I can't find this files on S3 volume

[leepc12]

I think something went wrong with Caper's AWS backend.

/bin/bash: /var/scratch/fetch_and_run.sh: Is a directory

I will test it on AWS soon and let you know but until then please try on Google Cloud Platform. It's much more stable.

[mziebagg]

Dears Thanks a lot for your help
We are Genegoggle startup and receive some free$ on AWS so we cant easy migrate to Google Cloud platform (costs)

@leepc12 In AWS we have prepared account/organization and configured vm for pipeline-hic .
So if we can accelarate we can give you acess to this test environment

If you find somethings will be great Thanks in advance

BR
Michał