2024-01-11 10:48:19,853 INFO - Running with database db.url = jdbc:hsqldb:mem:8d0942a2-d2bf-41f1-8eea-fbab4e66d11f;shutdown=false;hsqldb.tx=mvcc Running Changeset: changesets/db_schema.xml::db_schema_other_table_alldb::scottfrazer Running Changeset: changesets/db_schema.xml::db_schema_symbol_table_otherdb::scottfrazer Running Changeset: changesets/db_schema.xml::db_schema_constraints::scottfrazer Running Changeset: changesets/symbol_iteration_null.xml::symbol_iteration_null::kshakir Running Changeset: changesets/wdl_and_inputs.xml::WORKFLOW_EXECUTION_AUX::mcovarr Running Changeset: changesets/wdl_and_inputs.xml::WE_AUX_CONSTRAINTS::mcovarr Running Changeset: changesets/drop_workflow_uri_and_local_command.xml::drop_workflow_uri_and_local_command::kshakir Running Changeset: changesets/local_job_allow_null.xml::local_job_allow_null::kshakir Running Changeset: changesets/symbol_iteration_not_null.xml::symbol_iteration_not_null::kshakir Running Changeset: changesets/add_unique_constraints.xml::add_unique_constraints::kshakir Running Changeset: changesets/lengthen_wdl_value.xml::LENGTHEN_WDL_VALUE::sfrazer Running Changeset: changesets/add_index_in_execution.xml::add-index-in-execution::tjeandet Running Changeset: changesets/rename_iteration_to_index.xml::rename-iteration-to-index::tjeandet Running Changeset: changesets/sge.xml::sge::scottfrazer Running Changeset: changesets/sge.xml::db_schema_constraints::scottfrazer Running Changeset: changesets/change_execution_unique_constraint.xml::change_execution_unique_constraint::tjeandet Running Changeset: changesets/rc.xml::rc::mcovarr Running Changeset: changesets/workflow_options.xml::workflow-options::sfrazer Running Changeset: changesets/jes_id_update.xml::jes_id_update::chrisl Running Changeset: changesets/optional_sge_ids.xml::optional_sge_ids::chrisl Running Changeset: changesets/add_start_end_time_in_execution.xml::add-start-end-time-in-execution::jgentry Running Changeset: changesets/rename_jes_id.xml::rename-jes-id-to-jes-run-id::sfrazer Running Changeset: changesets/add_workflow_name.xml::workflow-name::mcovarr Running Changeset: changesets/top_level_output.xml::top_level_output::chrisl Running Changeset: changesets/workflow_execution_aux_not_null.xml::workflow_execution_aux_not_null::tjeandet Running Changeset: changesets/call_result_caching.xml::call_result_caching::chrisl Running Changeset: changesets/events_table.xml::execution_event_table::chrisl Running Changeset: changesets/sync_not_null_constraints.xml::job_not_null_columns::kshakir Running Changeset: changesets/sync_not_null_constraints.xml::events_end_dt_not_null::kshakir Running Changeset: changesets/sge_job_execution_unique_key.xml::sge_job_execution_unique_key::kshakir Running Changeset: changesets/add_attempt_in_execution.xml::add-attempt-in-execution::tjeandet Running Changeset: changesets/execution_backend_info.xml::CREATE_EXECUTION_INFO::mcovarr Running Changeset: changesets/execution_backend_info.xml::add_foreign_key_constraints::mcovarr Running Changeset: changesets/execution_backend_info.xml::add_backend_type_column::mcovarr Running Changeset: changesets/execution_backend_info.xml::insert_data_from_backend_tables::mcovarr Running Changeset: changesets/execution_backend_info.xml::set_backend_type::mcovarr Running Changeset: changesets/execution_backend_info.xml::drop_job_tables::mcovarr Running Changeset: changesets/execution_backend_info.xml::execution_info_uniqueness_constraint::mcovarr Running Changeset: changesets/execution_backend_info.xml::execution_backend_type_not_null::mcovarr Running Changeset: changesets/runtime_attributes_table.xml::runtime-attributes-table::tjeandet Running Changeset: changesets/failure_table.xml::failure_table::chrisl Running Changeset: changesets/metadata_journal.xml::metadata_journal::mcovarr Running Changeset: changesets/metadata_journal.xml::metadata_workflow_index::mcovarr Running Changeset: changesets/metadata_journal.xml::metadata_job_index::mcovarr Running Changeset: changesets/metadata_journal.xml::metadata_job_and_key_index::mcovarr Running Changeset: changesets/metadata_typed_values.xml::metadata_typed_values::tjeandet Running Changeset: changesets/workflow_metadata_summary.xml::metadata_journal_id_int_to_big_int::kshakir Running Changeset: changesets/workflow_metadata_summary.xml::workflow_metadata_summary::mcovarr Running Changeset: changesets/workflow_metadata_summary.xml::workflow_metadata_uuid_idx::mcovarr Running Changeset: changesets/workflow_metadata_summary.xml::workflow_metadata_name_idx::mcovarr Running Changeset: changesets/workflow_metadata_summary.xml::workflow_metadata_status_idx::mcovarr Running Changeset: changesets/metadata_journal_subsecond_timestamp.xml::metadata_journal_subsecond_timestamp::mcovarr Running Changeset: changesets/metadata_journal_subsecond_timestamp.xml::metadata_journal_timestamp_not_null::mcovarr Running Changeset: changesets/workflow_store.xml::WORKFLOW_STORE::chrisl Running Changeset: changesets/workflow_store.xml::workflow_store_uuid_index::kshakir Running Changeset: changesets/workflow_store.xml::workflow_store_state_index::chrisl Running Changeset: changesets/backend_KV_Store.xml::BACKEND_KV_STORE::rmunshi Running Changeset: changesets/backend_KV_Store.xml::backend_KV_store_job_key_constraint::rmunshi Running Changeset: changesets/job_store.xml::JOB_STORE::chrisl Running Changeset: changesets/job_store.xml::job_store_uuid_index::chrisl Running Changeset: changesets/job_store.xml::job_store_jobkey_index::chrisl Running Changeset: changesets/callcaching.xml::call_caching_result_metainfo::chrisl Running Changeset: changesets/callcaching.xml::call_caching_result_metainfo_uniqueness::chrisl Running Changeset: changesets/callcaching.xml::call_caching_hashes::chrisl Running Changeset: changesets/callcaching.xml::call_caching_result_hashes_uniqueness::chrisl Running Changeset: changesets/callcaching.xml::call_caching_result_hashes_uniqueness_foreign_key::chrisl Running Changeset: changesets/callcaching.xml::call_caching_result_simpletons::chrisl Running Changeset: changesets/callcaching.xml::call_caching_result_simpletons_uniqueness::chrisl Running Changeset: changesets/callcaching.xml::call_caching_result_simpletons_foreign_key::chrisl Running Changeset: changesets/call_caching_allow_result_reuse_fix.xml::call_caching_fix_boolean::mcovarr Running Changeset: changesets/call_caching_allow_result_reuse_fix.xml::call_caching_reuse_default::mcovarr Running Changeset: changesets/job_store_simpletons.xml::job_store_result_simpletons::mcovarr Running Changeset: changesets/job_store_simpletons.xml::job_store_result_simpletons_uniqueness::mcovarr Running Changeset: changesets/job_store_simpletons.xml::job_store_result_simpletons_foreign_key::mcovarr Running Changeset: changesets/job_store_simpletons.xml::job_store_remove_job_output::mcovarr Running Changeset: changesets/restart_and_recover_migration.xml::WORKFLOW_STORE_MIGRATION::mcovarr Running Changeset: changesets/restart_and_recover_migration.xml::JOB_STORE_MIGRATION::mcovarr Running Changeset: changesets/restart_and_recover_migration.xml::JOB_STORE_RESULT_SIMPLETON_VALUE_NULLABLE::mcovarr Running Changeset: changesets/restart_and_recover_migration.xml::JOB_STORE_RESULT_SIMPLETON_MIGRATION::mcovarr Running Changeset: changesets/restart_and_recover_migration.xml::BACKEND_KV_STORE_MIGRATION::mcovarr Running Changeset: changesets/restart_and_recover_migration.xml::WORKFLOW_OPTIONS_RENAME_MIGRATION::mcovarr Running Changeset: changesets/summary_status_table.xml::summary_status_table::kshakir Running Changeset: changesets/summary_status_table.xml::summary_status_summarized_table_name_index::kshakir Running Changeset: changesets/standardize_column_names.xml::workflow_execution_uuid::rmunshi Running Changeset: changesets/standardize_column_names.xml::call_job_identifiers::rmunshi Running Changeset: changesets/embiggen_metadata_value.xml::entry_or_journal_existence_xor::mcovarr Running Changeset: changesets/embiggen_metadata_value.xml::embiggen_metadata_entry::mcovarr Running Changeset: changesets/embiggen_metadata_value.xml::embiggen_metadata_journal::mcovarr Running Changeset: changesets/call_caching_job_detritus.xml::call_caching_job_detritus::rmunshi Running Changeset: changesets/call_caching_job_detritus.xml::call_caching_job_detritus_key_uniqueness::rmunshi Running Changeset: changesets/call_caching_job_detritus.xml::call_caching_job_detritus_foreign_key::rmunshi Running Changeset: changesets/job_store_tinyints.xml::job_store_fix_job_successful::mcovarr Running Changeset: changesets/job_store_tinyints.xml::job_store_fix_job_retryable_failure::mcovarr Running Changeset: changesets/job_store_tinyints.xml::job_store_not_nullable_job_successful::mcovarr Running Changeset: changesets/standardize_column_names_again.xml::standardize_column_names_again::kshakir Running Changeset: changesets/rename_workflow_options_in_metadata.xml::rename_workflow_options_in_metadata::tjeandet 2024-01-11 10:48:22,424 INFO - Running migration RenameWorkflowOptionsInMetadata with a read batch size of 100000 and a write batch size of 100000 2024-01-11 10:48:22,434 INFO - [RenameWorkflowOptionsInMetadata] 100% Running Changeset: changesets/encrypt_and_clear_workflow_options.xml::encrypt_workflow_store_entry_workflow_options::kshakir Running Changeset: changesets/encrypt_and_clear_workflow_options.xml::clear_metadata_entry_workflow_options::kshakir Running Changeset: changesets/sub_workflow_store.xml::SUB_WORKFLOW_STORE_ENTRY::tjeandet Running Changeset: changesets/sub_workflow_store.xml::sub_workflow_store_uuid_index::tjeandet Running Changeset: changesets/sub_workflow_store.xml::sub_workflow_store_jobkey_index::tjeandet Running Changeset: changesets/sub_workflow_store.xml::sub_workflow_store_root_workflow_fk::tjeandet Running Changeset: changesets/workflow_store_imports_file.xml::workflow-store-imports-file::cjllanwarne Running Changeset: changesets/workflow_store_labels_file.xml::workflow-store-labels-file::cjllanwarne Running Changeset: changesets/embiggen_detritus_value.xml::embiggen_call_caching_detritus_entry::rmunshi Running Changeset: changesets/standardize_column_names_patches.xml::standardize_column_names_patches::kshakir Running Changeset: changesets/standardize_column_names_patches.xml::hsqldb_longvarchar_to_longtext::kshakir Running Changeset: changesets/nullable_lobs.xml::nullable_lobs::kshakir Running Changeset: changesets/add_attempt_in_call_caching_entry.xml::add_attempt_in_call_caching_entry::tjeandet Running Changeset: changesets/replace_empty_custom_labels.xml::custom_labels_not_null::rmunshi Running Changeset: changesets/call_caching_aggregated_hashes.xml::call_caching_aggregation_entry::tjeandet Running Changeset: changesets/call_caching_aggregated_hashes.xml::call_caching_aggregation_entry_keys_and_indexes::tjeandet Running Changeset: changesets/custom_label_entry.xml::custom_LABEL_entry::rmunshi Running Changeset: changesets/custom_label_entry.xml::WMSE_workflow_execution_uuid_foreign_key::rmunshi Running Changeset: changesets/custom_label_entry.xml::custom_label_entry_index::rmunshi Running Changeset: changesets/custom_label_entry.xml::modify label key/value data types::rmunshi Running Changeset: changesets/docker_hash_store.xml::DOCKER_HASH_STORE_ENTRY::tjeandet Running Changeset: changesets/docker_hash_store.xml::docker_hash_store_uuid_unique_constraint::tjeandet Running Changeset: changesets/workflow_store_type_and_version.xml::workflow-store-type-and-version::mcovarr Running Changeset: changesets/remove_pre_pbe_tables.xml::drop-pre-pbe-tables::mcovarr Running Changeset: changesets/move_sql_metadata_changelog.xml::move_metadata_changelog::kshakir Running Changeset: changesets/workflow_store_state_widening.xml::workflow-store-state-widening::tjeandet Running Changeset: changesets/workflow_store_restarted_column.xml::workflow-store-restarted-column::tjeandet Running Changeset: changesets/workflow_store_restarted_column.xml::update-restartable::tjeandet Running Changeset: changesets/workflow_store_workflow_root_column.xml::workflow-store-workflow-root-column::tjeandet Running Changeset: changesets/workflow_store_horizontal_db.xml::workflow-store-horizontal-db::mcovarr Running Changeset: changesets/add_workflow_url_in_workflow_store_entry.xml::add_workflow_url_in_workflow_store_entry::sshah Running Changeset: changesets/change_max_size_workflow_url.xml::change_max_size_for_workflow_url::sshah Running Changeset: changesets/docker_hash_store_add_size_column.xml::docker_hash_store_entry_add_size_column::tjeandet Running Changeset: changesets/enlarge_call_caching_hash_entry_id.xml::restore_auto_increment_call_caching_hash_entry_id::mcovarr Running Changeset: changesets/add_hog_group_in_workflow_store.xml::add_hog_group_in_workflow_store::cjllanwarne Running Changeset: changesets/resync_engine_schema.xml::resync-engine-schema::kshakir Running Changeset: changesets/enlarge_job_store_ids.xml::drop_job_store_simpleton_entry_fk::mcovarr Running Changeset: changesets/enlarge_job_store_ids.xml::enlarge_job_store_entry_id::mcovarr Running Changeset: changesets/enlarge_job_store_ids.xml::enlarge_job_store_simpleton_entry_id::mcovarr Running Changeset: changesets/enlarge_job_store_ids.xml::enlarge_job_store_simpleton_entry_fk::mcovarr Running Changeset: changesets/enlarge_job_store_ids.xml::recreate_job_store_simpleton_entry_fk::mcovarr 2024-01-11 10:48:22,662 INFO - Running with database db.url = jdbc:hsqldb:mem:a369a306-1d6b-4c4a-b5fe-aa8a21aa8020;shutdown=false;hsqldb.tx=mvcc Running Changeset: metadata_changesets/move_sql_metadata_changelog.xml::move_metadata_changelog::kshakir Running Changeset: changesets/change_max_size_label_entry.xml::change_max_size_for_key_and_value_in_label::sshah Running Changeset: metadata_changesets/delete_duplicate_custom_labels.xml::delete_duplicate_custom_labels::kshakir Running Changeset: metadata_changesets/add_submission_timestamp_metadata_summary.xml::add_submission_timestamp_in_metadata_entry::sshah Running Changeset: metadata_changesets/custom_label_entry_key_value_index.xml::custom_label_entry_key_value_index::mcovarr Running Changeset: metadata_changesets/add_parent_and_root_workflow_execution_uuid.xml::add_parent_and_root_workflow_execution_uuid::kshakir Running Changeset: metadata_changesets/add_parent_and_root_workflow_typo_fix.xml::add_parent_and_root_workflow_typo_fix::kshakir Running Changeset: metadata_changesets/metadata_index_removals.xml::metadata_index_removals::mcovarr Running Changeset: metadata_changesets/add_metadata_archive_status.xml::add_metadata_archive_status::cjllanwarne Running Changeset: metadata_changesets/add_metadata_archive_status.xml::metadata_archive_status_index::cjllanwarne Running Changeset: metadata_changesets/summarization_queue_table.xml::summary_queue_table::gsterin Running Changeset: metadata_changesets/summarization_queue_table.xml::summary_queue_initialization::gsterin Running Changeset: metadata_changesets/summarization_queue_table_add_primary_key.xml::summary_queue_add_primary_key::gsterin Running Changeset: metadata_changesets/remove_non_summarizable_metadata_from_queue.xml::delete_non_summarizable_metadata_from_queue::mcovarr Running Changeset: metadata_changesets/update_metadata_archive_index.xml::update_old_metadata_archive_status_index::cjllanwarne Running Changeset: metadata_changesets/reset_archive_statuses_to_null.xml::reset_archive_statuses_to_null::cjllanwarne 2024-01-11 10:48:23,294 INFO - Slf4jLogger started 2024-01-11 10:48:23,687 cromwell-system-akka.dispatchers.engine-dispatcher-4 INFO - Workflow heartbeat configuration: { "cromwellId" : "cromid-8596e4a", "heartbeatInterval" : "2 minutes", "ttl" : "10 minutes", "failureShutdownDuration" : "5 minutes", "writeBatchSize" : 10000, "writeThreshold" : 10000 } 2024-01-11 10:48:23,778 cromwell-system-akka.dispatchers.service-dispatcher-10 INFO - Metadata summary refreshing every 1 second. 2024-01-11 10:48:23,780 cromwell-system-akka.dispatchers.service-dispatcher-10 INFO - No metadata archiver defined in config 2024-01-11 10:48:23,781 cromwell-system-akka.dispatchers.service-dispatcher-10 INFO - No metadata deleter defined in config 2024-01-11 10:48:23,789 cromwell-system-akka.actor.default-dispatcher-2 INFO - KvWriteActor configured to flush with batch size 200 and process rate 5 seconds. 2024-01-11 10:48:23,791 cromwell-system-akka.dispatchers.service-dispatcher-9 INFO - WriteMetadataActor configured to flush with batch size 200 and process rate 5 seconds. 2024-01-11 10:48:23,837 cromwell-system-akka.dispatchers.engine-dispatcher-39 INFO - CallCacheWriteActor configured to flush with batch size 100 and process rate 3 seconds. 2024-01-11 10:48:23,915 cromwell-system-akka.dispatchers.engine-dispatcher-48 INFO - JobRestartCheckTokenDispenser - Distribution rate: 50 per 1 seconds. 2024-01-11 10:48:23,935 cromwell-system-akka.dispatchers.engine-dispatcher-48 INFO - JobExecutionTokenDispenser - Distribution rate: 1 per 2 seconds. 2024-01-11 10:48:23,993 cromwell-system-akka.dispatchers.engine-dispatcher-4 INFO - SingleWorkflowRunnerActor: Version 82 2024-01-11 10:48:24,000 cromwell-system-akka.dispatchers.engine-dispatcher-4 INFO - SingleWorkflowRunnerActor: Submitting workflow 2024-01-11 10:48:24,145 cromwell-system-akka.dispatchers.api-dispatcher-50 INFO - Unspecified type (Unspecified version) workflow 067c7f61-ef2a-4b2b-b54f-bb1bce5217af submitted 2024-01-11 10:48:24,161 cromwell-system-akka.dispatchers.engine-dispatcher-39 INFO - SingleWorkflowRunnerActor: Workflow submitted UUID(067c7f61-ef2a-4b2b-b54f-bb1bce5217af) 2024-01-11 10:48:24,175 cromwell-system-akka.dispatchers.engine-dispatcher-39 INFO - 1 new workflows fetched by cromid-8596e4a: 067c7f61-ef2a-4b2b-b54f-bb1bce5217af 2024-01-11 10:48:24,179 cromwell-system-akka.dispatchers.engine-dispatcher-39 INFO - WorkflowManagerActor: Starting workflow UUID(067c7f61-ef2a-4b2b-b54f-bb1bce5217af) 2024-01-11 10:48:24,184 cromwell-system-akka.dispatchers.engine-dispatcher-39 INFO - WorkflowManagerActor: Successfully started WorkflowActor-067c7f61-ef2a-4b2b-b54f-bb1bce5217af 2024-01-11 10:48:24,184 cromwell-system-akka.dispatchers.engine-dispatcher-39 INFO - Retrieved 1 workflows from the WorkflowStoreActor 2024-01-11 10:48:24,200 cromwell-system-akka.dispatchers.engine-dispatcher-4 INFO - WorkflowStoreHeartbeatWriteActor configured to flush with batch size 10000 and process rate 2 minutes. 2024-01-11 10:48:24,261 cromwell-system-akka.dispatchers.engine-dispatcher-31 INFO - MaterializeWorkflowDescriptorActor [UUID(067c7f61)]: Parsing workflow as WDL 1.0 2024-01-11 10:48:25,915 cromwell-system-akka.dispatchers.engine-dispatcher-31 INFO - MaterializeWorkflowDescriptorActor [UUID(067c7f61)]: Call-to-Backend assignments: chip.gc_bias -> Local, chip.error_use_bowtie2_local_mode_for_non_bowtie2 -> Local, chip.align_ctl -> Local, chip.error_wrong_aligner -> Local, chip.pool_ta -> Local, chip.error_use_bwa_mem_for_non_bwa -> Local, chip.xcor -> Local, chip.subsample_ctl -> Local, chip.error_ctl_fastq_input_required_for_control_mode -> Local, chip.bam2ta_ctl -> Local, chip.overlap_pr -> Local, chip.error_subsample_pooled_control_with_mixed_endedness -> Local, chip.overlap_ppr -> Local, chip.pool_ta_ctl -> Local, chip.filter -> Local, chip.call_peak -> Local, chip.call_peak_ppr2 -> Local, chip.idr -> Local, chip.error_control_required -> Local, chip.count_signal_track -> Local, chip.call_peak_pr1 -> Local, chip.jsd -> Local, chip.filter_ctl -> Local, chip.subsample_ctl_pooled -> Local, chip.bam2ta_no_dedup -> Local, chip.align -> Local, chip.call_peak_pr2 -> Local, chip.reproducibility_idr -> Local, chip.qc_report -> Local, chip.overlap -> Local, chip.bam2ta_no_dedup_R1 -> Local, chip.pool_blacklist -> Local, chip.pool_ta_pr2 -> Local, chip.error_custom_aligner -> Local, chip.call_peak_pooled -> Local, chip.reproducibility_overlap -> Local, chip.macs2_signal_track_pooled -> Local, chip.choose_ctl -> Local, chip.filter_no_dedup -> Local, chip.idr_ppr -> Local, chip.read_genome_tsv -> Local, chip.error_ctl_input_defined_in_control_mode -> Local, chip.bam2ta -> Local, chip.idr_pr -> Local, chip.spr -> Local, chip.macs2_signal_track -> Local, chip.align_R1 -> Local, chip.call_peak_ppr1 -> Local, chip.count_signal_track_pooled -> Local, chip.filter_R1 -> Local, chip.error_input_data -> Local, chip.fraglen_mean -> Local, chip.pool_ta_pr1 -> Local 2024-01-11 10:48:26,071 cromwell-system-akka.dispatchers.backend-dispatcher-68 WARN - Local [UUID(067c7f61)]: Key/s [disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2024-01-11 10:48:26,074 cromwell-system-akka.dispatchers.backend-dispatcher-68 WARN - Local [UUID(067c7f61)]: Key/s [disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2024-01-11 10:48:26,074 cromwell-system-akka.dispatchers.backend-dispatcher-68 WARN - Local [UUID(067c7f61)]: Key/s [preemptible, disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2024-01-11 10:48:26,074 cromwell-system-akka.dispatchers.backend-dispatcher-68 WARN - Local [UUID(067c7f61)]: Key/s [disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2024-01-11 10:48:26,074 cromwell-system-akka.dispatchers.backend-dispatcher-68 WARN - Local [UUID(067c7f61)]: Key/s [disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2024-01-11 10:48:26,075 cromwell-system-akka.dispatchers.backend-dispatcher-68 WARN - Local [UUID(067c7f61)]: Key/s [disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2024-01-11 10:48:26,075 cromwell-system-akka.dispatchers.backend-dispatcher-68 WARN - Local [UUID(067c7f61)]: Key/s [disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2024-01-11 10:48:26,075 cromwell-system-akka.dispatchers.backend-dispatcher-68 WARN - Local [UUID(067c7f61)]: Key/s [disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2024-01-11 10:48:26,075 cromwell-system-akka.dispatchers.backend-dispatcher-68 WARN - Local [UUID(067c7f61)]: Key/s [disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2024-01-11 10:48:26,075 cromwell-system-akka.dispatchers.backend-dispatcher-68 WARN - Local [UUID(067c7f61)]: Key/s [disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2024-01-11 10:48:26,075 cromwell-system-akka.dispatchers.backend-dispatcher-68 WARN - Local [UUID(067c7f61)]: Key/s [disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2024-01-11 10:48:26,075 cromwell-system-akka.dispatchers.backend-dispatcher-68 WARN - Local [UUID(067c7f61)]: Key/s [disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2024-01-11 10:48:26,075 cromwell-system-akka.dispatchers.backend-dispatcher-68 WARN - Local [UUID(067c7f61)]: Key/s [disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2024-01-11 10:48:26,075 cromwell-system-akka.dispatchers.backend-dispatcher-68 WARN - Local [UUID(067c7f61)]: Key/s [disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2024-01-11 10:48:26,075 cromwell-system-akka.dispatchers.backend-dispatcher-68 WARN - Local [UUID(067c7f61)]: Key/s [disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2024-01-11 10:48:26,075 cromwell-system-akka.dispatchers.backend-dispatcher-68 WARN - Local [UUID(067c7f61)]: Key/s [preemptible, disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2024-01-11 10:48:26,075 cromwell-system-akka.dispatchers.backend-dispatcher-68 WARN - Local [UUID(067c7f61)]: Key/s [preemptible, disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2024-01-11 10:48:26,075 cromwell-system-akka.dispatchers.backend-dispatcher-68 WARN - Local [UUID(067c7f61)]: Key/s [disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2024-01-11 10:48:26,075 cromwell-system-akka.dispatchers.backend-dispatcher-68 WARN - Local [UUID(067c7f61)]: Key/s [disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2024-01-11 10:48:26,075 cromwell-system-akka.dispatchers.backend-dispatcher-68 WARN - Local [UUID(067c7f61)]: Key/s [disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2024-01-11 10:48:26,075 cromwell-system-akka.dispatchers.backend-dispatcher-68 WARN - Local [UUID(067c7f61)]: Key/s [preemptible, disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2024-01-11 10:48:26,075 cromwell-system-akka.dispatchers.backend-dispatcher-68 WARN - Local [UUID(067c7f61)]: Key/s [disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2024-01-11 10:48:26,075 cromwell-system-akka.dispatchers.backend-dispatcher-68 WARN - Local [UUID(067c7f61)]: Key/s [disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2024-01-11 10:48:26,075 cromwell-system-akka.dispatchers.backend-dispatcher-68 WARN - Local [UUID(067c7f61)]: Key/s [disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2024-01-11 10:48:26,075 cromwell-system-akka.dispatchers.backend-dispatcher-68 WARN - Local [UUID(067c7f61)]: Key/s [disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2024-01-11 10:48:26,075 cromwell-system-akka.dispatchers.backend-dispatcher-68 WARN - Local [UUID(067c7f61)]: Key/s [preemptible, disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2024-01-11 10:48:26,075 cromwell-system-akka.dispatchers.backend-dispatcher-68 WARN - Local [UUID(067c7f61)]: Key/s [preemptible, disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2024-01-11 10:48:26,075 cromwell-system-akka.dispatchers.backend-dispatcher-68 WARN - Local [UUID(067c7f61)]: Key/s [disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2024-01-11 10:48:26,076 cromwell-system-akka.dispatchers.backend-dispatcher-68 WARN - Local [UUID(067c7f61)]: Key/s [disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2024-01-11 10:48:26,076 cromwell-system-akka.dispatchers.backend-dispatcher-68 WARN - Local [UUID(067c7f61)]: Key/s [disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2024-01-11 10:48:26,076 cromwell-system-akka.dispatchers.backend-dispatcher-68 WARN - Local [UUID(067c7f61)]: Key/s [disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2024-01-11 10:48:26,076 cromwell-system-akka.dispatchers.backend-dispatcher-68 WARN - Local [UUID(067c7f61)]: Key/s [disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2024-01-11 10:48:26,076 cromwell-system-akka.dispatchers.backend-dispatcher-68 WARN - Local [UUID(067c7f61)]: Key/s [disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2024-01-11 10:48:26,076 cromwell-system-akka.dispatchers.backend-dispatcher-68 WARN - Local [UUID(067c7f61)]: Key/s [disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2024-01-11 10:48:26,076 cromwell-system-akka.dispatchers.backend-dispatcher-68 WARN - Local [UUID(067c7f61)]: Key/s [preemptible, disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2024-01-11 10:48:26,076 cromwell-system-akka.dispatchers.backend-dispatcher-68 WARN - Local [UUID(067c7f61)]: Key/s [disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2024-01-11 10:48:26,076 cromwell-system-akka.dispatchers.backend-dispatcher-68 WARN - Local [UUID(067c7f61)]: Key/s [preemptible, disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2024-01-11 10:48:26,076 cromwell-system-akka.dispatchers.backend-dispatcher-68 WARN - Local [UUID(067c7f61)]: Key/s [disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2024-01-11 10:48:26,076 cromwell-system-akka.dispatchers.backend-dispatcher-68 WARN - Local [UUID(067c7f61)]: Key/s [disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2024-01-11 10:48:26,076 cromwell-system-akka.dispatchers.backend-dispatcher-68 WARN - Local [UUID(067c7f61)]: Key/s [disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2024-01-11 10:48:26,076 cromwell-system-akka.dispatchers.backend-dispatcher-68 WARN - Local [UUID(067c7f61)]: Key/s [disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2024-01-11 10:48:26,076 cromwell-system-akka.dispatchers.backend-dispatcher-68 WARN - Local [UUID(067c7f61)]: Key/s [disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2024-01-11 10:48:26,076 cromwell-system-akka.dispatchers.backend-dispatcher-68 WARN - Local [UUID(067c7f61)]: Key/s [disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2024-01-11 10:48:26,076 cromwell-system-akka.dispatchers.backend-dispatcher-68 WARN - Local [UUID(067c7f61)]: Key/s [disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2024-01-11 10:48:26,076 cromwell-system-akka.dispatchers.backend-dispatcher-68 WARN - Local [UUID(067c7f61)]: Key/s [disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2024-01-11 10:48:26,076 cromwell-system-akka.dispatchers.backend-dispatcher-68 WARN - Local [UUID(067c7f61)]: Key/s [preemptible, disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2024-01-11 10:48:26,076 cromwell-system-akka.dispatchers.backend-dispatcher-68 WARN - Local [UUID(067c7f61)]: Key/s [preemptible, disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2024-01-11 10:48:26,076 cromwell-system-akka.dispatchers.backend-dispatcher-68 WARN - Local [UUID(067c7f61)]: Key/s [preemptible, disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2024-01-11 10:48:26,076 cromwell-system-akka.dispatchers.backend-dispatcher-68 WARN - Local [UUID(067c7f61)]: Key/s [disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2024-01-11 10:48:26,076 cromwell-system-akka.dispatchers.backend-dispatcher-68 WARN - Local [UUID(067c7f61)]: Key/s [disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2024-01-11 10:48:26,076 cromwell-system-akka.dispatchers.backend-dispatcher-68 WARN - Local [UUID(067c7f61)]: Key/s [disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2024-01-11 10:48:26,077 cromwell-system-akka.dispatchers.backend-dispatcher-68 WARN - Local [UUID(067c7f61)]: Key/s [disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2024-01-11 10:48:26,077 cromwell-system-akka.dispatchers.backend-dispatcher-68 WARN - Local [UUID(067c7f61)]: Key/s [disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2024-01-11 10:48:28,924 cromwell-system-akka.dispatchers.engine-dispatcher-45 INFO - Not triggering log of restart checking token queue status. Effective log interval = None 2024-01-11 10:48:28,944 cromwell-system-akka.dispatchers.engine-dispatcher-45 INFO - Not triggering log of execution token queue status. Effective log interval = None 2024-01-11 10:48:30,256 cromwell-system-akka.dispatchers.engine-dispatcher-47 INFO - WorkflowExecutionActor-067c7f61-ef2a-4b2b-b54f-bb1bce5217af [UUID(067c7f61)]: Starting chip.read_genome_tsv 2024-01-11 10:48:31,960 cromwell-system-akka.dispatchers.engine-dispatcher-47 INFO - Assigned new job execution tokens to the following groups: 067c7f61: 1 2024-01-11 10:48:32,450 INFO - Request threw an exception on attempt #1. Retrying after 776 milliseconds org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 10:48:33,234 INFO - Request threw an exception on attempt #2. Retrying after 2766 milliseconds org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 10:48:36,004 INFO - Request threw an exception on attempt #3. Retrying after 390 milliseconds org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 10:48:36,405 INFO - Request method=GET uri=https://auth.docker.io/token?service=registry.docker.io&scope=repository:encodedcc/chip-seq-pipeline:pull headers= threw an exception on attempt #4. Giving up. org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 10:48:36,412 cromwell-system-akka.dispatchers.engine-dispatcher-32 WARN - BackendPreparationActor_for_067c7f61:chip.read_genome_tsv:-1:1 [UUID(067c7f61)]: Docker lookup failed java.lang.Exception: Failed to get docker hash for encodedcc/chip-seq-pipeline:v2.2.1 Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at cromwell.engine.workflow.WorkflowDockerLookupActor.cromwell$engine$workflow$WorkflowDockerLookupActor$$handleLookupFailure(WorkflowDockerLookupActor.scala:222) at cromwell.engine.workflow.WorkflowDockerLookupActor$$anonfun$3.applyOrElse(WorkflowDockerLookupActor.scala:88) at cromwell.engine.workflow.WorkflowDockerLookupActor$$anonfun$3.applyOrElse(WorkflowDockerLookupActor.scala:73) at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35) at akka.actor.FSM.processEvent(FSM.scala:707) at akka.actor.FSM.processEvent$(FSM.scala:704) at cromwell.engine.workflow.WorkflowDockerLookupActor.akka$actor$LoggingFSM$$super$processEvent(WorkflowDockerLookupActor.scala:40) at akka.actor.LoggingFSM.processEvent(FSM.scala:847) at akka.actor.LoggingFSM.processEvent$(FSM.scala:829) at cromwell.engine.workflow.WorkflowDockerLookupActor.processEvent(WorkflowDockerLookupActor.scala:40) at akka.actor.FSM.akka$actor$FSM$$processMsg(FSM.scala:701) at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:695) at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35) at cromwell.docker.DockerClientHelper$$anonfun$dockerResponseReceive$1.applyOrElse(DockerClientHelper.scala:16) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:269) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:270) at akka.actor.Actor.aroundReceive(Actor.scala:539) at akka.actor.Actor.aroundReceive$(Actor.scala:537) at cromwell.engine.workflow.WorkflowDockerLookupActor.aroundReceive(WorkflowDockerLookupActor.scala:40) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:614) at akka.actor.ActorCell.invoke(ActorCell.scala:583) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:268) at akka.dispatch.Mailbox.run(Mailbox.scala:229) at akka.dispatch.Mailbox.exec(Mailbox.scala:241) at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) 2024-01-11 10:48:36,463 cromwell-system-akka.dispatchers.engine-dispatcher-32 INFO - BT-322 067c7f61:chip.read_genome_tsv:-1:1 is not eligible for call caching 2024-01-11 10:48:36,463 cromwell-system-akka.dispatchers.engine-dispatcher-32 WARN - BT-322 067c7f61:chip.read_genome_tsv:-1:1 disabling call caching due to error 2024-01-11 10:48:36,477 cromwell-system-akka.dispatchers.backend-dispatcher-75 WARN - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.read_genome_tsv:NA:1]: Unrecognized runtime attribute keys: disks, time, memory, cpu 2024-01-11 10:48:36,548 cromwell-system-akka.dispatchers.backend-dispatcher-75 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.read_genome_tsv:NA:1]: `echo "$(basename /cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-read_genome_tsv/inputs/977218590/hg19.tsv)" > genome_name # create empty files for all entries touch ref_fa bowtie2_idx_tar bwa_idx_tar chrsz gensz blacklist blacklist2 touch mito_chr_name touch regex_bfilt_peak_chr_name python < /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-read_genome_tsv/execution/docker_cid else # remap paths between inside and outside of a docker container shopt -s nullglob sed -i 's#/cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-read_genome_tsv#/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-read_genome_tsv#g' /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-read_genome_tsv/execution/script `dirname /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-read_genome_tsv/execution/script`/write_*.tmp if [ 'true' == 'true' ] && [ 'singularity' == 'conda' ] || \ [ 'true' == 'false' ] && [ 'true' == 'true' ] && [ ! -z 'encd-chip' ] then conda run --name=encd-chip /bin/bash /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-read_genome_tsv/execution/script & echo $! > /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-read_genome_tsv/execution/docker_cid else /bin/bash /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-read_genome_tsv/execution/script & echo $! > /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-read_genome_tsv/execution/docker_cid fi fi touch /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-read_genome_tsv/execution/docker_cid.not_docker wait `cat /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-read_genome_tsv/execution/docker_cid` rc=`echo $?` fi exit $rc 2024-01-11 10:48:38,813 cromwell-system-akka.dispatchers.backend-dispatcher-75 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.read_genome_tsv:NA:1]: job id: 1785471 2024-01-11 10:48:38,816 cromwell-system-akka.dispatchers.backend-dispatcher-74 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.read_genome_tsv:NA:1]: Status change from - to WaitingForReturnCode 2024-01-11 10:48:46,335 cromwell-system-akka.dispatchers.backend-dispatcher-74 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.read_genome_tsv:NA:1]: Status change from WaitingForReturnCode to Done 2024-01-11 10:48:49,655 cromwell-system-akka.dispatchers.engine-dispatcher-45 INFO - WorkflowExecutionActor-067c7f61-ef2a-4b2b-b54f-bb1bce5217af [UUID(067c7f61)]: Starting chip.bam2ta (2 shards), chip.bam2ta_ctl (2 shards) 2024-01-11 10:48:49,954 cromwell-system-akka.dispatchers.engine-dispatcher-45 INFO - Assigned new job execution tokens to the following groups: 067c7f61: 1 2024-01-11 10:48:49,964 INFO - Request threw an exception on attempt #1. Retrying after 381 milliseconds org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 10:48:50,347 INFO - Request threw an exception on attempt #2. Retrying after 711 milliseconds org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 10:48:51,061 INFO - Request threw an exception on attempt #3. Retrying after 5716 milliseconds org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 10:48:51,696 cromwell-system-akka.dispatchers.engine-dispatcher-35 INFO - WorkflowExecutionActor-067c7f61-ef2a-4b2b-b54f-bb1bce5217af [UUID(067c7f61)]: Starting chip.gc_bias (2 shards) 2024-01-11 10:48:56,780 INFO - Request method=GET uri=https://auth.docker.io/token?service=registry.docker.io&scope=repository:encodedcc/chip-seq-pipeline:pull headers= threw an exception on attempt #4. Giving up. org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 10:48:56,782 cromwell-system-akka.dispatchers.engine-dispatcher-34 WARN - BackendPreparationActor_for_067c7f61:chip.bam2ta_ctl:1:1 [UUID(067c7f61)]: Docker lookup failed java.lang.Exception: Failed to get docker hash for encodedcc/chip-seq-pipeline:v2.2.1 Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at cromwell.engine.workflow.WorkflowDockerLookupActor.cromwell$engine$workflow$WorkflowDockerLookupActor$$handleLookupFailure(WorkflowDockerLookupActor.scala:222) at cromwell.engine.workflow.WorkflowDockerLookupActor$$anonfun$3.applyOrElse(WorkflowDockerLookupActor.scala:88) at cromwell.engine.workflow.WorkflowDockerLookupActor$$anonfun$3.applyOrElse(WorkflowDockerLookupActor.scala:73) at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35) at akka.actor.FSM.processEvent(FSM.scala:707) at akka.actor.FSM.processEvent$(FSM.scala:704) at cromwell.engine.workflow.WorkflowDockerLookupActor.akka$actor$LoggingFSM$$super$processEvent(WorkflowDockerLookupActor.scala:40) at akka.actor.LoggingFSM.processEvent(FSM.scala:847) at akka.actor.LoggingFSM.processEvent$(FSM.scala:829) at cromwell.engine.workflow.WorkflowDockerLookupActor.processEvent(WorkflowDockerLookupActor.scala:40) at akka.actor.FSM.akka$actor$FSM$$processMsg(FSM.scala:701) at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:695) at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35) at cromwell.docker.DockerClientHelper$$anonfun$dockerResponseReceive$1.applyOrElse(DockerClientHelper.scala:16) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:269) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:270) at akka.actor.Actor.aroundReceive(Actor.scala:539) at akka.actor.Actor.aroundReceive$(Actor.scala:537) at cromwell.engine.workflow.WorkflowDockerLookupActor.aroundReceive(WorkflowDockerLookupActor.scala:40) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:614) at akka.actor.ActorCell.invoke(ActorCell.scala:583) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:268) at akka.dispatch.Mailbox.run(Mailbox.scala:229) at akka.dispatch.Mailbox.exec(Mailbox.scala:241) at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) 2024-01-11 10:48:56,784 cromwell-system-akka.dispatchers.engine-dispatcher-31 INFO - BT-322 067c7f61:chip.bam2ta_ctl:1:1 is not eligible for call caching 2024-01-11 10:48:56,785 cromwell-system-akka.dispatchers.engine-dispatcher-31 WARN - BT-322 067c7f61:chip.bam2ta_ctl:1:1 disabling call caching due to error 2024-01-11 10:48:56,786 cromwell-system-akka.dispatchers.backend-dispatcher-79 WARN - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.bam2ta_ctl:1:1]: Unrecognized runtime attribute keys: disks, time, memory, cpu 2024-01-11 10:48:56,795 cromwell-system-akka.dispatchers.backend-dispatcher-79 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.bam2ta_ctl:1:1]: `set -e python3 $(which encode_task_bam2ta.py) \ /cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta_ctl/shard-1/inputs/2043588855/ENCFF267FMB.bam \ --disable-tn5-shift \ \ --mito-chr-name chrM \ --subsample 0 \ --mem-gb 3.3650515387207274 \ --nth 2` 2024-01-11 10:48:56,795 cromwell-system-akka.dispatchers.engine-dispatcher-34 INFO - WorkflowExecutionActor-067c7f61-ef2a-4b2b-b54f-bb1bce5217af [UUID(067c7f61)]: Starting chip.jsd 2024-01-11 10:48:56,808 cromwell-system-akka.dispatchers.backend-dispatcher-79 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.bam2ta_ctl:1:1]: executing: rm -f /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta_ctl/shard-1/execution/docker_cid if [ 'true' == 'true' ] && [ 'singularity' == 'docker' ] || \ [ 'true' == 'false' ] && [ 'true' == 'true' ] && [ ! -z 'encodedcc/chip-seq-pipeline:v2.2.1' ] then docker run -i --cidfile=/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta_ctl/shard-1/execution/docker_cid --user= --entrypoint=/bin/bash \ --volume=/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta_ctl/shard-1:/cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta_ctl/shard-1:delegated encodedcc/chip-seq-pipeline:v2.2.1 /cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta_ctl/shard-1/execution/script rc=$(docker wait `cat /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta_ctl/shard-1/execution/docker_cid`) docker rm `cat /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta_ctl/shard-1/execution/docker_cid` else # recover GID lost due to Cromwell running chmod 777 on CWD chown :`stat -c '%G' /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta_ctl/shard-1` -R /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta_ctl/shard-1 chmod g+s /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta_ctl/shard-1 if [ 'true' == 'true' ] && [ 'singularity' == 'singularity' ] || \ [ 'true' == 'false' ] && [ 'true' == 'true' ] && [ ! -z 'https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif' ] then mkdir -p $HOME/.singularity/lock/ flock --exclusive --timeout 600 \ $HOME/.singularity/lock/`echo -n 'https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif' | md5sum | cut -d' ' -f1` \ singularity exec --containall https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif echo 'Successfully pulled https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif' singularity exec --cleanenv --home=`dirname /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta_ctl/shard-1` \ --bind=/lustre/home/acct-medzy/medzy-cai,/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta_ctl/shard-1:/cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta_ctl/shard-1 \ \ https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif /bin/bash /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta_ctl/shard-1/execution/script & echo $! > /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta_ctl/shard-1/execution/docker_cid else # remap paths between inside and outside of a docker container shopt -s nullglob sed -i 's#/cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta_ctl/shard-1#/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta_ctl/shard-1#g' /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta_ctl/shard-1/execution/script `dirname /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta_ctl/shard-1/execution/script`/write_*.tmp if [ 'true' == 'true' ] && [ 'singularity' == 'conda' ] || \ [ 'true' == 'false' ] && [ 'true' == 'true' ] && [ ! -z 'encd-chip' ] then conda run --name=encd-chip /bin/bash /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta_ctl/shard-1/execution/script & echo $! > /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta_ctl/shard-1/execution/docker_cid else /bin/bash /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta_ctl/shard-1/execution/script & echo $! > /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta_ctl/shard-1/execution/docker_cid fi fi touch /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta_ctl/shard-1/execution/docker_cid.not_docker wait `cat /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta_ctl/shard-1/execution/docker_cid` rc=`echo $?` fi exit $rc 2024-01-11 10:48:58,806 cromwell-system-akka.dispatchers.backend-dispatcher-79 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.bam2ta_ctl:1:1]: job id: 1785675 2024-01-11 10:48:58,807 cromwell-system-akka.dispatchers.backend-dispatcher-79 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.bam2ta_ctl:1:1]: Status change from - to WaitingForReturnCode 2024-01-11 10:49:32,015 cromwell-system-akka.dispatchers.backend-dispatcher-79 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.bam2ta_ctl:1:1]: Status change from WaitingForReturnCode to Done 2024-01-11 10:49:33,955 cromwell-system-akka.dispatchers.engine-dispatcher-30 INFO - Assigned new job execution tokens to the following groups: 067c7f61: 1 2024-01-11 10:49:33,965 INFO - Request threw an exception on attempt #1. Retrying after 195 milliseconds org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 10:49:34,162 INFO - Request threw an exception on attempt #2. Retrying after 966 milliseconds org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 10:49:35,130 INFO - Request threw an exception on attempt #3. Retrying after 2754 milliseconds org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 10:49:37,887 INFO - Request method=GET uri=https://auth.docker.io/token?service=registry.docker.io&scope=repository:encodedcc/chip-seq-pipeline:pull headers= threw an exception on attempt #4. Giving up. org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 10:49:37,889 cromwell-system-akka.dispatchers.engine-dispatcher-29 WARN - BackendPreparationActor_for_067c7f61:chip.bam2ta:0:1 [UUID(067c7f61)]: Docker lookup failed java.lang.Exception: Failed to get docker hash for encodedcc/chip-seq-pipeline:v2.2.1 Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at cromwell.engine.workflow.WorkflowDockerLookupActor.cromwell$engine$workflow$WorkflowDockerLookupActor$$handleLookupFailure(WorkflowDockerLookupActor.scala:222) at cromwell.engine.workflow.WorkflowDockerLookupActor$$anonfun$3.applyOrElse(WorkflowDockerLookupActor.scala:88) at cromwell.engine.workflow.WorkflowDockerLookupActor$$anonfun$3.applyOrElse(WorkflowDockerLookupActor.scala:73) at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35) at akka.actor.FSM.processEvent(FSM.scala:707) at akka.actor.FSM.processEvent$(FSM.scala:704) at cromwell.engine.workflow.WorkflowDockerLookupActor.akka$actor$LoggingFSM$$super$processEvent(WorkflowDockerLookupActor.scala:40) at akka.actor.LoggingFSM.processEvent(FSM.scala:847) at akka.actor.LoggingFSM.processEvent$(FSM.scala:829) at cromwell.engine.workflow.WorkflowDockerLookupActor.processEvent(WorkflowDockerLookupActor.scala:40) at akka.actor.FSM.akka$actor$FSM$$processMsg(FSM.scala:701) at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:695) at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35) at cromwell.docker.DockerClientHelper$$anonfun$dockerResponseReceive$1.applyOrElse(DockerClientHelper.scala:16) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:269) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:270) at akka.actor.Actor.aroundReceive(Actor.scala:539) at akka.actor.Actor.aroundReceive$(Actor.scala:537) at cromwell.engine.workflow.WorkflowDockerLookupActor.aroundReceive(WorkflowDockerLookupActor.scala:40) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:614) at akka.actor.ActorCell.invoke(ActorCell.scala:583) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:268) at akka.dispatch.Mailbox.run(Mailbox.scala:229) at akka.dispatch.Mailbox.exec(Mailbox.scala:241) at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) 2024-01-11 10:49:37,890 cromwell-system-akka.dispatchers.engine-dispatcher-41 INFO - BT-322 067c7f61:chip.bam2ta:0:1 is not eligible for call caching 2024-01-11 10:49:37,890 cromwell-system-akka.dispatchers.engine-dispatcher-41 WARN - BT-322 067c7f61:chip.bam2ta:0:1 disabling call caching due to error 2024-01-11 10:49:37,892 cromwell-system-akka.dispatchers.backend-dispatcher-89 WARN - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.bam2ta:0:1]: Unrecognized runtime attribute keys: disks, time, memory, cpu 2024-01-11 10:49:37,902 cromwell-system-akka.dispatchers.backend-dispatcher-89 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.bam2ta:0:1]: `set -e python3 $(which encode_task_bam2ta.py) \ /cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta/shard-0/inputs/2043588855/ENCFF169ZIP.bam \ --disable-tn5-shift \ \ --mito-chr-name chrM \ --subsample 0 \ --mem-gb 3.391791918091476 \ --nth 2` 2024-01-11 10:49:37,909 cromwell-system-akka.dispatchers.backend-dispatcher-89 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.bam2ta:0:1]: executing: rm -f /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta/shard-0/execution/docker_cid if [ 'true' == 'true' ] && [ 'singularity' == 'docker' ] || \ [ 'true' == 'false' ] && [ 'true' == 'true' ] && [ ! -z 'encodedcc/chip-seq-pipeline:v2.2.1' ] then docker run -i --cidfile=/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta/shard-0/execution/docker_cid --user= --entrypoint=/bin/bash \ --volume=/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta/shard-0:/cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta/shard-0:delegated encodedcc/chip-seq-pipeline:v2.2.1 /cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta/shard-0/execution/script rc=$(docker wait `cat /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta/shard-0/execution/docker_cid`) docker rm `cat /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta/shard-0/execution/docker_cid` else # recover GID lost due to Cromwell running chmod 777 on CWD chown :`stat -c '%G' /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta/shard-0` -R /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta/shard-0 chmod g+s /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta/shard-0 if [ 'true' == 'true' ] && [ 'singularity' == 'singularity' ] || \ [ 'true' == 'false' ] && [ 'true' == 'true' ] && [ ! -z 'https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif' ] then mkdir -p $HOME/.singularity/lock/ flock --exclusive --timeout 600 \ $HOME/.singularity/lock/`echo -n 'https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif' | md5sum | cut -d' ' -f1` \ singularity exec --containall https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif echo 'Successfully pulled https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif' singularity exec --cleanenv --home=`dirname /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta/shard-0` \ --bind=/lustre/home/acct-medzy/medzy-cai,/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta/shard-0:/cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta/shard-0 \ \ https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif /bin/bash /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta/shard-0/execution/script & echo $! > /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta/shard-0/execution/docker_cid else # remap paths between inside and outside of a docker container shopt -s nullglob sed -i 's#/cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta/shard-0#/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta/shard-0#g' /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta/shard-0/execution/script `dirname /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta/shard-0/execution/script`/write_*.tmp if [ 'true' == 'true' ] && [ 'singularity' == 'conda' ] || \ [ 'true' == 'false' ] && [ 'true' == 'true' ] && [ ! -z 'encd-chip' ] then conda run --name=encd-chip /bin/bash /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta/shard-0/execution/script & echo $! > /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta/shard-0/execution/docker_cid else /bin/bash /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta/shard-0/execution/script & echo $! > /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta/shard-0/execution/docker_cid fi fi touch /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta/shard-0/execution/docker_cid.not_docker wait `cat /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta/shard-0/execution/docker_cid` rc=`echo $?` fi exit $rc 2024-01-11 10:49:38,806 cromwell-system-akka.dispatchers.backend-dispatcher-89 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.bam2ta:0:1]: job id: 1785968 2024-01-11 10:49:38,807 cromwell-system-akka.dispatchers.backend-dispatcher-92 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.bam2ta:0:1]: Status change from - to WaitingForReturnCode 2024-01-11 10:50:20,495 cromwell-system-akka.dispatchers.backend-dispatcher-91 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.bam2ta:0:1]: Status change from WaitingForReturnCode to Done 2024-01-11 10:50:21,954 cromwell-system-akka.dispatchers.engine-dispatcher-4 INFO - Assigned new job execution tokens to the following groups: 067c7f61: 1 2024-01-11 10:50:21,961 INFO - Request threw an exception on attempt #1. Retrying after 271 milliseconds org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 10:50:22,234 INFO - Request threw an exception on attempt #2. Retrying after 723 milliseconds org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 10:50:22,959 INFO - Request threw an exception on attempt #3. Retrying after 3194 milliseconds org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 10:50:26,156 INFO - Request method=GET uri=https://auth.docker.io/token?service=registry.docker.io&scope=repository:encodedcc/chip-seq-pipeline:pull headers= threw an exception on attempt #4. Giving up. org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 10:50:26,158 cromwell-system-akka.dispatchers.engine-dispatcher-41 WARN - BackendPreparationActor_for_067c7f61:chip.bam2ta:1:1 [UUID(067c7f61)]: Docker lookup failed java.lang.Exception: Failed to get docker hash for encodedcc/chip-seq-pipeline:v2.2.1 Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at cromwell.engine.workflow.WorkflowDockerLookupActor.cromwell$engine$workflow$WorkflowDockerLookupActor$$handleLookupFailure(WorkflowDockerLookupActor.scala:222) at cromwell.engine.workflow.WorkflowDockerLookupActor$$anonfun$3.applyOrElse(WorkflowDockerLookupActor.scala:88) at cromwell.engine.workflow.WorkflowDockerLookupActor$$anonfun$3.applyOrElse(WorkflowDockerLookupActor.scala:73) at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35) at akka.actor.FSM.processEvent(FSM.scala:707) at akka.actor.FSM.processEvent$(FSM.scala:704) at cromwell.engine.workflow.WorkflowDockerLookupActor.akka$actor$LoggingFSM$$super$processEvent(WorkflowDockerLookupActor.scala:40) at akka.actor.LoggingFSM.processEvent(FSM.scala:847) at akka.actor.LoggingFSM.processEvent$(FSM.scala:829) at cromwell.engine.workflow.WorkflowDockerLookupActor.processEvent(WorkflowDockerLookupActor.scala:40) at akka.actor.FSM.akka$actor$FSM$$processMsg(FSM.scala:701) at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:695) at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35) at cromwell.docker.DockerClientHelper$$anonfun$dockerResponseReceive$1.applyOrElse(DockerClientHelper.scala:16) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:269) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:270) at akka.actor.Actor.aroundReceive(Actor.scala:539) at akka.actor.Actor.aroundReceive$(Actor.scala:537) at cromwell.engine.workflow.WorkflowDockerLookupActor.aroundReceive(WorkflowDockerLookupActor.scala:40) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:614) at akka.actor.ActorCell.invoke(ActorCell.scala:583) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:268) at akka.dispatch.Mailbox.run(Mailbox.scala:229) at akka.dispatch.Mailbox.exec(Mailbox.scala:241) at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) 2024-01-11 10:50:26,159 cromwell-system-akka.dispatchers.engine-dispatcher-32 INFO - BT-322 067c7f61:chip.bam2ta:1:1 is not eligible for call caching 2024-01-11 10:50:26,159 cromwell-system-akka.dispatchers.engine-dispatcher-32 WARN - BT-322 067c7f61:chip.bam2ta:1:1 disabling call caching due to error 2024-01-11 10:50:26,160 cromwell-system-akka.dispatchers.backend-dispatcher-104 WARN - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.bam2ta:1:1]: Unrecognized runtime attribute keys: disks, time, memory, cpu 2024-01-11 10:50:26,168 cromwell-system-akka.dispatchers.backend-dispatcher-104 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.bam2ta:1:1]: `set -e python3 $(which encode_task_bam2ta.py) \ /cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta/shard-1/inputs/2043588855/ENCFF597JXK.bam \ --disable-tn5-shift \ \ --mito-chr-name chrM \ --subsample 0 \ --mem-gb 3.3582566055282954 \ --nth 2` 2024-01-11 10:50:26,174 cromwell-system-akka.dispatchers.backend-dispatcher-104 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.bam2ta:1:1]: executing: rm -f /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta/shard-1/execution/docker_cid if [ 'true' == 'true' ] && [ 'singularity' == 'docker' ] || \ [ 'true' == 'false' ] && [ 'true' == 'true' ] && [ ! -z 'encodedcc/chip-seq-pipeline:v2.2.1' ] then docker run -i --cidfile=/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta/shard-1/execution/docker_cid --user= --entrypoint=/bin/bash \ --volume=/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta/shard-1:/cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta/shard-1:delegated encodedcc/chip-seq-pipeline:v2.2.1 /cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta/shard-1/execution/script rc=$(docker wait `cat /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta/shard-1/execution/docker_cid`) docker rm `cat /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta/shard-1/execution/docker_cid` else # recover GID lost due to Cromwell running chmod 777 on CWD chown :`stat -c '%G' /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta/shard-1` -R /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta/shard-1 chmod g+s /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta/shard-1 if [ 'true' == 'true' ] && [ 'singularity' == 'singularity' ] || \ [ 'true' == 'false' ] && [ 'true' == 'true' ] && [ ! -z 'https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif' ] then mkdir -p $HOME/.singularity/lock/ flock --exclusive --timeout 600 \ $HOME/.singularity/lock/`echo -n 'https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif' | md5sum | cut -d' ' -f1` \ singularity exec --containall https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif echo 'Successfully pulled https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif' singularity exec --cleanenv --home=`dirname /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta/shard-1` \ --bind=/lustre/home/acct-medzy/medzy-cai,/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta/shard-1:/cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta/shard-1 \ \ https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif /bin/bash /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta/shard-1/execution/script & echo $! > /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta/shard-1/execution/docker_cid else # remap paths between inside and outside of a docker container shopt -s nullglob sed -i 's#/cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta/shard-1#/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta/shard-1#g' /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta/shard-1/execution/script `dirname /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta/shard-1/execution/script`/write_*.tmp if [ 'true' == 'true' ] && [ 'singularity' == 'conda' ] || \ [ 'true' == 'false' ] && [ 'true' == 'true' ] && [ ! -z 'encd-chip' ] then conda run --name=encd-chip /bin/bash /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta/shard-1/execution/script & echo $! > /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta/shard-1/execution/docker_cid else /bin/bash /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta/shard-1/execution/script & echo $! > /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta/shard-1/execution/docker_cid fi fi touch /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta/shard-1/execution/docker_cid.not_docker wait `cat /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta/shard-1/execution/docker_cid` rc=`echo $?` fi exit $rc 2024-01-11 10:50:26,557 cromwell-system-akka.dispatchers.engine-dispatcher-41 INFO - WorkflowExecutionActor-067c7f61-ef2a-4b2b-b54f-bb1bce5217af [UUID(067c7f61)]: Starting chip.spr 2024-01-11 10:50:28,595 cromwell-system-akka.dispatchers.engine-dispatcher-41 INFO - WorkflowExecutionActor-067c7f61-ef2a-4b2b-b54f-bb1bce5217af [UUID(067c7f61)]: Starting chip.xcor 2024-01-11 10:50:28,805 cromwell-system-akka.dispatchers.backend-dispatcher-104 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.bam2ta:1:1]: job id: 1786298 2024-01-11 10:50:28,806 cromwell-system-akka.dispatchers.backend-dispatcher-106 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.bam2ta:1:1]: Status change from - to WaitingForReturnCode 2024-01-11 10:51:04,594 cromwell-system-akka.dispatchers.backend-dispatcher-104 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.bam2ta:1:1]: Status change from WaitingForReturnCode to Done 2024-01-11 10:51:05,954 cromwell-system-akka.dispatchers.engine-dispatcher-39 INFO - Assigned new job execution tokens to the following groups: 067c7f61: 1 2024-01-11 10:51:05,963 INFO - Request threw an exception on attempt #1. Retrying after 153 milliseconds org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 10:51:06,117 INFO - Request threw an exception on attempt #2. Retrying after 773 milliseconds org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 10:51:06,892 INFO - Request threw an exception on attempt #3. Retrying after 6417 milliseconds org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 10:51:10,416 cromwell-system-akka.dispatchers.engine-dispatcher-32 INFO - WorkflowExecutionActor-067c7f61-ef2a-4b2b-b54f-bb1bce5217af [UUID(067c7f61)]: Starting chip.spr 2024-01-11 10:51:12,456 cromwell-system-akka.dispatchers.engine-dispatcher-32 INFO - WorkflowExecutionActor-067c7f61-ef2a-4b2b-b54f-bb1bce5217af [UUID(067c7f61)]: Starting chip.pool_ta, chip.xcor 2024-01-11 10:51:13,312 INFO - Request method=GET uri=https://auth.docker.io/token?service=registry.docker.io&scope=repository:encodedcc/chip-seq-pipeline:pull headers= threw an exception on attempt #4. Giving up. org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 10:51:13,314 cromwell-system-akka.dispatchers.engine-dispatcher-49 WARN - BackendPreparationActor_for_067c7f61:chip.bam2ta_ctl:0:1 [UUID(067c7f61)]: Docker lookup failed java.lang.Exception: Failed to get docker hash for encodedcc/chip-seq-pipeline:v2.2.1 Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at cromwell.engine.workflow.WorkflowDockerLookupActor.cromwell$engine$workflow$WorkflowDockerLookupActor$$handleLookupFailure(WorkflowDockerLookupActor.scala:222) at cromwell.engine.workflow.WorkflowDockerLookupActor$$anonfun$3.applyOrElse(WorkflowDockerLookupActor.scala:88) at cromwell.engine.workflow.WorkflowDockerLookupActor$$anonfun$3.applyOrElse(WorkflowDockerLookupActor.scala:73) at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35) at akka.actor.FSM.processEvent(FSM.scala:707) at akka.actor.FSM.processEvent$(FSM.scala:704) at cromwell.engine.workflow.WorkflowDockerLookupActor.akka$actor$LoggingFSM$$super$processEvent(WorkflowDockerLookupActor.scala:40) at akka.actor.LoggingFSM.processEvent(FSM.scala:847) at akka.actor.LoggingFSM.processEvent$(FSM.scala:829) at cromwell.engine.workflow.WorkflowDockerLookupActor.processEvent(WorkflowDockerLookupActor.scala:40) at akka.actor.FSM.akka$actor$FSM$$processMsg(FSM.scala:701) at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:695) at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35) at cromwell.docker.DockerClientHelper$$anonfun$dockerResponseReceive$1.applyOrElse(DockerClientHelper.scala:16) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:269) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:270) at akka.actor.Actor.aroundReceive(Actor.scala:539) at akka.actor.Actor.aroundReceive$(Actor.scala:537) at cromwell.engine.workflow.WorkflowDockerLookupActor.aroundReceive(WorkflowDockerLookupActor.scala:40) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:614) at akka.actor.ActorCell.invoke(ActorCell.scala:583) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:268) at akka.dispatch.Mailbox.run(Mailbox.scala:229) at akka.dispatch.Mailbox.exec(Mailbox.scala:241) at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) 2024-01-11 10:51:13,315 cromwell-system-akka.dispatchers.engine-dispatcher-49 INFO - BT-322 067c7f61:chip.bam2ta_ctl:0:1 is not eligible for call caching 2024-01-11 10:51:13,315 cromwell-system-akka.dispatchers.engine-dispatcher-49 WARN - BT-322 067c7f61:chip.bam2ta_ctl:0:1 disabling call caching due to error 2024-01-11 10:51:13,316 cromwell-system-akka.dispatchers.backend-dispatcher-108 WARN - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.bam2ta_ctl:0:1]: Unrecognized runtime attribute keys: disks, time, memory, cpu 2024-01-11 10:51:13,325 cromwell-system-akka.dispatchers.backend-dispatcher-108 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.bam2ta_ctl:0:1]: `set -e python3 $(which encode_task_bam2ta.py) \ /cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta_ctl/shard-0/inputs/2043588855/ENCFF378YAW.bam \ --disable-tn5-shift \ \ --mito-chr-name chrM \ --subsample 0 \ --mem-gb 3.298334961719811 \ --nth 2` 2024-01-11 10:51:13,332 cromwell-system-akka.dispatchers.backend-dispatcher-108 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.bam2ta_ctl:0:1]: executing: rm -f /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta_ctl/shard-0/execution/docker_cid if [ 'true' == 'true' ] && [ 'singularity' == 'docker' ] || \ [ 'true' == 'false' ] && [ 'true' == 'true' ] && [ ! -z 'encodedcc/chip-seq-pipeline:v2.2.1' ] then docker run -i --cidfile=/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta_ctl/shard-0/execution/docker_cid --user= --entrypoint=/bin/bash \ --volume=/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta_ctl/shard-0:/cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta_ctl/shard-0:delegated encodedcc/chip-seq-pipeline:v2.2.1 /cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta_ctl/shard-0/execution/script rc=$(docker wait `cat /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta_ctl/shard-0/execution/docker_cid`) docker rm `cat /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta_ctl/shard-0/execution/docker_cid` else # recover GID lost due to Cromwell running chmod 777 on CWD chown :`stat -c '%G' /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta_ctl/shard-0` -R /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta_ctl/shard-0 chmod g+s /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta_ctl/shard-0 if [ 'true' == 'true' ] && [ 'singularity' == 'singularity' ] || \ [ 'true' == 'false' ] && [ 'true' == 'true' ] && [ ! -z 'https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif' ] then mkdir -p $HOME/.singularity/lock/ flock --exclusive --timeout 600 \ $HOME/.singularity/lock/`echo -n 'https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif' | md5sum | cut -d' ' -f1` \ singularity exec --containall https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif echo 'Successfully pulled https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif' singularity exec --cleanenv --home=`dirname /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta_ctl/shard-0` \ --bind=/lustre/home/acct-medzy/medzy-cai,/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta_ctl/shard-0:/cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta_ctl/shard-0 \ \ https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif /bin/bash /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta_ctl/shard-0/execution/script & echo $! > /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta_ctl/shard-0/execution/docker_cid else # remap paths between inside and outside of a docker container shopt -s nullglob sed -i 's#/cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta_ctl/shard-0#/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta_ctl/shard-0#g' /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta_ctl/shard-0/execution/script `dirname /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta_ctl/shard-0/execution/script`/write_*.tmp if [ 'true' == 'true' ] && [ 'singularity' == 'conda' ] || \ [ 'true' == 'false' ] && [ 'true' == 'true' ] && [ ! -z 'encd-chip' ] then conda run --name=encd-chip /bin/bash /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta_ctl/shard-0/execution/script & echo $! > /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta_ctl/shard-0/execution/docker_cid else /bin/bash /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta_ctl/shard-0/execution/script & echo $! > /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta_ctl/shard-0/execution/docker_cid fi fi touch /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta_ctl/shard-0/execution/docker_cid.not_docker wait `cat /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-bam2ta_ctl/shard-0/execution/docker_cid` rc=`echo $?` fi exit $rc 2024-01-11 10:51:13,805 cromwell-system-akka.dispatchers.backend-dispatcher-108 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.bam2ta_ctl:0:1]: job id: 1786530 2024-01-11 10:51:13,807 cromwell-system-akka.dispatchers.backend-dispatcher-111 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.bam2ta_ctl:0:1]: Status change from - to WaitingForReturnCode 2024-01-11 10:51:37,475 cromwell-system-akka.dispatchers.backend-dispatcher-111 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.bam2ta_ctl:0:1]: Status change from WaitingForReturnCode to Done 2024-01-11 10:51:37,954 cromwell-system-akka.dispatchers.engine-dispatcher-37 INFO - Assigned new job execution tokens to the following groups: 067c7f61: 1 2024-01-11 10:51:38,027 INFO - Request threw an exception on attempt #1. Retrying after 467 milliseconds org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 10:51:38,498 INFO - Request threw an exception on attempt #2. Retrying after 334 milliseconds org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 10:51:38,834 INFO - Request threw an exception on attempt #3. Retrying after 5502 milliseconds org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 10:51:44,339 INFO - Request method=GET uri=https://auth.docker.io/token?service=registry.docker.io&scope=repository:encodedcc/chip-seq-pipeline:pull headers= threw an exception on attempt #4. Giving up. org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 10:51:44,341 cromwell-system-akka.dispatchers.engine-dispatcher-48 WARN - BackendPreparationActor_for_067c7f61:chip.gc_bias:0:1 [UUID(067c7f61)]: Docker lookup failed java.lang.Exception: Failed to get docker hash for encodedcc/chip-seq-pipeline:v2.2.1 Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at cromwell.engine.workflow.WorkflowDockerLookupActor.cromwell$engine$workflow$WorkflowDockerLookupActor$$handleLookupFailure(WorkflowDockerLookupActor.scala:222) at cromwell.engine.workflow.WorkflowDockerLookupActor$$anonfun$3.applyOrElse(WorkflowDockerLookupActor.scala:88) at cromwell.engine.workflow.WorkflowDockerLookupActor$$anonfun$3.applyOrElse(WorkflowDockerLookupActor.scala:73) at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35) at akka.actor.FSM.processEvent(FSM.scala:707) at akka.actor.FSM.processEvent$(FSM.scala:704) at cromwell.engine.workflow.WorkflowDockerLookupActor.akka$actor$LoggingFSM$$super$processEvent(WorkflowDockerLookupActor.scala:40) at akka.actor.LoggingFSM.processEvent(FSM.scala:847) at akka.actor.LoggingFSM.processEvent$(FSM.scala:829) at cromwell.engine.workflow.WorkflowDockerLookupActor.processEvent(WorkflowDockerLookupActor.scala:40) at akka.actor.FSM.akka$actor$FSM$$processMsg(FSM.scala:701) at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:695) at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35) at cromwell.docker.DockerClientHelper$$anonfun$dockerResponseReceive$1.applyOrElse(DockerClientHelper.scala:16) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:269) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:270) at akka.actor.Actor.aroundReceive(Actor.scala:539) at akka.actor.Actor.aroundReceive$(Actor.scala:537) at cromwell.engine.workflow.WorkflowDockerLookupActor.aroundReceive(WorkflowDockerLookupActor.scala:40) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:614) at akka.actor.ActorCell.invoke(ActorCell.scala:583) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:268) at akka.dispatch.Mailbox.run(Mailbox.scala:229) at akka.dispatch.Mailbox.exec(Mailbox.scala:241) at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) 2024-01-11 10:51:44,342 cromwell-system-akka.dispatchers.engine-dispatcher-48 INFO - BT-322 067c7f61:chip.gc_bias:0:1 is not eligible for call caching 2024-01-11 10:51:44,342 cromwell-system-akka.dispatchers.engine-dispatcher-48 WARN - BT-322 067c7f61:chip.gc_bias:0:1 disabling call caching due to error 2024-01-11 10:51:44,344 cromwell-system-akka.dispatchers.backend-dispatcher-112 WARN - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.gc_bias:0:1]: Unrecognized runtime attribute keys: disks, time, memory, cpu 2024-01-11 10:51:44,355 cromwell-system-akka.dispatchers.backend-dispatcher-112 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.gc_bias:0:1]: `set -e python3 $(which encode_task_gc_bias.py) \ --nodup-bam /cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-gc_bias/shard-0/inputs/2043588855/ENCFF169ZIP.bam \ --ref-fa /cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-gc_bias/shard-0/inputs/977218590/male.hg19.fa.gz \ --picard-java-heap 4G` 2024-01-11 10:51:44,361 cromwell-system-akka.dispatchers.backend-dispatcher-112 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.gc_bias:0:1]: executing: rm -f /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-gc_bias/shard-0/execution/docker_cid if [ 'true' == 'true' ] && [ 'singularity' == 'docker' ] || \ [ 'true' == 'false' ] && [ 'true' == 'true' ] && [ ! -z 'encodedcc/chip-seq-pipeline:v2.2.1' ] then docker run -i --cidfile=/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-gc_bias/shard-0/execution/docker_cid --user= --entrypoint=/bin/bash \ --volume=/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-gc_bias/shard-0:/cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-gc_bias/shard-0:delegated encodedcc/chip-seq-pipeline:v2.2.1 /cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-gc_bias/shard-0/execution/script rc=$(docker wait `cat /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-gc_bias/shard-0/execution/docker_cid`) docker rm `cat /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-gc_bias/shard-0/execution/docker_cid` else # recover GID lost due to Cromwell running chmod 777 on CWD chown :`stat -c '%G' /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-gc_bias/shard-0` -R /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-gc_bias/shard-0 chmod g+s /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-gc_bias/shard-0 if [ 'true' == 'true' ] && [ 'singularity' == 'singularity' ] || \ [ 'true' == 'false' ] && [ 'true' == 'true' ] && [ ! -z 'https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif' ] then mkdir -p $HOME/.singularity/lock/ flock --exclusive --timeout 600 \ $HOME/.singularity/lock/`echo -n 'https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif' | md5sum | cut -d' ' -f1` \ singularity exec --containall https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif echo 'Successfully pulled https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif' singularity exec --cleanenv --home=`dirname /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-gc_bias/shard-0` \ --bind=/lustre/home/acct-medzy/medzy-cai,/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-gc_bias/shard-0:/cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-gc_bias/shard-0 \ \ https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif /bin/bash /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-gc_bias/shard-0/execution/script & echo $! > /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-gc_bias/shard-0/execution/docker_cid else # remap paths between inside and outside of a docker container shopt -s nullglob sed -i 's#/cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-gc_bias/shard-0#/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-gc_bias/shard-0#g' /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-gc_bias/shard-0/execution/script `dirname /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-gc_bias/shard-0/execution/script`/write_*.tmp if [ 'true' == 'true' ] && [ 'singularity' == 'conda' ] || \ [ 'true' == 'false' ] && [ 'true' == 'true' ] && [ ! -z 'encd-chip' ] then conda run --name=encd-chip /bin/bash /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-gc_bias/shard-0/execution/script & echo $! > /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-gc_bias/shard-0/execution/docker_cid else /bin/bash /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-gc_bias/shard-0/execution/script & echo $! > /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-gc_bias/shard-0/execution/docker_cid fi fi touch /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-gc_bias/shard-0/execution/docker_cid.not_docker wait `cat /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-gc_bias/shard-0/execution/docker_cid` rc=`echo $?` fi exit $rc 2024-01-11 10:51:45,096 cromwell-system-akka.dispatchers.engine-dispatcher-33 INFO - WorkflowExecutionActor-067c7f61-ef2a-4b2b-b54f-bb1bce5217af [UUID(067c7f61)]: Starting chip.pool_ta_ctl 2024-01-11 10:51:48,805 cromwell-system-akka.dispatchers.backend-dispatcher-112 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.gc_bias:0:1]: job id: 1786825 2024-01-11 10:51:48,807 cromwell-system-akka.dispatchers.backend-dispatcher-116 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.gc_bias:0:1]: Status change from - to WaitingForReturnCode 2024-01-11 10:56:31,296 cromwell-system-akka.dispatchers.backend-dispatcher-114 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.gc_bias:0:1]: Status change from WaitingForReturnCode to Done 2024-01-11 10:56:31,955 cromwell-system-akka.dispatchers.engine-dispatcher-43 INFO - Assigned new job execution tokens to the following groups: 067c7f61: 1 2024-01-11 10:56:31,966 INFO - Request threw an exception on attempt #1. Retrying after 570 milliseconds org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 10:56:32,538 INFO - Request threw an exception on attempt #2. Retrying after 931 milliseconds org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 10:56:33,472 INFO - Request threw an exception on attempt #3. Retrying after 4787 milliseconds org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 10:56:38,261 INFO - Request method=GET uri=https://auth.docker.io/token?service=registry.docker.io&scope=repository:encodedcc/chip-seq-pipeline:pull headers= threw an exception on attempt #4. Giving up. org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 10:56:38,263 cromwell-system-akka.dispatchers.engine-dispatcher-40 WARN - BackendPreparationActor_for_067c7f61:chip.gc_bias:1:1 [UUID(067c7f61)]: Docker lookup failed java.lang.Exception: Failed to get docker hash for encodedcc/chip-seq-pipeline:v2.2.1 Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at cromwell.engine.workflow.WorkflowDockerLookupActor.cromwell$engine$workflow$WorkflowDockerLookupActor$$handleLookupFailure(WorkflowDockerLookupActor.scala:222) at cromwell.engine.workflow.WorkflowDockerLookupActor$$anonfun$3.applyOrElse(WorkflowDockerLookupActor.scala:88) at cromwell.engine.workflow.WorkflowDockerLookupActor$$anonfun$3.applyOrElse(WorkflowDockerLookupActor.scala:73) at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35) at akka.actor.FSM.processEvent(FSM.scala:707) at akka.actor.FSM.processEvent$(FSM.scala:704) at cromwell.engine.workflow.WorkflowDockerLookupActor.akka$actor$LoggingFSM$$super$processEvent(WorkflowDockerLookupActor.scala:40) at akka.actor.LoggingFSM.processEvent(FSM.scala:847) at akka.actor.LoggingFSM.processEvent$(FSM.scala:829) at cromwell.engine.workflow.WorkflowDockerLookupActor.processEvent(WorkflowDockerLookupActor.scala:40) at akka.actor.FSM.akka$actor$FSM$$processMsg(FSM.scala:701) at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:695) at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35) at cromwell.docker.DockerClientHelper$$anonfun$dockerResponseReceive$1.applyOrElse(DockerClientHelper.scala:16) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:269) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:270) at akka.actor.Actor.aroundReceive(Actor.scala:539) at akka.actor.Actor.aroundReceive$(Actor.scala:537) at cromwell.engine.workflow.WorkflowDockerLookupActor.aroundReceive(WorkflowDockerLookupActor.scala:40) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:614) at akka.actor.ActorCell.invoke(ActorCell.scala:583) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:268) at akka.dispatch.Mailbox.run(Mailbox.scala:229) at akka.dispatch.Mailbox.exec(Mailbox.scala:241) at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) 2024-01-11 10:56:38,265 cromwell-system-akka.dispatchers.engine-dispatcher-40 INFO - BT-322 067c7f61:chip.gc_bias:1:1 is not eligible for call caching 2024-01-11 10:56:38,265 cromwell-system-akka.dispatchers.engine-dispatcher-40 WARN - BT-322 067c7f61:chip.gc_bias:1:1 disabling call caching due to error 2024-01-11 10:56:38,266 cromwell-system-akka.dispatchers.backend-dispatcher-118 WARN - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.gc_bias:1:1]: Unrecognized runtime attribute keys: disks, time, memory, cpu 2024-01-11 10:56:38,278 cromwell-system-akka.dispatchers.backend-dispatcher-118 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.gc_bias:1:1]: `set -e python3 $(which encode_task_gc_bias.py) \ --nodup-bam /cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-gc_bias/shard-1/inputs/2043588855/ENCFF597JXK.bam \ --ref-fa /cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-gc_bias/shard-1/inputs/977218590/male.hg19.fa.gz \ --picard-java-heap 4G` 2024-01-11 10:56:38,284 cromwell-system-akka.dispatchers.backend-dispatcher-118 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.gc_bias:1:1]: executing: rm -f /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-gc_bias/shard-1/execution/docker_cid if [ 'true' == 'true' ] && [ 'singularity' == 'docker' ] || \ [ 'true' == 'false' ] && [ 'true' == 'true' ] && [ ! -z 'encodedcc/chip-seq-pipeline:v2.2.1' ] then docker run -i --cidfile=/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-gc_bias/shard-1/execution/docker_cid --user= --entrypoint=/bin/bash \ --volume=/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-gc_bias/shard-1:/cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-gc_bias/shard-1:delegated encodedcc/chip-seq-pipeline:v2.2.1 /cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-gc_bias/shard-1/execution/script rc=$(docker wait `cat /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-gc_bias/shard-1/execution/docker_cid`) docker rm `cat /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-gc_bias/shard-1/execution/docker_cid` else # recover GID lost due to Cromwell running chmod 777 on CWD chown :`stat -c '%G' /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-gc_bias/shard-1` -R /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-gc_bias/shard-1 chmod g+s /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-gc_bias/shard-1 if [ 'true' == 'true' ] && [ 'singularity' == 'singularity' ] || \ [ 'true' == 'false' ] && [ 'true' == 'true' ] && [ ! -z 'https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif' ] then mkdir -p $HOME/.singularity/lock/ flock --exclusive --timeout 600 \ $HOME/.singularity/lock/`echo -n 'https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif' | md5sum | cut -d' ' -f1` \ singularity exec --containall https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif echo 'Successfully pulled https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif' singularity exec --cleanenv --home=`dirname /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-gc_bias/shard-1` \ --bind=/lustre/home/acct-medzy/medzy-cai,/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-gc_bias/shard-1:/cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-gc_bias/shard-1 \ \ https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif /bin/bash /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-gc_bias/shard-1/execution/script & echo $! > /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-gc_bias/shard-1/execution/docker_cid else # remap paths between inside and outside of a docker container shopt -s nullglob sed -i 's#/cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-gc_bias/shard-1#/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-gc_bias/shard-1#g' /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-gc_bias/shard-1/execution/script `dirname /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-gc_bias/shard-1/execution/script`/write_*.tmp if [ 'true' == 'true' ] && [ 'singularity' == 'conda' ] || \ [ 'true' == 'false' ] && [ 'true' == 'true' ] && [ ! -z 'encd-chip' ] then conda run --name=encd-chip /bin/bash /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-gc_bias/shard-1/execution/script & echo $! > /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-gc_bias/shard-1/execution/docker_cid else /bin/bash /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-gc_bias/shard-1/execution/script & echo $! > /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-gc_bias/shard-1/execution/docker_cid fi fi touch /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-gc_bias/shard-1/execution/docker_cid.not_docker wait `cat /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-gc_bias/shard-1/execution/docker_cid` rc=`echo $?` fi exit $rc 2024-01-11 10:56:38,805 cromwell-system-akka.dispatchers.backend-dispatcher-119 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.gc_bias:1:1]: job id: 1788870 2024-01-11 10:56:38,806 cromwell-system-akka.dispatchers.backend-dispatcher-120 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.gc_bias:1:1]: Status change from - to WaitingForReturnCode 2024-01-11 11:01:18,135 cromwell-system-akka.dispatchers.backend-dispatcher-119 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.gc_bias:1:1]: Status change from WaitingForReturnCode to Done 2024-01-11 11:01:19,954 cromwell-system-akka.dispatchers.engine-dispatcher-48 INFO - Assigned new job execution tokens to the following groups: 067c7f61: 1 2024-01-11 11:01:19,969 INFO - Request threw an exception on attempt #1. Retrying after 851 milliseconds org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 11:01:20,822 INFO - Request threw an exception on attempt #2. Retrying after 2911 milliseconds org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 11:01:23,737 INFO - Request threw an exception on attempt #3. Retrying after 286 milliseconds org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 11:01:24,026 INFO - Request method=GET uri=https://auth.docker.io/token?service=registry.docker.io&scope=repository:encodedcc/chip-seq-pipeline:pull headers= threw an exception on attempt #4. Giving up. org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 11:01:24,028 cromwell-system-akka.dispatchers.engine-dispatcher-44 WARN - BackendPreparationActor_for_067c7f61:chip.jsd:-1:1 [UUID(067c7f61)]: Docker lookup failed java.lang.Exception: Failed to get docker hash for encodedcc/chip-seq-pipeline:v2.2.1 Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at cromwell.engine.workflow.WorkflowDockerLookupActor.cromwell$engine$workflow$WorkflowDockerLookupActor$$handleLookupFailure(WorkflowDockerLookupActor.scala:222) at cromwell.engine.workflow.WorkflowDockerLookupActor$$anonfun$3.applyOrElse(WorkflowDockerLookupActor.scala:88) at cromwell.engine.workflow.WorkflowDockerLookupActor$$anonfun$3.applyOrElse(WorkflowDockerLookupActor.scala:73) at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35) at akka.actor.FSM.processEvent(FSM.scala:707) at akka.actor.FSM.processEvent$(FSM.scala:704) at cromwell.engine.workflow.WorkflowDockerLookupActor.akka$actor$LoggingFSM$$super$processEvent(WorkflowDockerLookupActor.scala:40) at akka.actor.LoggingFSM.processEvent(FSM.scala:847) at akka.actor.LoggingFSM.processEvent$(FSM.scala:829) at cromwell.engine.workflow.WorkflowDockerLookupActor.processEvent(WorkflowDockerLookupActor.scala:40) at akka.actor.FSM.akka$actor$FSM$$processMsg(FSM.scala:701) at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:695) at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35) at cromwell.docker.DockerClientHelper$$anonfun$dockerResponseReceive$1.applyOrElse(DockerClientHelper.scala:16) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:269) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:270) at akka.actor.Actor.aroundReceive(Actor.scala:539) at akka.actor.Actor.aroundReceive$(Actor.scala:537) at cromwell.engine.workflow.WorkflowDockerLookupActor.aroundReceive(WorkflowDockerLookupActor.scala:40) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:614) at akka.actor.ActorCell.invoke(ActorCell.scala:583) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:268) at akka.dispatch.Mailbox.run(Mailbox.scala:229) at akka.dispatch.Mailbox.exec(Mailbox.scala:241) at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) 2024-01-11 11:01:24,029 cromwell-system-akka.dispatchers.engine-dispatcher-5 INFO - BT-322 067c7f61:chip.jsd:-1:1 is not eligible for call caching 2024-01-11 11:01:24,029 cromwell-system-akka.dispatchers.engine-dispatcher-5 WARN - BT-322 067c7f61:chip.jsd:-1:1 disabling call caching due to error 2024-01-11 11:01:24,031 cromwell-system-akka.dispatchers.backend-dispatcher-125 WARN - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.jsd:NA:1]: Unrecognized runtime attribute keys: disks, time, memory, cpu 2024-01-11 11:01:24,047 cromwell-system-akka.dispatchers.backend-dispatcher-125 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.jsd:NA:1]: `set -e python3 $(which encode_task_jsd.py) \ /cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-jsd/inputs/2043588855/ENCFF169ZIP.bam /cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-jsd/inputs/2043588855/ENCFF597JXK.bam \ --ctl-bam /cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-jsd/inputs/2043588855/ENCFF378YAW.bam \ --mapq-thresh 30 \ --blacklist /cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-jsd/inputs/977218590/wgEncodeDacMapabilityConsensusExcludable.bed.gz \ --nth 4` 2024-01-11 11:01:24,052 cromwell-system-akka.dispatchers.backend-dispatcher-125 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.jsd:NA:1]: executing: rm -f /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-jsd/execution/docker_cid if [ 'true' == 'true' ] && [ 'singularity' == 'docker' ] || \ [ 'true' == 'false' ] && [ 'true' == 'true' ] && [ ! -z 'encodedcc/chip-seq-pipeline:v2.2.1' ] then docker run -i --cidfile=/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-jsd/execution/docker_cid --user= --entrypoint=/bin/bash \ --volume=/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-jsd:/cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-jsd:delegated encodedcc/chip-seq-pipeline:v2.2.1 /cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-jsd/execution/script rc=$(docker wait `cat /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-jsd/execution/docker_cid`) docker rm `cat /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-jsd/execution/docker_cid` else # recover GID lost due to Cromwell running chmod 777 on CWD chown :`stat -c '%G' /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-jsd` -R /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-jsd chmod g+s /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-jsd if [ 'true' == 'true' ] && [ 'singularity' == 'singularity' ] || \ [ 'true' == 'false' ] && [ 'true' == 'true' ] && [ ! -z 'https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif' ] then mkdir -p $HOME/.singularity/lock/ flock --exclusive --timeout 600 \ $HOME/.singularity/lock/`echo -n 'https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif' | md5sum | cut -d' ' -f1` \ singularity exec --containall https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif echo 'Successfully pulled https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif' singularity exec --cleanenv --home=`dirname /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-jsd` \ --bind=/lustre/home/acct-medzy/medzy-cai,/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-jsd:/cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-jsd \ \ https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif /bin/bash /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-jsd/execution/script & echo $! > /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-jsd/execution/docker_cid else # remap paths between inside and outside of a docker container shopt -s nullglob sed -i 's#/cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-jsd#/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-jsd#g' /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-jsd/execution/script `dirname /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-jsd/execution/script`/write_*.tmp if [ 'true' == 'true' ] && [ 'singularity' == 'conda' ] || \ [ 'true' == 'false' ] && [ 'true' == 'true' ] && [ ! -z 'encd-chip' ] then conda run --name=encd-chip /bin/bash /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-jsd/execution/script & echo $! > /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-jsd/execution/docker_cid else /bin/bash /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-jsd/execution/script & echo $! > /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-jsd/execution/docker_cid fi fi touch /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-jsd/execution/docker_cid.not_docker wait `cat /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-jsd/execution/docker_cid` rc=`echo $?` fi exit $rc 2024-01-11 11:01:28,805 cromwell-system-akka.dispatchers.backend-dispatcher-125 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.jsd:NA:1]: job id: 1791190 2024-01-11 11:01:28,806 cromwell-system-akka.dispatchers.backend-dispatcher-128 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.jsd:NA:1]: Status change from - to WaitingForReturnCode 2024-01-11 11:13:20,717 cromwell-system-akka.dispatchers.backend-dispatcher-126 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.jsd:NA:1]: Status change from WaitingForReturnCode to Done 2024-01-11 11:13:21,954 cromwell-system-akka.dispatchers.engine-dispatcher-43 INFO - Assigned new job execution tokens to the following groups: 067c7f61: 1 2024-01-11 11:13:21,963 INFO - Request threw an exception on attempt #1. Retrying after 73 milliseconds org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 11:13:22,038 INFO - Request threw an exception on attempt #2. Retrying after 498 milliseconds org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 11:13:22,538 INFO - Request threw an exception on attempt #3. Retrying after 6807 milliseconds org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 11:13:29,347 INFO - Request method=GET uri=https://auth.docker.io/token?service=registry.docker.io&scope=repository:encodedcc/chip-seq-pipeline:pull headers= threw an exception on attempt #4. Giving up. org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 11:13:29,348 cromwell-system-akka.dispatchers.engine-dispatcher-34 WARN - BackendPreparationActor_for_067c7f61:chip.spr:0:1 [UUID(067c7f61)]: Docker lookup failed java.lang.Exception: Failed to get docker hash for encodedcc/chip-seq-pipeline:v2.2.1 Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at cromwell.engine.workflow.WorkflowDockerLookupActor.cromwell$engine$workflow$WorkflowDockerLookupActor$$handleLookupFailure(WorkflowDockerLookupActor.scala:222) at cromwell.engine.workflow.WorkflowDockerLookupActor$$anonfun$3.applyOrElse(WorkflowDockerLookupActor.scala:88) at cromwell.engine.workflow.WorkflowDockerLookupActor$$anonfun$3.applyOrElse(WorkflowDockerLookupActor.scala:73) at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35) at akka.actor.FSM.processEvent(FSM.scala:707) at akka.actor.FSM.processEvent$(FSM.scala:704) at cromwell.engine.workflow.WorkflowDockerLookupActor.akka$actor$LoggingFSM$$super$processEvent(WorkflowDockerLookupActor.scala:40) at akka.actor.LoggingFSM.processEvent(FSM.scala:847) at akka.actor.LoggingFSM.processEvent$(FSM.scala:829) at cromwell.engine.workflow.WorkflowDockerLookupActor.processEvent(WorkflowDockerLookupActor.scala:40) at akka.actor.FSM.akka$actor$FSM$$processMsg(FSM.scala:701) at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:695) at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35) at cromwell.docker.DockerClientHelper$$anonfun$dockerResponseReceive$1.applyOrElse(DockerClientHelper.scala:16) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:269) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:270) at akka.actor.Actor.aroundReceive(Actor.scala:539) at akka.actor.Actor.aroundReceive$(Actor.scala:537) at cromwell.engine.workflow.WorkflowDockerLookupActor.aroundReceive(WorkflowDockerLookupActor.scala:40) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:614) at akka.actor.ActorCell.invoke(ActorCell.scala:583) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:268) at akka.dispatch.Mailbox.run(Mailbox.scala:229) at akka.dispatch.Mailbox.exec(Mailbox.scala:241) at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) 2024-01-11 11:13:29,350 cromwell-system-akka.dispatchers.engine-dispatcher-34 INFO - BT-322 067c7f61:chip.spr:0:1 is not eligible for call caching 2024-01-11 11:13:29,350 cromwell-system-akka.dispatchers.engine-dispatcher-34 WARN - BT-322 067c7f61:chip.spr:0:1 disabling call caching due to error 2024-01-11 11:13:29,351 cromwell-system-akka.dispatchers.backend-dispatcher-135 WARN - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.spr:0:1]: Unrecognized runtime attribute keys: disks, time, memory, cpu 2024-01-11 11:13:29,360 cromwell-system-akka.dispatchers.backend-dispatcher-135 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.spr:0:1]: `set -e python3 $(which encode_task_spr.py) \ /cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-spr/shard-0/inputs/-1550963907/ENCFF169ZIP.tagAlign.gz \ --pseudoreplication-random-seed 0 \` 2024-01-11 11:13:29,364 cromwell-system-akka.dispatchers.backend-dispatcher-135 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.spr:0:1]: executing: rm -f /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-spr/shard-0/execution/docker_cid if [ 'true' == 'true' ] && [ 'singularity' == 'docker' ] || \ [ 'true' == 'false' ] && [ 'true' == 'true' ] && [ ! -z 'encodedcc/chip-seq-pipeline:v2.2.1' ] then docker run -i --cidfile=/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-spr/shard-0/execution/docker_cid --user= --entrypoint=/bin/bash \ --volume=/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-spr/shard-0:/cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-spr/shard-0:delegated encodedcc/chip-seq-pipeline:v2.2.1 /cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-spr/shard-0/execution/script rc=$(docker wait `cat /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-spr/shard-0/execution/docker_cid`) docker rm `cat /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-spr/shard-0/execution/docker_cid` else # recover GID lost due to Cromwell running chmod 777 on CWD chown :`stat -c '%G' /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-spr/shard-0` -R /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-spr/shard-0 chmod g+s /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-spr/shard-0 if [ 'true' == 'true' ] && [ 'singularity' == 'singularity' ] || \ [ 'true' == 'false' ] && [ 'true' == 'true' ] && [ ! -z 'https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif' ] then mkdir -p $HOME/.singularity/lock/ flock --exclusive --timeout 600 \ $HOME/.singularity/lock/`echo -n 'https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif' | md5sum | cut -d' ' -f1` \ singularity exec --containall https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif echo 'Successfully pulled https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif' singularity exec --cleanenv --home=`dirname /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-spr/shard-0` \ --bind=/lustre/home/acct-medzy/medzy-cai,/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-spr/shard-0:/cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-spr/shard-0 \ \ https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif /bin/bash /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-spr/shard-0/execution/script & echo $! > /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-spr/shard-0/execution/docker_cid else # remap paths between inside and outside of a docker container shopt -s nullglob sed -i 's#/cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-spr/shard-0#/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-spr/shard-0#g' /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-spr/shard-0/execution/script `dirname /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-spr/shard-0/execution/script`/write_*.tmp if [ 'true' == 'true' ] && [ 'singularity' == 'conda' ] || \ [ 'true' == 'false' ] && [ 'true' == 'true' ] && [ ! -z 'encd-chip' ] then conda run --name=encd-chip /bin/bash /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-spr/shard-0/execution/script & echo $! > /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-spr/shard-0/execution/docker_cid else /bin/bash /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-spr/shard-0/execution/script & echo $! > /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-spr/shard-0/execution/docker_cid fi fi touch /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-spr/shard-0/execution/docker_cid.not_docker wait `cat /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-spr/shard-0/execution/docker_cid` rc=`echo $?` fi exit $rc 2024-01-11 11:13:33,805 cromwell-system-akka.dispatchers.backend-dispatcher-135 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.spr:0:1]: job id: 1793459 2024-01-11 11:13:33,805 cromwell-system-akka.dispatchers.backend-dispatcher-138 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.spr:0:1]: Status change from - to WaitingForReturnCode 2024-01-11 11:14:30,434 cromwell-system-akka.dispatchers.backend-dispatcher-137 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.spr:0:1]: Status change from WaitingForReturnCode to Done 2024-01-11 11:14:31,955 cromwell-system-akka.dispatchers.engine-dispatcher-45 INFO - Assigned new job execution tokens to the following groups: 067c7f61: 1 2024-01-11 11:14:31,961 INFO - Request threw an exception on attempt #1. Retrying after 659 milliseconds org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 11:14:32,621 INFO - Request threw an exception on attempt #2. Retrying after 2892 milliseconds org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 11:14:35,516 INFO - Request threw an exception on attempt #3. Retrying after 1340 milliseconds org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 11:14:36,858 INFO - Request method=GET uri=https://auth.docker.io/token?service=registry.docker.io&scope=repository:encodedcc/chip-seq-pipeline:pull headers= threw an exception on attempt #4. Giving up. org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 11:14:36,859 cromwell-system-akka.dispatchers.engine-dispatcher-4 WARN - BackendPreparationActor_for_067c7f61:chip.xcor:0:1 [UUID(067c7f61)]: Docker lookup failed java.lang.Exception: Failed to get docker hash for encodedcc/chip-seq-pipeline:v2.2.1 Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at cromwell.engine.workflow.WorkflowDockerLookupActor.cromwell$engine$workflow$WorkflowDockerLookupActor$$handleLookupFailure(WorkflowDockerLookupActor.scala:222) at cromwell.engine.workflow.WorkflowDockerLookupActor$$anonfun$3.applyOrElse(WorkflowDockerLookupActor.scala:88) at cromwell.engine.workflow.WorkflowDockerLookupActor$$anonfun$3.applyOrElse(WorkflowDockerLookupActor.scala:73) at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35) at akka.actor.FSM.processEvent(FSM.scala:707) at akka.actor.FSM.processEvent$(FSM.scala:704) at cromwell.engine.workflow.WorkflowDockerLookupActor.akka$actor$LoggingFSM$$super$processEvent(WorkflowDockerLookupActor.scala:40) at akka.actor.LoggingFSM.processEvent(FSM.scala:847) at akka.actor.LoggingFSM.processEvent$(FSM.scala:829) at cromwell.engine.workflow.WorkflowDockerLookupActor.processEvent(WorkflowDockerLookupActor.scala:40) at akka.actor.FSM.akka$actor$FSM$$processMsg(FSM.scala:701) at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:695) at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35) at cromwell.docker.DockerClientHelper$$anonfun$dockerResponseReceive$1.applyOrElse(DockerClientHelper.scala:16) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:269) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:270) at akka.actor.Actor.aroundReceive(Actor.scala:539) at akka.actor.Actor.aroundReceive$(Actor.scala:537) at cromwell.engine.workflow.WorkflowDockerLookupActor.aroundReceive(WorkflowDockerLookupActor.scala:40) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:614) at akka.actor.ActorCell.invoke(ActorCell.scala:583) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:268) at akka.dispatch.Mailbox.run(Mailbox.scala:229) at akka.dispatch.Mailbox.exec(Mailbox.scala:241) at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) 2024-01-11 11:14:36,861 cromwell-system-akka.dispatchers.engine-dispatcher-4 INFO - BT-322 067c7f61:chip.xcor:0:1 is not eligible for call caching 2024-01-11 11:14:36,861 cromwell-system-akka.dispatchers.engine-dispatcher-4 WARN - BT-322 067c7f61:chip.xcor:0:1 disabling call caching due to error 2024-01-11 11:14:36,862 cromwell-system-akka.dispatchers.backend-dispatcher-140 WARN - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.xcor:0:1]: Unrecognized runtime attribute keys: disks, time, memory, cpu 2024-01-11 11:14:36,872 cromwell-system-akka.dispatchers.backend-dispatcher-140 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.xcor:0:1]: `set -e python3 $(which encode_task_xcor.py) \ /cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-xcor/shard-0/inputs/-1550963907/ENCFF169ZIP.tagAlign.gz \ \ --mito-chr-name chrM \ --subsample 15000000 \ --chip-seq-type tf \ --exclusion-range-min -500 \ \ --subsample 15000000 \ --nth 2` 2024-01-11 11:14:36,878 cromwell-system-akka.dispatchers.backend-dispatcher-140 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.xcor:0:1]: executing: rm -f /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-xcor/shard-0/execution/docker_cid if [ 'true' == 'true' ] && [ 'singularity' == 'docker' ] || \ [ 'true' == 'false' ] && [ 'true' == 'true' ] && [ ! -z 'encodedcc/chip-seq-pipeline:v2.2.1' ] then docker run -i --cidfile=/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-xcor/shard-0/execution/docker_cid --user= --entrypoint=/bin/bash \ --volume=/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-xcor/shard-0:/cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-xcor/shard-0:delegated encodedcc/chip-seq-pipeline:v2.2.1 /cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-xcor/shard-0/execution/script rc=$(docker wait `cat /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-xcor/shard-0/execution/docker_cid`) docker rm `cat /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-xcor/shard-0/execution/docker_cid` else # recover GID lost due to Cromwell running chmod 777 on CWD chown :`stat -c '%G' /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-xcor/shard-0` -R /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-xcor/shard-0 chmod g+s /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-xcor/shard-0 if [ 'true' == 'true' ] && [ 'singularity' == 'singularity' ] || \ [ 'true' == 'false' ] && [ 'true' == 'true' ] && [ ! -z 'https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif' ] then mkdir -p $HOME/.singularity/lock/ flock --exclusive --timeout 600 \ $HOME/.singularity/lock/`echo -n 'https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif' | md5sum | cut -d' ' -f1` \ singularity exec --containall https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif echo 'Successfully pulled https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif' singularity exec --cleanenv --home=`dirname /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-xcor/shard-0` \ --bind=/lustre/home/acct-medzy/medzy-cai,/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-xcor/shard-0:/cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-xcor/shard-0 \ \ https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif /bin/bash /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-xcor/shard-0/execution/script & echo $! > /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-xcor/shard-0/execution/docker_cid else # remap paths between inside and outside of a docker container shopt -s nullglob sed -i 's#/cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-xcor/shard-0#/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-xcor/shard-0#g' /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-xcor/shard-0/execution/script `dirname /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-xcor/shard-0/execution/script`/write_*.tmp if [ 'true' == 'true' ] && [ 'singularity' == 'conda' ] || \ [ 'true' == 'false' ] && [ 'true' == 'true' ] && [ ! -z 'encd-chip-spp' ] then conda run --name=encd-chip-spp /bin/bash /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-xcor/shard-0/execution/script & echo $! > /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-xcor/shard-0/execution/docker_cid else /bin/bash /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-xcor/shard-0/execution/script & echo $! > /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-xcor/shard-0/execution/docker_cid fi fi touch /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-xcor/shard-0/execution/docker_cid.not_docker wait `cat /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-xcor/shard-0/execution/docker_cid` rc=`echo $?` fi exit $rc 2024-01-11 11:14:38,805 cromwell-system-akka.dispatchers.backend-dispatcher-141 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.xcor:0:1]: job id: 1793808 2024-01-11 11:14:38,806 cromwell-system-akka.dispatchers.backend-dispatcher-142 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.xcor:0:1]: Status change from - to WaitingForReturnCode 2024-01-11 11:21:29,875 cromwell-system-akka.dispatchers.backend-dispatcher-144 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.xcor:0:1]: Status change from WaitingForReturnCode to Done 2024-01-11 11:21:29,954 cromwell-system-akka.dispatchers.engine-dispatcher-48 INFO - Assigned new job execution tokens to the following groups: 067c7f61: 1 2024-01-11 11:21:29,963 INFO - Request threw an exception on attempt #1. Retrying after 24 milliseconds org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 11:21:29,989 INFO - Request threw an exception on attempt #2. Retrying after 460 milliseconds org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 11:21:30,451 INFO - Request threw an exception on attempt #3. Retrying after 3746 milliseconds org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 11:21:34,200 INFO - Request method=GET uri=https://auth.docker.io/token?service=registry.docker.io&scope=repository:encodedcc/chip-seq-pipeline:pull headers= threw an exception on attempt #4. Giving up. org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 11:21:34,201 cromwell-system-akka.dispatchers.engine-dispatcher-34 WARN - BackendPreparationActor_for_067c7f61:chip.spr:1:1 [UUID(067c7f61)]: Docker lookup failed java.lang.Exception: Failed to get docker hash for encodedcc/chip-seq-pipeline:v2.2.1 Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at cromwell.engine.workflow.WorkflowDockerLookupActor.cromwell$engine$workflow$WorkflowDockerLookupActor$$handleLookupFailure(WorkflowDockerLookupActor.scala:222) at cromwell.engine.workflow.WorkflowDockerLookupActor$$anonfun$3.applyOrElse(WorkflowDockerLookupActor.scala:88) at cromwell.engine.workflow.WorkflowDockerLookupActor$$anonfun$3.applyOrElse(WorkflowDockerLookupActor.scala:73) at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35) at akka.actor.FSM.processEvent(FSM.scala:707) at akka.actor.FSM.processEvent$(FSM.scala:704) at cromwell.engine.workflow.WorkflowDockerLookupActor.akka$actor$LoggingFSM$$super$processEvent(WorkflowDockerLookupActor.scala:40) at akka.actor.LoggingFSM.processEvent(FSM.scala:847) at akka.actor.LoggingFSM.processEvent$(FSM.scala:829) at cromwell.engine.workflow.WorkflowDockerLookupActor.processEvent(WorkflowDockerLookupActor.scala:40) at akka.actor.FSM.akka$actor$FSM$$processMsg(FSM.scala:701) at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:695) at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35) at cromwell.docker.DockerClientHelper$$anonfun$dockerResponseReceive$1.applyOrElse(DockerClientHelper.scala:16) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:269) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:270) at akka.actor.Actor.aroundReceive(Actor.scala:539) at akka.actor.Actor.aroundReceive$(Actor.scala:537) at cromwell.engine.workflow.WorkflowDockerLookupActor.aroundReceive(WorkflowDockerLookupActor.scala:40) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:614) at akka.actor.ActorCell.invoke(ActorCell.scala:583) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:268) at akka.dispatch.Mailbox.run(Mailbox.scala:229) at akka.dispatch.Mailbox.exec(Mailbox.scala:241) at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) 2024-01-11 11:21:34,203 cromwell-system-akka.dispatchers.engine-dispatcher-34 INFO - BT-322 067c7f61:chip.spr:1:1 is not eligible for call caching 2024-01-11 11:21:34,203 cromwell-system-akka.dispatchers.engine-dispatcher-34 WARN - BT-322 067c7f61:chip.spr:1:1 disabling call caching due to error 2024-01-11 11:21:34,204 cromwell-system-akka.dispatchers.backend-dispatcher-148 WARN - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.spr:1:1]: Unrecognized runtime attribute keys: disks, time, memory, cpu 2024-01-11 11:21:34,211 cromwell-system-akka.dispatchers.backend-dispatcher-148 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.spr:1:1]: `set -e python3 $(which encode_task_spr.py) \ /cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-spr/shard-1/inputs/-1558723266/ENCFF597JXK.tagAlign.gz \ --pseudoreplication-random-seed 0 \` 2024-01-11 11:21:34,216 cromwell-system-akka.dispatchers.backend-dispatcher-148 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.spr:1:1]: executing: rm -f /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-spr/shard-1/execution/docker_cid if [ 'true' == 'true' ] && [ 'singularity' == 'docker' ] || \ [ 'true' == 'false' ] && [ 'true' == 'true' ] && [ ! -z 'encodedcc/chip-seq-pipeline:v2.2.1' ] then docker run -i --cidfile=/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-spr/shard-1/execution/docker_cid --user= --entrypoint=/bin/bash \ --volume=/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-spr/shard-1:/cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-spr/shard-1:delegated encodedcc/chip-seq-pipeline:v2.2.1 /cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-spr/shard-1/execution/script rc=$(docker wait `cat /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-spr/shard-1/execution/docker_cid`) docker rm `cat /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-spr/shard-1/execution/docker_cid` else # recover GID lost due to Cromwell running chmod 777 on CWD chown :`stat -c '%G' /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-spr/shard-1` -R /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-spr/shard-1 chmod g+s /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-spr/shard-1 if [ 'true' == 'true' ] && [ 'singularity' == 'singularity' ] || \ [ 'true' == 'false' ] && [ 'true' == 'true' ] && [ ! -z 'https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif' ] then mkdir -p $HOME/.singularity/lock/ flock --exclusive --timeout 600 \ $HOME/.singularity/lock/`echo -n 'https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif' | md5sum | cut -d' ' -f1` \ singularity exec --containall https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif echo 'Successfully pulled https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif' singularity exec --cleanenv --home=`dirname /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-spr/shard-1` \ --bind=/lustre/home/acct-medzy/medzy-cai,/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-spr/shard-1:/cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-spr/shard-1 \ \ https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif /bin/bash /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-spr/shard-1/execution/script & echo $! > /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-spr/shard-1/execution/docker_cid else # remap paths between inside and outside of a docker container shopt -s nullglob sed -i 's#/cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-spr/shard-1#/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-spr/shard-1#g' /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-spr/shard-1/execution/script `dirname /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-spr/shard-1/execution/script`/write_*.tmp if [ 'true' == 'true' ] && [ 'singularity' == 'conda' ] || \ [ 'true' == 'false' ] && [ 'true' == 'true' ] && [ ! -z 'encd-chip' ] then conda run --name=encd-chip /bin/bash /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-spr/shard-1/execution/script & echo $! > /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-spr/shard-1/execution/docker_cid else /bin/bash /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-spr/shard-1/execution/script & echo $! > /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-spr/shard-1/execution/docker_cid fi fi touch /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-spr/shard-1/execution/docker_cid.not_docker wait `cat /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-spr/shard-1/execution/docker_cid` rc=`echo $?` fi exit $rc 2024-01-11 11:21:38,805 cromwell-system-akka.dispatchers.backend-dispatcher-148 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.spr:1:1]: job id: 1796306 2024-01-11 11:21:38,806 cromwell-system-akka.dispatchers.backend-dispatcher-152 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.spr:1:1]: Status change from - to WaitingForReturnCode 2024-01-11 11:22:25,755 cromwell-system-akka.dispatchers.backend-dispatcher-151 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.spr:1:1]: Status change from WaitingForReturnCode to Done 2024-01-11 11:22:25,954 cromwell-system-akka.dispatchers.engine-dispatcher-46 INFO - Assigned new job execution tokens to the following groups: 067c7f61: 1 2024-01-11 11:22:25,958 INFO - Request threw an exception on attempt #1. Retrying after 918 milliseconds org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 11:22:26,878 INFO - Request threw an exception on attempt #2. Retrying after 250 milliseconds org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 11:22:27,129 INFO - Request threw an exception on attempt #3. Retrying after 5953 milliseconds org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 11:22:32,316 cromwell-system-akka.dispatchers.engine-dispatcher-45 INFO - WorkflowExecutionActor-067c7f61-ef2a-4b2b-b54f-bb1bce5217af [UUID(067c7f61)]: Starting chip.pool_ta_pr2, chip.pool_ta_pr1 2024-01-11 11:22:33,085 INFO - Request method=GET uri=https://auth.docker.io/token?service=registry.docker.io&scope=repository:encodedcc/chip-seq-pipeline:pull headers= threw an exception on attempt #4. Giving up. org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 11:22:33,087 cromwell-system-akka.dispatchers.engine-dispatcher-45 WARN - BackendPreparationActor_for_067c7f61:chip.pool_ta:-1:1 [UUID(067c7f61)]: Docker lookup failed java.lang.Exception: Failed to get docker hash for encodedcc/chip-seq-pipeline:v2.2.1 Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at cromwell.engine.workflow.WorkflowDockerLookupActor.cromwell$engine$workflow$WorkflowDockerLookupActor$$handleLookupFailure(WorkflowDockerLookupActor.scala:222) at cromwell.engine.workflow.WorkflowDockerLookupActor$$anonfun$3.applyOrElse(WorkflowDockerLookupActor.scala:88) at cromwell.engine.workflow.WorkflowDockerLookupActor$$anonfun$3.applyOrElse(WorkflowDockerLookupActor.scala:73) at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35) at akka.actor.FSM.processEvent(FSM.scala:707) at akka.actor.FSM.processEvent$(FSM.scala:704) at cromwell.engine.workflow.WorkflowDockerLookupActor.akka$actor$LoggingFSM$$super$processEvent(WorkflowDockerLookupActor.scala:40) at akka.actor.LoggingFSM.processEvent(FSM.scala:847) at akka.actor.LoggingFSM.processEvent$(FSM.scala:829) at cromwell.engine.workflow.WorkflowDockerLookupActor.processEvent(WorkflowDockerLookupActor.scala:40) at akka.actor.FSM.akka$actor$FSM$$processMsg(FSM.scala:701) at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:695) at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35) at cromwell.docker.DockerClientHelper$$anonfun$dockerResponseReceive$1.applyOrElse(DockerClientHelper.scala:16) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:269) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:270) at akka.actor.Actor.aroundReceive(Actor.scala:539) at akka.actor.Actor.aroundReceive$(Actor.scala:537) at cromwell.engine.workflow.WorkflowDockerLookupActor.aroundReceive(WorkflowDockerLookupActor.scala:40) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:614) at akka.actor.ActorCell.invoke(ActorCell.scala:583) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:268) at akka.dispatch.Mailbox.run(Mailbox.scala:229) at akka.dispatch.Mailbox.exec(Mailbox.scala:241) at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) 2024-01-11 11:22:33,088 cromwell-system-akka.dispatchers.engine-dispatcher-45 INFO - BT-322 067c7f61:chip.pool_ta:-1:1 is not eligible for call caching 2024-01-11 11:22:33,088 cromwell-system-akka.dispatchers.engine-dispatcher-45 WARN - BT-322 067c7f61:chip.pool_ta:-1:1 disabling call caching due to error 2024-01-11 11:22:33,089 cromwell-system-akka.dispatchers.backend-dispatcher-153 WARN - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.pool_ta:NA:1]: Unrecognized runtime attribute keys: disks, time, memory, cpu 2024-01-11 11:22:33,104 cromwell-system-akka.dispatchers.backend-dispatcher-153 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.pool_ta:NA:1]: `set -e python3 $(which encode_task_pool_ta.py) \ /cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta/inputs/-1550963907/ENCFF169ZIP.tagAlign.gz /cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta/inputs/-1558723266/ENCFF597JXK.tagAlign.gz \ --prefix rep \` 2024-01-11 11:22:33,109 cromwell-system-akka.dispatchers.backend-dispatcher-153 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.pool_ta:NA:1]: executing: rm -f /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta/execution/docker_cid if [ 'true' == 'true' ] && [ 'singularity' == 'docker' ] || \ [ 'true' == 'false' ] && [ 'true' == 'true' ] && [ ! -z 'encodedcc/chip-seq-pipeline:v2.2.1' ] then docker run -i --cidfile=/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta/execution/docker_cid --user= --entrypoint=/bin/bash \ --volume=/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta:/cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta:delegated encodedcc/chip-seq-pipeline:v2.2.1 /cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta/execution/script rc=$(docker wait `cat /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta/execution/docker_cid`) docker rm `cat /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta/execution/docker_cid` else # recover GID lost due to Cromwell running chmod 777 on CWD chown :`stat -c '%G' /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta` -R /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta chmod g+s /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta if [ 'true' == 'true' ] && [ 'singularity' == 'singularity' ] || \ [ 'true' == 'false' ] && [ 'true' == 'true' ] && [ ! -z 'https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif' ] then mkdir -p $HOME/.singularity/lock/ flock --exclusive --timeout 600 \ $HOME/.singularity/lock/`echo -n 'https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif' | md5sum | cut -d' ' -f1` \ singularity exec --containall https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif echo 'Successfully pulled https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif' singularity exec --cleanenv --home=`dirname /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta` \ --bind=/lustre/home/acct-medzy/medzy-cai,/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta:/cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta \ \ https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif /bin/bash /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta/execution/script & echo $! > /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta/execution/docker_cid else # remap paths between inside and outside of a docker container shopt -s nullglob sed -i 's#/cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta#/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta#g' /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta/execution/script `dirname /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta/execution/script`/write_*.tmp if [ 'true' == 'true' ] && [ 'singularity' == 'conda' ] || \ [ 'true' == 'false' ] && [ 'true' == 'true' ] && [ ! -z 'encd-chip' ] then conda run --name=encd-chip /bin/bash /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta/execution/script & echo $! > /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta/execution/docker_cid else /bin/bash /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta/execution/script & echo $! > /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta/execution/docker_cid fi fi touch /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta/execution/docker_cid.not_docker wait `cat /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta/execution/docker_cid` rc=`echo $?` fi exit $rc 2024-01-11 11:22:33,805 cromwell-system-akka.dispatchers.backend-dispatcher-153 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.pool_ta:NA:1]: job id: 1796746 2024-01-11 11:22:33,806 cromwell-system-akka.dispatchers.backend-dispatcher-157 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.pool_ta:NA:1]: Status change from - to WaitingForReturnCode 2024-01-11 11:23:30,295 cromwell-system-akka.dispatchers.backend-dispatcher-157 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.pool_ta:NA:1]: Status change from WaitingForReturnCode to Done 2024-01-11 11:23:31,954 cromwell-system-akka.dispatchers.engine-dispatcher-29 INFO - Assigned new job execution tokens to the following groups: 067c7f61: 1 2024-01-11 11:23:31,959 INFO - Request threw an exception on attempt #1. Retrying after 378 milliseconds org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 11:23:32,339 INFO - Request threw an exception on attempt #2. Retrying after 1358 milliseconds org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 11:23:33,699 INFO - Request threw an exception on attempt #3. Retrying after 5133 milliseconds org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 11:23:38,834 INFO - Request method=GET uri=https://auth.docker.io/token?service=registry.docker.io&scope=repository:encodedcc/chip-seq-pipeline:pull headers= threw an exception on attempt #4. Giving up. org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 11:23:38,835 cromwell-system-akka.dispatchers.engine-dispatcher-44 WARN - BackendPreparationActor_for_067c7f61:chip.xcor:1:1 [UUID(067c7f61)]: Docker lookup failed java.lang.Exception: Failed to get docker hash for encodedcc/chip-seq-pipeline:v2.2.1 Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at cromwell.engine.workflow.WorkflowDockerLookupActor.cromwell$engine$workflow$WorkflowDockerLookupActor$$handleLookupFailure(WorkflowDockerLookupActor.scala:222) at cromwell.engine.workflow.WorkflowDockerLookupActor$$anonfun$3.applyOrElse(WorkflowDockerLookupActor.scala:88) at cromwell.engine.workflow.WorkflowDockerLookupActor$$anonfun$3.applyOrElse(WorkflowDockerLookupActor.scala:73) at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35) at akka.actor.FSM.processEvent(FSM.scala:707) at akka.actor.FSM.processEvent$(FSM.scala:704) at cromwell.engine.workflow.WorkflowDockerLookupActor.akka$actor$LoggingFSM$$super$processEvent(WorkflowDockerLookupActor.scala:40) at akka.actor.LoggingFSM.processEvent(FSM.scala:847) at akka.actor.LoggingFSM.processEvent$(FSM.scala:829) at cromwell.engine.workflow.WorkflowDockerLookupActor.processEvent(WorkflowDockerLookupActor.scala:40) at akka.actor.FSM.akka$actor$FSM$$processMsg(FSM.scala:701) at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:695) at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35) at cromwell.docker.DockerClientHelper$$anonfun$dockerResponseReceive$1.applyOrElse(DockerClientHelper.scala:16) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:269) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:270) at akka.actor.Actor.aroundReceive(Actor.scala:539) at akka.actor.Actor.aroundReceive$(Actor.scala:537) at cromwell.engine.workflow.WorkflowDockerLookupActor.aroundReceive(WorkflowDockerLookupActor.scala:40) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:614) at akka.actor.ActorCell.invoke(ActorCell.scala:583) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:268) at akka.dispatch.Mailbox.run(Mailbox.scala:229) at akka.dispatch.Mailbox.exec(Mailbox.scala:241) at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) 2024-01-11 11:23:38,836 cromwell-system-akka.dispatchers.engine-dispatcher-44 INFO - BT-322 067c7f61:chip.xcor:1:1 is not eligible for call caching 2024-01-11 11:23:38,836 cromwell-system-akka.dispatchers.engine-dispatcher-44 WARN - BT-322 067c7f61:chip.xcor:1:1 disabling call caching due to error 2024-01-11 11:23:38,837 cromwell-system-akka.dispatchers.backend-dispatcher-158 WARN - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.xcor:1:1]: Unrecognized runtime attribute keys: disks, time, memory, cpu 2024-01-11 11:23:38,846 cromwell-system-akka.dispatchers.backend-dispatcher-158 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.xcor:1:1]: `set -e python3 $(which encode_task_xcor.py) \ /cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-xcor/shard-1/inputs/-1558723266/ENCFF597JXK.tagAlign.gz \ \ --mito-chr-name chrM \ --subsample 15000000 \ --chip-seq-type tf \ --exclusion-range-min -500 \ \ --subsample 15000000 \ --nth 2` 2024-01-11 11:23:38,851 cromwell-system-akka.dispatchers.backend-dispatcher-158 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.xcor:1:1]: executing: rm -f /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-xcor/shard-1/execution/docker_cid if [ 'true' == 'true' ] && [ 'singularity' == 'docker' ] || \ [ 'true' == 'false' ] && [ 'true' == 'true' ] && [ ! -z 'encodedcc/chip-seq-pipeline:v2.2.1' ] then docker run -i --cidfile=/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-xcor/shard-1/execution/docker_cid --user= --entrypoint=/bin/bash \ --volume=/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-xcor/shard-1:/cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-xcor/shard-1:delegated encodedcc/chip-seq-pipeline:v2.2.1 /cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-xcor/shard-1/execution/script rc=$(docker wait `cat /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-xcor/shard-1/execution/docker_cid`) docker rm `cat /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-xcor/shard-1/execution/docker_cid` else # recover GID lost due to Cromwell running chmod 777 on CWD chown :`stat -c '%G' /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-xcor/shard-1` -R /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-xcor/shard-1 chmod g+s /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-xcor/shard-1 if [ 'true' == 'true' ] && [ 'singularity' == 'singularity' ] || \ [ 'true' == 'false' ] && [ 'true' == 'true' ] && [ ! -z 'https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif' ] then mkdir -p $HOME/.singularity/lock/ flock --exclusive --timeout 600 \ $HOME/.singularity/lock/`echo -n 'https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif' | md5sum | cut -d' ' -f1` \ singularity exec --containall https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif echo 'Successfully pulled https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif' singularity exec --cleanenv --home=`dirname /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-xcor/shard-1` \ --bind=/lustre/home/acct-medzy/medzy-cai,/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-xcor/shard-1:/cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-xcor/shard-1 \ \ https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif /bin/bash /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-xcor/shard-1/execution/script & echo $! > /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-xcor/shard-1/execution/docker_cid else # remap paths between inside and outside of a docker container shopt -s nullglob sed -i 's#/cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-xcor/shard-1#/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-xcor/shard-1#g' /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-xcor/shard-1/execution/script `dirname /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-xcor/shard-1/execution/script`/write_*.tmp if [ 'true' == 'true' ] && [ 'singularity' == 'conda' ] || \ [ 'true' == 'false' ] && [ 'true' == 'true' ] && [ ! -z 'encd-chip-spp' ] then conda run --name=encd-chip-spp /bin/bash /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-xcor/shard-1/execution/script & echo $! > /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-xcor/shard-1/execution/docker_cid else /bin/bash /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-xcor/shard-1/execution/script & echo $! > /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-xcor/shard-1/execution/docker_cid fi fi touch /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-xcor/shard-1/execution/docker_cid.not_docker wait `cat /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-xcor/shard-1/execution/docker_cid` rc=`echo $?` fi exit $rc 2024-01-11 11:23:43,806 cromwell-system-akka.dispatchers.backend-dispatcher-159 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.xcor:1:1]: job id: 1797192 2024-01-11 11:23:43,809 cromwell-system-akka.dispatchers.backend-dispatcher-159 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.xcor:1:1]: Status change from - to WaitingForReturnCode 2024-01-11 11:29:52,375 cromwell-system-akka.dispatchers.backend-dispatcher-159 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.xcor:1:1]: Status change from WaitingForReturnCode to Done 2024-01-11 11:29:53,955 cromwell-system-akka.dispatchers.engine-dispatcher-46 INFO - Assigned new job execution tokens to the following groups: 067c7f61: 1 2024-01-11 11:29:53,962 INFO - Request threw an exception on attempt #1. Retrying after 147 milliseconds org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 11:29:54,111 INFO - Request threw an exception on attempt #2. Retrying after 1115 milliseconds org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 11:29:55,228 INFO - Request threw an exception on attempt #3. Retrying after 1091 milliseconds org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 11:29:56,321 INFO - Request method=GET uri=https://auth.docker.io/token?service=registry.docker.io&scope=repository:encodedcc/chip-seq-pipeline:pull headers= threw an exception on attempt #4. Giving up. org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 11:29:56,322 cromwell-system-akka.dispatchers.engine-dispatcher-46 WARN - BackendPreparationActor_for_067c7f61:chip.pool_ta_ctl:-1:1 [UUID(067c7f61)]: Docker lookup failed java.lang.Exception: Failed to get docker hash for encodedcc/chip-seq-pipeline:v2.2.1 Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at cromwell.engine.workflow.WorkflowDockerLookupActor.cromwell$engine$workflow$WorkflowDockerLookupActor$$handleLookupFailure(WorkflowDockerLookupActor.scala:222) at cromwell.engine.workflow.WorkflowDockerLookupActor$$anonfun$3.applyOrElse(WorkflowDockerLookupActor.scala:88) at cromwell.engine.workflow.WorkflowDockerLookupActor$$anonfun$3.applyOrElse(WorkflowDockerLookupActor.scala:73) at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35) at akka.actor.FSM.processEvent(FSM.scala:707) at akka.actor.FSM.processEvent$(FSM.scala:704) at cromwell.engine.workflow.WorkflowDockerLookupActor.akka$actor$LoggingFSM$$super$processEvent(WorkflowDockerLookupActor.scala:40) at akka.actor.LoggingFSM.processEvent(FSM.scala:847) at akka.actor.LoggingFSM.processEvent$(FSM.scala:829) at cromwell.engine.workflow.WorkflowDockerLookupActor.processEvent(WorkflowDockerLookupActor.scala:40) at akka.actor.FSM.akka$actor$FSM$$processMsg(FSM.scala:701) at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:695) at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35) at cromwell.docker.DockerClientHelper$$anonfun$dockerResponseReceive$1.applyOrElse(DockerClientHelper.scala:16) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:269) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:270) at akka.actor.Actor.aroundReceive(Actor.scala:539) at akka.actor.Actor.aroundReceive$(Actor.scala:537) at cromwell.engine.workflow.WorkflowDockerLookupActor.aroundReceive(WorkflowDockerLookupActor.scala:40) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:614) at akka.actor.ActorCell.invoke(ActorCell.scala:583) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:268) at akka.dispatch.Mailbox.run(Mailbox.scala:229) at akka.dispatch.Mailbox.exec(Mailbox.scala:241) at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) 2024-01-11 11:29:56,323 cromwell-system-akka.dispatchers.engine-dispatcher-46 INFO - BT-322 067c7f61:chip.pool_ta_ctl:-1:1 is not eligible for call caching 2024-01-11 11:29:56,323 cromwell-system-akka.dispatchers.engine-dispatcher-46 WARN - BT-322 067c7f61:chip.pool_ta_ctl:-1:1 disabling call caching due to error 2024-01-11 11:29:56,324 cromwell-system-akka.dispatchers.backend-dispatcher-165 WARN - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.pool_ta_ctl:NA:1]: Unrecognized runtime attribute keys: disks, time, memory, cpu 2024-01-11 11:29:56,342 cromwell-system-akka.dispatchers.backend-dispatcher-165 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.pool_ta_ctl:NA:1]: `set -e python3 $(which encode_task_pool_ta.py) \ /cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_ctl/inputs/-1193497575/ENCFF378YAW.tagAlign.gz /cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_ctl/inputs/-1201256934/ENCFF267FMB.tagAlign.gz \ --prefix ctl \` 2024-01-11 11:29:56,346 cromwell-system-akka.dispatchers.backend-dispatcher-165 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.pool_ta_ctl:NA:1]: executing: rm -f /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_ctl/execution/docker_cid if [ 'true' == 'true' ] && [ 'singularity' == 'docker' ] || \ [ 'true' == 'false' ] && [ 'true' == 'true' ] && [ ! -z 'encodedcc/chip-seq-pipeline:v2.2.1' ] then docker run -i --cidfile=/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_ctl/execution/docker_cid --user= --entrypoint=/bin/bash \ --volume=/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_ctl:/cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_ctl:delegated encodedcc/chip-seq-pipeline:v2.2.1 /cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_ctl/execution/script rc=$(docker wait `cat /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_ctl/execution/docker_cid`) docker rm `cat /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_ctl/execution/docker_cid` else # recover GID lost due to Cromwell running chmod 777 on CWD chown :`stat -c '%G' /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_ctl` -R /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_ctl chmod g+s /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_ctl if [ 'true' == 'true' ] && [ 'singularity' == 'singularity' ] || \ [ 'true' == 'false' ] && [ 'true' == 'true' ] && [ ! -z 'https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif' ] then mkdir -p $HOME/.singularity/lock/ flock --exclusive --timeout 600 \ $HOME/.singularity/lock/`echo -n 'https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif' | md5sum | cut -d' ' -f1` \ singularity exec --containall https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif echo 'Successfully pulled https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif' singularity exec --cleanenv --home=`dirname /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_ctl` \ --bind=/lustre/home/acct-medzy/medzy-cai,/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_ctl:/cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_ctl \ \ https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif /bin/bash /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_ctl/execution/script & echo $! > /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_ctl/execution/docker_cid else # remap paths between inside and outside of a docker container shopt -s nullglob sed -i 's#/cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_ctl#/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_ctl#g' /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_ctl/execution/script `dirname /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_ctl/execution/script`/write_*.tmp if [ 'true' == 'true' ] && [ 'singularity' == 'conda' ] || \ [ 'true' == 'false' ] && [ 'true' == 'true' ] && [ ! -z 'encd-chip' ] then conda run --name=encd-chip /bin/bash /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_ctl/execution/script & echo $! > /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_ctl/execution/docker_cid else /bin/bash /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_ctl/execution/script & echo $! > /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_ctl/execution/docker_cid fi fi touch /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_ctl/execution/docker_cid.not_docker wait `cat /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_ctl/execution/docker_cid` rc=`echo $?` fi exit $rc 2024-01-11 11:29:58,056 cromwell-system-akka.dispatchers.engine-dispatcher-34 INFO - WorkflowExecutionActor-067c7f61-ef2a-4b2b-b54f-bb1bce5217af [UUID(067c7f61)]: Starting chip.fraglen_mean 2024-01-11 11:29:58,804 cromwell-system-akka.dispatchers.backend-dispatcher-165 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.pool_ta_ctl:NA:1]: job id: 1798231 2024-01-11 11:29:58,805 cromwell-system-akka.dispatchers.backend-dispatcher-169 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.pool_ta_ctl:NA:1]: Status change from - to WaitingForReturnCode 2024-01-11 11:30:30,034 cromwell-system-akka.dispatchers.backend-dispatcher-168 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.pool_ta_ctl:NA:1]: Status change from WaitingForReturnCode to Done 2024-01-11 11:30:31,954 cromwell-system-akka.dispatchers.engine-dispatcher-36 INFO - Assigned new job execution tokens to the following groups: 067c7f61: 1 2024-01-11 11:30:31,958 INFO - Request threw an exception on attempt #1. Retrying after 623 milliseconds org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 11:30:32,583 INFO - Request threw an exception on attempt #2. Retrying after 2234 milliseconds org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 11:30:32,736 cromwell-system-akka.dispatchers.engine-dispatcher-29 INFO - WorkflowExecutionActor-067c7f61-ef2a-4b2b-b54f-bb1bce5217af [UUID(067c7f61)]: Starting chip.choose_ctl 2024-01-11 11:30:34,819 INFO - Request threw an exception on attempt #3. Retrying after 5426 milliseconds org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 11:30:40,247 INFO - Request method=GET uri=https://auth.docker.io/token?service=registry.docker.io&scope=repository:encodedcc/chip-seq-pipeline:pull headers= threw an exception on attempt #4. Giving up. org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 11:30:40,248 cromwell-system-akka.dispatchers.engine-dispatcher-5 WARN - BackendPreparationActor_for_067c7f61:chip.pool_ta_pr2:-1:1 [UUID(067c7f61)]: Docker lookup failed java.lang.Exception: Failed to get docker hash for encodedcc/chip-seq-pipeline:v2.2.1 Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at cromwell.engine.workflow.WorkflowDockerLookupActor.cromwell$engine$workflow$WorkflowDockerLookupActor$$handleLookupFailure(WorkflowDockerLookupActor.scala:222) at cromwell.engine.workflow.WorkflowDockerLookupActor$$anonfun$3.applyOrElse(WorkflowDockerLookupActor.scala:88) at cromwell.engine.workflow.WorkflowDockerLookupActor$$anonfun$3.applyOrElse(WorkflowDockerLookupActor.scala:73) at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35) at akka.actor.FSM.processEvent(FSM.scala:707) at akka.actor.FSM.processEvent$(FSM.scala:704) at cromwell.engine.workflow.WorkflowDockerLookupActor.akka$actor$LoggingFSM$$super$processEvent(WorkflowDockerLookupActor.scala:40) at akka.actor.LoggingFSM.processEvent(FSM.scala:847) at akka.actor.LoggingFSM.processEvent$(FSM.scala:829) at cromwell.engine.workflow.WorkflowDockerLookupActor.processEvent(WorkflowDockerLookupActor.scala:40) at akka.actor.FSM.akka$actor$FSM$$processMsg(FSM.scala:701) at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:695) at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35) at cromwell.docker.DockerClientHelper$$anonfun$dockerResponseReceive$1.applyOrElse(DockerClientHelper.scala:16) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:269) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:270) at akka.actor.Actor.aroundReceive(Actor.scala:539) at akka.actor.Actor.aroundReceive$(Actor.scala:537) at cromwell.engine.workflow.WorkflowDockerLookupActor.aroundReceive(WorkflowDockerLookupActor.scala:40) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:614) at akka.actor.ActorCell.invoke(ActorCell.scala:583) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:268) at akka.dispatch.Mailbox.run(Mailbox.scala:229) at akka.dispatch.Mailbox.exec(Mailbox.scala:241) at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) 2024-01-11 11:30:40,250 cromwell-system-akka.dispatchers.engine-dispatcher-5 INFO - BT-322 067c7f61:chip.pool_ta_pr2:-1:1 is not eligible for call caching 2024-01-11 11:30:40,250 cromwell-system-akka.dispatchers.engine-dispatcher-5 WARN - BT-322 067c7f61:chip.pool_ta_pr2:-1:1 disabling call caching due to error 2024-01-11 11:30:40,251 cromwell-system-akka.dispatchers.backend-dispatcher-170 WARN - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.pool_ta_pr2:NA:1]: Unrecognized runtime attribute keys: disks, time, memory, cpu 2024-01-11 11:30:40,260 cromwell-system-akka.dispatchers.backend-dispatcher-170 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.pool_ta_pr2:NA:1]: `set -e python3 $(which encode_task_pool_ta.py) \ /cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_pr2/inputs/1647332612/ENCFF169ZIP.pr2.tagAlign.gz /cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_pr2/inputs/1639573253/ENCFF597JXK.pr2.tagAlign.gz \ --prefix rep-pr2 \` 2024-01-11 11:30:40,264 cromwell-system-akka.dispatchers.backend-dispatcher-170 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.pool_ta_pr2:NA:1]: executing: rm -f /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_pr2/execution/docker_cid if [ 'true' == 'true' ] && [ 'singularity' == 'docker' ] || \ [ 'true' == 'false' ] && [ 'true' == 'true' ] && [ ! -z 'encodedcc/chip-seq-pipeline:v2.2.1' ] then docker run -i --cidfile=/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_pr2/execution/docker_cid --user= --entrypoint=/bin/bash \ --volume=/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_pr2:/cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_pr2:delegated encodedcc/chip-seq-pipeline:v2.2.1 /cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_pr2/execution/script rc=$(docker wait `cat /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_pr2/execution/docker_cid`) docker rm `cat /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_pr2/execution/docker_cid` else # recover GID lost due to Cromwell running chmod 777 on CWD chown :`stat -c '%G' /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_pr2` -R /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_pr2 chmod g+s /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_pr2 if [ 'true' == 'true' ] && [ 'singularity' == 'singularity' ] || \ [ 'true' == 'false' ] && [ 'true' == 'true' ] && [ ! -z 'https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif' ] then mkdir -p $HOME/.singularity/lock/ flock --exclusive --timeout 600 \ $HOME/.singularity/lock/`echo -n 'https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif' | md5sum | cut -d' ' -f1` \ singularity exec --containall https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif echo 'Successfully pulled https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif' singularity exec --cleanenv --home=`dirname /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_pr2` \ --bind=/lustre/home/acct-medzy/medzy-cai,/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_pr2:/cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_pr2 \ \ https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif /bin/bash /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_pr2/execution/script & echo $! > /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_pr2/execution/docker_cid else # remap paths between inside and outside of a docker container shopt -s nullglob sed -i 's#/cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_pr2#/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_pr2#g' /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_pr2/execution/script `dirname /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_pr2/execution/script`/write_*.tmp if [ 'true' == 'true' ] && [ 'singularity' == 'conda' ] || \ [ 'true' == 'false' ] && [ 'true' == 'true' ] && [ ! -z 'encd-chip' ] then conda run --name=encd-chip /bin/bash /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_pr2/execution/script & echo $! > /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_pr2/execution/docker_cid else /bin/bash /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_pr2/execution/script & echo $! > /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_pr2/execution/docker_cid fi fi touch /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_pr2/execution/docker_cid.not_docker wait `cat /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_pr2/execution/docker_cid` rc=`echo $?` fi exit $rc 2024-01-11 11:30:43,805 cromwell-system-akka.dispatchers.backend-dispatcher-170 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.pool_ta_pr2:NA:1]: job id: 1798706 2024-01-11 11:30:43,806 cromwell-system-akka.dispatchers.backend-dispatcher-172 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.pool_ta_pr2:NA:1]: Status change from - to WaitingForReturnCode 2024-01-11 11:31:14,485 cromwell-system-akka.dispatchers.backend-dispatcher-172 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.pool_ta_pr2:NA:1]: Status change from WaitingForReturnCode to Done 2024-01-11 11:31:15,954 cromwell-system-akka.dispatchers.engine-dispatcher-45 INFO - Assigned new job execution tokens to the following groups: 067c7f61: 1 2024-01-11 11:31:15,958 INFO - Request threw an exception on attempt #1. Retrying after 317 milliseconds org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 11:31:16,276 INFO - Request threw an exception on attempt #2. Retrying after 181 milliseconds org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 11:31:16,459 INFO - Request threw an exception on attempt #3. Retrying after 6472 milliseconds org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 11:31:22,933 INFO - Request method=GET uri=https://auth.docker.io/token?service=registry.docker.io&scope=repository:encodedcc/chip-seq-pipeline:pull headers= threw an exception on attempt #4. Giving up. org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 11:31:22,935 cromwell-system-akka.dispatchers.engine-dispatcher-37 WARN - BackendPreparationActor_for_067c7f61:chip.pool_ta_pr1:-1:1 [UUID(067c7f61)]: Docker lookup failed java.lang.Exception: Failed to get docker hash for encodedcc/chip-seq-pipeline:v2.2.1 Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at cromwell.engine.workflow.WorkflowDockerLookupActor.cromwell$engine$workflow$WorkflowDockerLookupActor$$handleLookupFailure(WorkflowDockerLookupActor.scala:222) at cromwell.engine.workflow.WorkflowDockerLookupActor$$anonfun$3.applyOrElse(WorkflowDockerLookupActor.scala:88) at cromwell.engine.workflow.WorkflowDockerLookupActor$$anonfun$3.applyOrElse(WorkflowDockerLookupActor.scala:73) at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35) at akka.actor.FSM.processEvent(FSM.scala:707) at akka.actor.FSM.processEvent$(FSM.scala:704) at cromwell.engine.workflow.WorkflowDockerLookupActor.akka$actor$LoggingFSM$$super$processEvent(WorkflowDockerLookupActor.scala:40) at akka.actor.LoggingFSM.processEvent(FSM.scala:847) at akka.actor.LoggingFSM.processEvent$(FSM.scala:829) at cromwell.engine.workflow.WorkflowDockerLookupActor.processEvent(WorkflowDockerLookupActor.scala:40) at akka.actor.FSM.akka$actor$FSM$$processMsg(FSM.scala:701) at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:695) at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35) at cromwell.docker.DockerClientHelper$$anonfun$dockerResponseReceive$1.applyOrElse(DockerClientHelper.scala:16) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:269) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:270) at akka.actor.Actor.aroundReceive(Actor.scala:539) at akka.actor.Actor.aroundReceive$(Actor.scala:537) at cromwell.engine.workflow.WorkflowDockerLookupActor.aroundReceive(WorkflowDockerLookupActor.scala:40) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:614) at akka.actor.ActorCell.invoke(ActorCell.scala:583) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:268) at akka.dispatch.Mailbox.run(Mailbox.scala:229) at akka.dispatch.Mailbox.exec(Mailbox.scala:241) at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) 2024-01-11 11:31:22,936 cromwell-system-akka.dispatchers.engine-dispatcher-37 INFO - BT-322 067c7f61:chip.pool_ta_pr1:-1:1 is not eligible for call caching 2024-01-11 11:31:22,936 cromwell-system-akka.dispatchers.engine-dispatcher-37 WARN - BT-322 067c7f61:chip.pool_ta_pr1:-1:1 disabling call caching due to error 2024-01-11 11:31:22,937 cromwell-system-akka.dispatchers.backend-dispatcher-175 WARN - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.pool_ta_pr1:NA:1]: Unrecognized runtime attribute keys: disks, time, memory, cpu 2024-01-11 11:31:22,946 cromwell-system-akka.dispatchers.backend-dispatcher-175 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.pool_ta_pr1:NA:1]: `set -e python3 $(which encode_task_pool_ta.py) \ /cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_pr1/inputs/-684013381/ENCFF169ZIP.pr1.tagAlign.gz /cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_pr1/inputs/-691772740/ENCFF597JXK.pr1.tagAlign.gz \ --prefix rep-pr1 \` 2024-01-11 11:31:22,950 cromwell-system-akka.dispatchers.backend-dispatcher-175 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.pool_ta_pr1:NA:1]: executing: rm -f /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_pr1/execution/docker_cid if [ 'true' == 'true' ] && [ 'singularity' == 'docker' ] || \ [ 'true' == 'false' ] && [ 'true' == 'true' ] && [ ! -z 'encodedcc/chip-seq-pipeline:v2.2.1' ] then docker run -i --cidfile=/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_pr1/execution/docker_cid --user= --entrypoint=/bin/bash \ --volume=/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_pr1:/cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_pr1:delegated encodedcc/chip-seq-pipeline:v2.2.1 /cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_pr1/execution/script rc=$(docker wait `cat /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_pr1/execution/docker_cid`) docker rm `cat /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_pr1/execution/docker_cid` else # recover GID lost due to Cromwell running chmod 777 on CWD chown :`stat -c '%G' /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_pr1` -R /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_pr1 chmod g+s /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_pr1 if [ 'true' == 'true' ] && [ 'singularity' == 'singularity' ] || \ [ 'true' == 'false' ] && [ 'true' == 'true' ] && [ ! -z 'https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif' ] then mkdir -p $HOME/.singularity/lock/ flock --exclusive --timeout 600 \ $HOME/.singularity/lock/`echo -n 'https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif' | md5sum | cut -d' ' -f1` \ singularity exec --containall https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif echo 'Successfully pulled https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif' singularity exec --cleanenv --home=`dirname /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_pr1` \ --bind=/lustre/home/acct-medzy/medzy-cai,/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_pr1:/cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_pr1 \ \ https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif /bin/bash /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_pr1/execution/script & echo $! > /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_pr1/execution/docker_cid else # remap paths between inside and outside of a docker container shopt -s nullglob sed -i 's#/cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_pr1#/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_pr1#g' /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_pr1/execution/script `dirname /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_pr1/execution/script`/write_*.tmp if [ 'true' == 'true' ] && [ 'singularity' == 'conda' ] || \ [ 'true' == 'false' ] && [ 'true' == 'true' ] && [ ! -z 'encd-chip' ] then conda run --name=encd-chip /bin/bash /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_pr1/execution/script & echo $! > /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_pr1/execution/docker_cid else /bin/bash /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_pr1/execution/script & echo $! > /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_pr1/execution/docker_cid fi fi touch /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_pr1/execution/docker_cid.not_docker wait `cat /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-pool_ta_pr1/execution/docker_cid` rc=`echo $?` fi exit $rc 2024-01-11 11:31:23,805 cromwell-system-akka.dispatchers.backend-dispatcher-176 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.pool_ta_pr1:NA:1]: job id: 1798870 2024-01-11 11:31:23,807 cromwell-system-akka.dispatchers.backend-dispatcher-176 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.pool_ta_pr1:NA:1]: Status change from - to WaitingForReturnCode 2024-01-11 11:32:03,135 cromwell-system-akka.dispatchers.backend-dispatcher-175 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.pool_ta_pr1:NA:1]: Status change from WaitingForReturnCode to Done 2024-01-11 11:32:03,954 cromwell-system-akka.dispatchers.engine-dispatcher-37 INFO - Assigned new job execution tokens to the following groups: 067c7f61: 1 2024-01-11 11:32:03,958 INFO - Request threw an exception on attempt #1. Retrying after 100 milliseconds org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 11:32:04,060 INFO - Request threw an exception on attempt #2. Retrying after 1883 milliseconds org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 11:32:05,945 INFO - Request threw an exception on attempt #3. Retrying after 1376 milliseconds org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 11:32:07,323 INFO - Request method=GET uri=https://auth.docker.io/token?service=registry.docker.io&scope=repository:encodedcc/chip-seq-pipeline:pull headers= threw an exception on attempt #4. Giving up. org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 11:32:07,324 cromwell-system-akka.dispatchers.engine-dispatcher-33 WARN - BackendPreparationActor_for_067c7f61:chip.fraglen_mean:-1:1 [UUID(067c7f61)]: Docker lookup failed java.lang.Exception: Failed to get docker hash for encodedcc/chip-seq-pipeline:v2.2.1 Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at cromwell.engine.workflow.WorkflowDockerLookupActor.cromwell$engine$workflow$WorkflowDockerLookupActor$$handleLookupFailure(WorkflowDockerLookupActor.scala:222) at cromwell.engine.workflow.WorkflowDockerLookupActor$$anonfun$3.applyOrElse(WorkflowDockerLookupActor.scala:88) at cromwell.engine.workflow.WorkflowDockerLookupActor$$anonfun$3.applyOrElse(WorkflowDockerLookupActor.scala:73) at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35) at akka.actor.FSM.processEvent(FSM.scala:707) at akka.actor.FSM.processEvent$(FSM.scala:704) at cromwell.engine.workflow.WorkflowDockerLookupActor.akka$actor$LoggingFSM$$super$processEvent(WorkflowDockerLookupActor.scala:40) at akka.actor.LoggingFSM.processEvent(FSM.scala:847) at akka.actor.LoggingFSM.processEvent$(FSM.scala:829) at cromwell.engine.workflow.WorkflowDockerLookupActor.processEvent(WorkflowDockerLookupActor.scala:40) at akka.actor.FSM.akka$actor$FSM$$processMsg(FSM.scala:701) at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:695) at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35) at cromwell.docker.DockerClientHelper$$anonfun$dockerResponseReceive$1.applyOrElse(DockerClientHelper.scala:16) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:269) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:270) at akka.actor.Actor.aroundReceive(Actor.scala:539) at akka.actor.Actor.aroundReceive$(Actor.scala:537) at cromwell.engine.workflow.WorkflowDockerLookupActor.aroundReceive(WorkflowDockerLookupActor.scala:40) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:614) at akka.actor.ActorCell.invoke(ActorCell.scala:583) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:268) at akka.dispatch.Mailbox.run(Mailbox.scala:229) at akka.dispatch.Mailbox.exec(Mailbox.scala:241) at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) 2024-01-11 11:32:07,325 cromwell-system-akka.dispatchers.engine-dispatcher-33 INFO - BT-322 067c7f61:chip.fraglen_mean:-1:1 is not eligible for call caching 2024-01-11 11:32:07,325 cromwell-system-akka.dispatchers.engine-dispatcher-33 WARN - BT-322 067c7f61:chip.fraglen_mean:-1:1 disabling call caching due to error 2024-01-11 11:32:07,326 cromwell-system-akka.dispatchers.backend-dispatcher-179 WARN - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.fraglen_mean:NA:1]: Unrecognized runtime attribute keys: disks, time, memory, cpu 2024-01-11 11:32:07,330 cromwell-system-akka.dispatchers.backend-dispatcher-179 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.fraglen_mean:NA:1]: `python < /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-fraglen_mean/execution/docker_cid else # remap paths between inside and outside of a docker container shopt -s nullglob sed -i 's#/cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-fraglen_mean#/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-fraglen_mean#g' /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-fraglen_mean/execution/script `dirname /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-fraglen_mean/execution/script`/write_*.tmp if [ 'true' == 'true' ] && [ 'singularity' == 'conda' ] || \ [ 'true' == 'false' ] && [ 'true' == 'true' ] && [ ! -z 'encd-chip' ] then conda run --name=encd-chip /bin/bash /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-fraglen_mean/execution/script & echo $! > /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-fraglen_mean/execution/docker_cid else /bin/bash /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-fraglen_mean/execution/script & echo $! > /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-fraglen_mean/execution/docker_cid fi fi touch /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-fraglen_mean/execution/docker_cid.not_docker wait `cat /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-fraglen_mean/execution/docker_cid` rc=`echo $?` fi exit $rc 2024-01-11 11:32:08,804 cromwell-system-akka.dispatchers.backend-dispatcher-179 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.fraglen_mean:NA:1]: job id: 1799265 2024-01-11 11:32:08,805 cromwell-system-akka.dispatchers.backend-dispatcher-182 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.fraglen_mean:NA:1]: Status change from - to WaitingForReturnCode 2024-01-11 11:32:15,554 cromwell-system-akka.dispatchers.backend-dispatcher-182 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.fraglen_mean:NA:1]: Status change from WaitingForReturnCode to Done 2024-01-11 11:32:15,954 cromwell-system-akka.dispatchers.engine-dispatcher-42 INFO - Assigned new job execution tokens to the following groups: 067c7f61: 1 2024-01-11 11:32:15,957 INFO - Request threw an exception on attempt #1. Retrying after 162 milliseconds org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 11:32:16,121 INFO - Request threw an exception on attempt #2. Retrying after 1851 milliseconds org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 11:32:17,974 INFO - Request threw an exception on attempt #3. Retrying after 2400 milliseconds org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 11:32:20,375 INFO - Request method=GET uri=https://auth.docker.io/token?service=registry.docker.io&scope=repository:encodedcc/chip-seq-pipeline:pull headers= threw an exception on attempt #4. Giving up. org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 11:32:20,377 cromwell-system-akka.dispatchers.engine-dispatcher-48 WARN - BackendPreparationActor_for_067c7f61:chip.choose_ctl:-1:1 [UUID(067c7f61)]: Docker lookup failed java.lang.Exception: Failed to get docker hash for encodedcc/chip-seq-pipeline:v2.2.1 Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at cromwell.engine.workflow.WorkflowDockerLookupActor.cromwell$engine$workflow$WorkflowDockerLookupActor$$handleLookupFailure(WorkflowDockerLookupActor.scala:222) at cromwell.engine.workflow.WorkflowDockerLookupActor$$anonfun$3.applyOrElse(WorkflowDockerLookupActor.scala:88) at cromwell.engine.workflow.WorkflowDockerLookupActor$$anonfun$3.applyOrElse(WorkflowDockerLookupActor.scala:73) at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35) at akka.actor.FSM.processEvent(FSM.scala:707) at akka.actor.FSM.processEvent$(FSM.scala:704) at cromwell.engine.workflow.WorkflowDockerLookupActor.akka$actor$LoggingFSM$$super$processEvent(WorkflowDockerLookupActor.scala:40) at akka.actor.LoggingFSM.processEvent(FSM.scala:847) at akka.actor.LoggingFSM.processEvent$(FSM.scala:829) at cromwell.engine.workflow.WorkflowDockerLookupActor.processEvent(WorkflowDockerLookupActor.scala:40) at akka.actor.FSM.akka$actor$FSM$$processMsg(FSM.scala:701) at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:695) at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35) at cromwell.docker.DockerClientHelper$$anonfun$dockerResponseReceive$1.applyOrElse(DockerClientHelper.scala:16) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:269) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:270) at akka.actor.Actor.aroundReceive(Actor.scala:539) at akka.actor.Actor.aroundReceive$(Actor.scala:537) at cromwell.engine.workflow.WorkflowDockerLookupActor.aroundReceive(WorkflowDockerLookupActor.scala:40) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:614) at akka.actor.ActorCell.invoke(ActorCell.scala:583) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:268) at akka.dispatch.Mailbox.run(Mailbox.scala:229) at akka.dispatch.Mailbox.exec(Mailbox.scala:241) at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) 2024-01-11 11:32:20,378 cromwell-system-akka.dispatchers.engine-dispatcher-48 INFO - BT-322 067c7f61:chip.choose_ctl:-1:1 is not eligible for call caching 2024-01-11 11:32:20,378 cromwell-system-akka.dispatchers.engine-dispatcher-48 WARN - BT-322 067c7f61:chip.choose_ctl:-1:1 disabling call caching due to error 2024-01-11 11:32:20,379 cromwell-system-akka.dispatchers.backend-dispatcher-183 WARN - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.choose_ctl:NA:1]: Unrecognized runtime attribute keys: disks, time, memory, cpu 2024-01-11 11:32:20,398 cromwell-system-akka.dispatchers.backend-dispatcher-183 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.choose_ctl:NA:1]: `set -e python3 $(which encode_task_choose_ctl.py) \ --tas /cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-choose_ctl/inputs/-1550963907/ENCFF169ZIP.tagAlign.gz /cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-choose_ctl/inputs/-1558723266/ENCFF597JXK.tagAlign.gz \ --ctl-tas /cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-choose_ctl/inputs/-1193497575/ENCFF378YAW.tagAlign.gz /cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-choose_ctl/inputs/-1201256934/ENCFF267FMB.tagAlign.gz \ --ta-pooled /cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-choose_ctl/inputs/2082174970/rep.pooled.tagAlign.gz \ --ctl-ta-pooled /cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-choose_ctl/inputs/-2032425642/ctl.pooled.tagAlign.gz \ --always-use-pooled-ctl \ --ctl-depth-ratio 1.2 \ --ctl-depth-limit 200000000 \ --exp-ctl-depth-ratio-limit 5.0` 2024-01-11 11:32:20,403 cromwell-system-akka.dispatchers.backend-dispatcher-183 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.choose_ctl:NA:1]: executing: rm -f /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-choose_ctl/execution/docker_cid if [ 'true' == 'true' ] && [ 'singularity' == 'docker' ] || \ [ 'true' == 'false' ] && [ 'true' == 'true' ] && [ ! -z 'encodedcc/chip-seq-pipeline:v2.2.1' ] then docker run -i --cidfile=/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-choose_ctl/execution/docker_cid --user= --entrypoint=/bin/bash \ --volume=/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-choose_ctl:/cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-choose_ctl:delegated encodedcc/chip-seq-pipeline:v2.2.1 /cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-choose_ctl/execution/script rc=$(docker wait `cat /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-choose_ctl/execution/docker_cid`) docker rm `cat /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-choose_ctl/execution/docker_cid` else # recover GID lost due to Cromwell running chmod 777 on CWD chown :`stat -c '%G' /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-choose_ctl` -R /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-choose_ctl chmod g+s /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-choose_ctl if [ 'true' == 'true' ] && [ 'singularity' == 'singularity' ] || \ [ 'true' == 'false' ] && [ 'true' == 'true' ] && [ ! -z 'https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif' ] then mkdir -p $HOME/.singularity/lock/ flock --exclusive --timeout 600 \ $HOME/.singularity/lock/`echo -n 'https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif' | md5sum | cut -d' ' -f1` \ singularity exec --containall https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif echo 'Successfully pulled https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif' singularity exec --cleanenv --home=`dirname /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-choose_ctl` \ --bind=/lustre/home/acct-medzy/medzy-cai,/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-choose_ctl:/cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-choose_ctl \ \ https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif /bin/bash /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-choose_ctl/execution/script & echo $! > /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-choose_ctl/execution/docker_cid else # remap paths between inside and outside of a docker container shopt -s nullglob sed -i 's#/cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-choose_ctl#/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-choose_ctl#g' /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-choose_ctl/execution/script `dirname /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-choose_ctl/execution/script`/write_*.tmp if [ 'true' == 'true' ] && [ 'singularity' == 'conda' ] || \ [ 'true' == 'false' ] && [ 'true' == 'true' ] && [ ! -z 'encd-chip' ] then conda run --name=encd-chip /bin/bash /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-choose_ctl/execution/script & echo $! > /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-choose_ctl/execution/docker_cid else /bin/bash /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-choose_ctl/execution/script & echo $! > /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-choose_ctl/execution/docker_cid fi fi touch /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-choose_ctl/execution/docker_cid.not_docker wait `cat /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-choose_ctl/execution/docker_cid` rc=`echo $?` fi exit $rc 2024-01-11 11:32:23,805 cromwell-system-akka.dispatchers.backend-dispatcher-183 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.choose_ctl:NA:1]: job id: 1799463 2024-01-11 11:32:23,807 cromwell-system-akka.dispatchers.backend-dispatcher-187 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.choose_ctl:NA:1]: Status change from - to WaitingForReturnCode 2024-01-11 11:32:42,385 cromwell-system-akka.dispatchers.backend-dispatcher-184 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.choose_ctl:NA:1]: Status change from WaitingForReturnCode to Done 2024-01-11 11:32:49,416 cromwell-system-akka.dispatchers.engine-dispatcher-29 INFO - WorkflowExecutionActor-067c7f61-ef2a-4b2b-b54f-bb1bce5217af [UUID(067c7f61)]: Starting chip.call_peak_ppr2, chip.call_peak_pooled, chip.macs2_signal_track_pooled, chip.call_peak_ppr1 2024-01-11 11:32:49,954 cromwell-system-akka.dispatchers.engine-dispatcher-39 INFO - Assigned new job execution tokens to the following groups: 067c7f61: 1 2024-01-11 11:32:49,962 INFO - Request threw an exception on attempt #1. Retrying after 944 milliseconds org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 11:32:50,435 cromwell-system-akka.dispatchers.engine-dispatcher-39 INFO - WorkflowExecutionActor-067c7f61-ef2a-4b2b-b54f-bb1bce5217af [UUID(067c7f61)]: Starting chip.call_peak (2 shards), chip.call_peak_pr1 (2 shards), chip.call_peak_pr2 (2 shards), chip.macs2_signal_track (2 shards) 2024-01-11 11:32:50,908 INFO - Request threw an exception on attempt #2. Retrying after 583 milliseconds org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 11:32:51,493 INFO - Request threw an exception on attempt #3. Retrying after 3343 milliseconds org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 11:32:54,841 INFO - Request method=GET uri=https://auth.docker.io/token?service=registry.docker.io&scope=repository:encodedcc/chip-seq-pipeline:pull headers= threw an exception on attempt #4. Giving up. org.http4s.client.ConnectionFailure: Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at org.http4s.client.blaze.Http1Support.$anonfun$buildPipeline$1(Http1Support.scala:90) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at async @ org.http4s.internal.package$.$anonfun$fromFuture$1(package.scala:144) at flatMap @ org.http4s.internal.package$.fromFuture(package.scala:139) at flatMap @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at shift @ org.http4s.client.PoolManager.$anonfun$createConnection$2(PoolManager.scala:119) at uncancelable @ org.http4s.client.ConnectionManager$.pool(ConnectionManager.scala:83) at unsafeRunSync @ cromwell.docker.DockerInfoActor.preStart(DockerInfoActor.scala:172) Caused by: java.net.SocketException: Network is unreachable at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:483) at java.base/sun.nio.ch.Net.connect(Net.java:472) at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implConnect(UnixAsynchronousSocketChannelImpl.java:331) at java.base/sun.nio.ch.AsynchronousSocketChannelImpl.connect(AsynchronousSocketChannelImpl.java:210) at org.http4s.blaze.channel.nio2.ClientChannelFactory.connect(ClientChannelFactory.scala:88) at org.http4s.client.blaze.Http1Support.buildPipeline(Http1Support.scala:77) at org.http4s.client.blaze.Http1Support.$anonfun$makeClient$1(Http1Support.scala:70) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:104) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:463) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:484) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:422) at cats.effect.internals.Callback$AsyncIdempotentCallback.run(Callback.scala:131) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:67) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:35) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:90) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:90) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:90) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:42) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:137) at cats.effect.internals.Callback$AsyncIdempotentCallback.apply(Callback.scala:126) at cats.effect.Async$$anon$1.run(Async.scala:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-01-11 11:32:54,842 cromwell-system-akka.dispatchers.engine-dispatcher-35 WARN - BackendPreparationActor_for_067c7f61:chip.macs2_signal_track_pooled:-1:1 [UUID(067c7f61)]: Docker lookup failed java.lang.Exception: Failed to get docker hash for encodedcc/chip-seq-pipeline:v2.2.1 Error connecting to https://auth.docker.io using address auth.docker.io:443 (unresolved: false) at cromwell.engine.workflow.WorkflowDockerLookupActor.cromwell$engine$workflow$WorkflowDockerLookupActor$$handleLookupFailure(WorkflowDockerLookupActor.scala:222) at cromwell.engine.workflow.WorkflowDockerLookupActor$$anonfun$3.applyOrElse(WorkflowDockerLookupActor.scala:88) at cromwell.engine.workflow.WorkflowDockerLookupActor$$anonfun$3.applyOrElse(WorkflowDockerLookupActor.scala:73) at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35) at akka.actor.FSM.processEvent(FSM.scala:707) at akka.actor.FSM.processEvent$(FSM.scala:704) at cromwell.engine.workflow.WorkflowDockerLookupActor.akka$actor$LoggingFSM$$super$processEvent(WorkflowDockerLookupActor.scala:40) at akka.actor.LoggingFSM.processEvent(FSM.scala:847) at akka.actor.LoggingFSM.processEvent$(FSM.scala:829) at cromwell.engine.workflow.WorkflowDockerLookupActor.processEvent(WorkflowDockerLookupActor.scala:40) at akka.actor.FSM.akka$actor$FSM$$processMsg(FSM.scala:701) at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:695) at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35) at cromwell.docker.DockerClientHelper$$anonfun$dockerResponseReceive$1.applyOrElse(DockerClientHelper.scala:16) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:269) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:270) at akka.actor.Actor.aroundReceive(Actor.scala:539) at akka.actor.Actor.aroundReceive$(Actor.scala:537) at cromwell.engine.workflow.WorkflowDockerLookupActor.aroundReceive(WorkflowDockerLookupActor.scala:40) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:614) at akka.actor.ActorCell.invoke(ActorCell.scala:583) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:268) at akka.dispatch.Mailbox.run(Mailbox.scala:229) at akka.dispatch.Mailbox.exec(Mailbox.scala:241) at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) 2024-01-11 11:32:54,843 cromwell-system-akka.dispatchers.engine-dispatcher-29 INFO - BT-322 067c7f61:chip.macs2_signal_track_pooled:-1:1 is not eligible for call caching 2024-01-11 11:32:54,843 cromwell-system-akka.dispatchers.engine-dispatcher-29 WARN - BT-322 067c7f61:chip.macs2_signal_track_pooled:-1:1 disabling call caching due to error 2024-01-11 11:32:54,844 cromwell-system-akka.dispatchers.backend-dispatcher-188 WARN - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.macs2_signal_track_pooled:NA:1]: Unrecognized runtime attribute keys: preemptible, disks, cpu, time, memory 2024-01-11 11:32:54,856 cromwell-system-akka.dispatchers.backend-dispatcher-188 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.macs2_signal_track_pooled:NA:1]: `set -e python3 $(which encode_task_macs2_signal_track_chip.py) \ /cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-macs2_signal_track_pooled/inputs/2082174970/rep.pooled.tagAlign.gz /cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-macs2_signal_track_pooled/inputs/-2032425642/ctl.pooled.tagAlign.gz \ --gensz hs \ --chrsz /cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-macs2_signal_track_pooled/inputs/977218590/hg19.chrom.sizes \ --fraglen 118 \ --pval-thresh 0.01 \ --mem-gb 8.532029051333666` 2024-01-11 11:32:54,860 cromwell-system-akka.dispatchers.backend-dispatcher-188 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.macs2_signal_track_pooled:NA:1]: executing: rm -f /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-macs2_signal_track_pooled/execution/docker_cid if [ 'true' == 'true' ] && [ 'singularity' == 'docker' ] || \ [ 'true' == 'false' ] && [ 'true' == 'true' ] && [ ! -z 'encodedcc/chip-seq-pipeline:v2.2.1' ] then docker run -i --cidfile=/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-macs2_signal_track_pooled/execution/docker_cid --user= --entrypoint=/bin/bash \ --volume=/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-macs2_signal_track_pooled:/cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-macs2_signal_track_pooled:delegated encodedcc/chip-seq-pipeline:v2.2.1 /cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-macs2_signal_track_pooled/execution/script rc=$(docker wait `cat /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-macs2_signal_track_pooled/execution/docker_cid`) docker rm `cat /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-macs2_signal_track_pooled/execution/docker_cid` else # recover GID lost due to Cromwell running chmod 777 on CWD chown :`stat -c '%G' /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-macs2_signal_track_pooled` -R /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-macs2_signal_track_pooled chmod g+s /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-macs2_signal_track_pooled if [ 'true' == 'true' ] && [ 'singularity' == 'singularity' ] || \ [ 'true' == 'false' ] && [ 'true' == 'true' ] && [ ! -z 'https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif' ] then mkdir -p $HOME/.singularity/lock/ flock --exclusive --timeout 600 \ $HOME/.singularity/lock/`echo -n 'https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif' | md5sum | cut -d' ' -f1` \ singularity exec --containall https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif echo 'Successfully pulled https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif' singularity exec --cleanenv --home=`dirname /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-macs2_signal_track_pooled` \ --bind=/lustre/home/acct-medzy/medzy-cai,/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-macs2_signal_track_pooled:/cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-macs2_signal_track_pooled \ \ https://encode-pipeline-singularity-image.s3.us-west-2.amazonaws.com/chip-seq-pipeline_v2.2.1.sif /bin/bash /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-macs2_signal_track_pooled/execution/script & echo $! > /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-macs2_signal_track_pooled/execution/docker_cid else # remap paths between inside and outside of a docker container shopt -s nullglob sed -i 's#/cromwell-executions/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-macs2_signal_track_pooled#/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-macs2_signal_track_pooled#g' /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-macs2_signal_track_pooled/execution/script `dirname /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-macs2_signal_track_pooled/execution/script`/write_*.tmp if [ 'true' == 'true' ] && [ 'singularity' == 'conda' ] || \ [ 'true' == 'false' ] && [ 'true' == 'true' ] && [ ! -z 'encd-chip-macs2' ] then conda run --name=encd-chip-macs2 /bin/bash /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-macs2_signal_track_pooled/execution/script & echo $! > /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-macs2_signal_track_pooled/execution/docker_cid else /bin/bash /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-macs2_signal_track_pooled/execution/script & echo $! > /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-macs2_signal_track_pooled/execution/docker_cid fi fi touch /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-macs2_signal_track_pooled/execution/docker_cid.not_docker wait `cat /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-macs2_signal_track_pooled/execution/docker_cid` rc=`echo $?` fi exit $rc 2024-01-11 11:32:58,804 cromwell-system-akka.dispatchers.backend-dispatcher-188 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.macs2_signal_track_pooled:NA:1]: job id: 1799706 2024-01-11 11:32:58,805 cromwell-system-akka.dispatchers.backend-dispatcher-192 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.macs2_signal_track_pooled:NA:1]: Status change from - to WaitingForReturnCode 2024-01-11 12:00:24,214 cromwell-system-akka.dispatchers.engine-dispatcher-45 WARN - Found 1 stale workflow heartbeats (more than 210 seconds old): 067c7f61-ef2a-4b2b-b54f-bb1bce5217af 2024-01-11 12:00:24,309 cromwell-system-akka.dispatchers.engine-dispatcher-45 ERROR - Shutting down Cromwell instance cromid-8596e4a as 1 stale workflow heartbeats (more than 5 minutes old) were found: 067c7f61-ef2a-4b2b-b54f-bb1bce5217af 2024-01-11 12:00:24,310 INFO - Workflow polling stopped 2024-01-11 12:00:24,415 INFO - 0 workflows released by cromid-8596e4a 2024-01-11 12:00:24,416 INFO - Shutting down WorkflowStoreActor - Timeout = 5 seconds 2024-01-11 12:00:24,417 INFO - Shutting down WorkflowLogCopyRouter - Timeout = 5 seconds 2024-01-11 12:00:24,419 INFO - Shutting down JobExecutionTokenDispenser - Timeout = 5 seconds 2024-01-11 12:00:24,423 INFO - JobExecutionTokenDispenser stopped 2024-01-11 12:00:24,453 cromwell-system-akka.dispatchers.engine-dispatcher-35 INFO - Aborting all running workflows. 2024-01-11 12:00:24,453 INFO - WorkflowStoreActor stopped 2024-01-11 12:00:24,480 INFO - WorkflowLogCopyRouter stopped 2024-01-11 12:00:24,480 INFO - Shutting down WorkflowManagerActor - Timeout = 3600 seconds 2024-01-11 12:00:24,480 cromwell-system-akka.dispatchers.engine-dispatcher-44 INFO - WorkflowManagerActor: Aborting all workflows 2024-01-11 12:00:24,480 cromwell-system-akka.dispatchers.engine-dispatcher-44 INFO - WorkflowExecutionActor-067c7f61-ef2a-4b2b-b54f-bb1bce5217af [UUID(067c7f61)]: Aborting workflow 2024-01-11 12:00:24,486 cromwell-system-akka.dispatchers.engine-dispatcher-44 INFO - WorkflowExecutionActor-067c7f61-ef2a-4b2b-b54f-bb1bce5217af [UUID(067c7f61)]: WorkflowExecutionActor [UUID(067c7f61)] aborted: chip.call_peak_ppr2:NA:1 2024-01-11 12:00:24,487 cromwell-system-akka.dispatchers.engine-dispatcher-44 INFO - WorkflowExecutionActor-067c7f61-ef2a-4b2b-b54f-bb1bce5217af [UUID(067c7f61)]: WorkflowExecutionActor [UUID(067c7f61)] aborted: chip.call_peak:1:1 2024-01-11 12:00:24,487 cromwell-system-akka.dispatchers.engine-dispatcher-44 INFO - WorkflowExecutionActor-067c7f61-ef2a-4b2b-b54f-bb1bce5217af [UUID(067c7f61)]: WorkflowExecutionActor [UUID(067c7f61)] aborted: chip.call_peak_pr2:1:1 2024-01-11 12:00:24,487 cromwell-system-akka.dispatchers.engine-dispatcher-44 INFO - WorkflowExecutionActor-067c7f61-ef2a-4b2b-b54f-bb1bce5217af [UUID(067c7f61)]: WorkflowExecutionActor [UUID(067c7f61)] aborted: chip.macs2_signal_track:1:1 2024-01-11 12:00:24,487 cromwell-system-akka.dispatchers.engine-dispatcher-44 INFO - WorkflowExecutionActor-067c7f61-ef2a-4b2b-b54f-bb1bce5217af [UUID(067c7f61)]: WorkflowExecutionActor [UUID(067c7f61)] aborted: chip.macs2_signal_track:0:1 2024-01-11 12:00:24,487 cromwell-system-akka.dispatchers.engine-dispatcher-44 INFO - WorkflowExecutionActor-067c7f61-ef2a-4b2b-b54f-bb1bce5217af [UUID(067c7f61)]: WorkflowExecutionActor [UUID(067c7f61)] aborted: chip.call_peak_pr2:0:1 2024-01-11 12:00:24,487 cromwell-system-akka.dispatchers.engine-dispatcher-44 INFO - WorkflowExecutionActor-067c7f61-ef2a-4b2b-b54f-bb1bce5217af [UUID(067c7f61)]: WorkflowExecutionActor [UUID(067c7f61)] aborted: chip.call_peak_pr1:1:1 2024-01-11 12:00:24,487 cromwell-system-akka.dispatchers.engine-dispatcher-44 INFO - WorkflowExecutionActor-067c7f61-ef2a-4b2b-b54f-bb1bce5217af [UUID(067c7f61)]: WorkflowExecutionActor [UUID(067c7f61)] aborted: chip.call_peak_ppr1:NA:1 2024-01-11 12:00:24,487 cromwell-system-akka.dispatchers.engine-dispatcher-44 INFO - WorkflowExecutionActor-067c7f61-ef2a-4b2b-b54f-bb1bce5217af [UUID(067c7f61)]: WorkflowExecutionActor [UUID(067c7f61)] aborted: chip.call_peak_pooled:NA:1 2024-01-11 12:00:24,487 cromwell-system-akka.dispatchers.engine-dispatcher-44 INFO - WorkflowExecutionActor-067c7f61-ef2a-4b2b-b54f-bb1bce5217af [UUID(067c7f61)]: WorkflowExecutionActor [UUID(067c7f61)] aborted: chip.call_peak_pr1:0:1 2024-01-11 12:00:24,487 cromwell-system-akka.dispatchers.engine-dispatcher-44 INFO - WorkflowExecutionActor-067c7f61-ef2a-4b2b-b54f-bb1bce5217af [UUID(067c7f61)]: WorkflowExecutionActor [UUID(067c7f61)] aborted: chip.call_peak:0:1 2024-01-11 12:06:17,872 cromwell-system-akka.dispatchers.backend-dispatcher-190 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.macs2_signal_track_pooled:NA:1]: executing: if [ -f '/lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-macs2_signal_track_pooled/execution/docker_cid.not_docker' ] then kill `cat /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-macs2_signal_track_pooled/execution/docker_cid` else docker kill `cat /lustre/home/acct-medzy/medzy-cai/project/project_encode_chip/result/ChIP-cmf-1/chip/067c7f61-ef2a-4b2b-b54f-bb1bce5217af/call-macs2_signal_track_pooled/execution/docker_cid` fi 2024-01-11 12:06:17,885 cromwell-system-akka.dispatchers.backend-dispatcher-190 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.macs2_signal_track_pooled:NA:1]: BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61):chip.macs2_signal_track_pooled:NA:1] Aborted StandardAsyncJob(1799706) 2024-01-11 12:08:45,075 cromwell-system-akka.dispatchers.backend-dispatcher-213 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(067c7f61)chip.macs2_signal_track_pooled:NA:1]: Status change from WaitingForReturnCode to Done 2024-01-11 12:08:45,082 cromwell-system-akka.dispatchers.engine-dispatcher-37 INFO - WorkflowExecutionActor-067c7f61-ef2a-4b2b-b54f-bb1bce5217af [UUID(067c7f61)]: WorkflowExecutionActor [UUID(067c7f61)] aborted: chip.macs2_signal_track_pooled:NA:1 2024-01-11 12:08:45,726 cromwell-system-akka.dispatchers.engine-dispatcher-37 INFO - WorkflowManagerActor: All workflows are aborted 2024-01-11 12:08:45,726 cromwell-system-akka.dispatchers.engine-dispatcher-37 INFO - WorkflowManagerActor: All workflows finished 2024-01-11 12:08:45,727 INFO - WorkflowManagerActor stopped 2024-01-11 12:08:46,267 INFO - Connection pools shut down 2024-01-11 12:08:46,269 INFO - Shutting down SubWorkflowStoreActor - Timeout = 1800 seconds 2024-01-11 12:08:46,269 INFO - Shutting down JobStoreActor - Timeout = 1800 seconds 2024-01-11 12:08:46,269 INFO - Shutting down CallCacheWriteActor - Timeout = 1800 seconds 2024-01-11 12:08:46,269 INFO - Shutting down ServiceRegistryActor - Timeout = 1800 seconds 2024-01-11 12:08:46,269 INFO - Shutting down DockerHashActor - Timeout = 1800 seconds 2024-01-11 12:08:46,269 INFO - Shutting down IoProxy - Timeout = 1800 seconds 2024-01-11 12:08:46,269 INFO - SubWorkflowStoreActor stopped 2024-01-11 12:08:46,270 INFO - CallCacheWriteActor Shutting down: 0 queued messages to process 2024-01-11 12:08:46,270 INFO - JobStoreActor stopped 2024-01-11 12:08:46,270 INFO - CallCacheWriteActor stopped 2024-01-11 12:08:46,270 INFO - WriteMetadataActor Shutting down: 25 queued messages to process 2024-01-11 12:08:46,270 INFO - KvWriteActor Shutting down: 0 queued messages to process 2024-01-11 12:08:46,271 INFO - DockerHashActor stopped 2024-01-11 12:08:46,271 INFO - IoProxy stopped 2024-01-11 12:08:46,275 INFO - WriteMetadataActor Shutting down: processing 0 queued messages 2024-01-11 12:08:46,276 INFO - ServiceRegistryActor stopped 2024-01-11 12:08:46,277 INFO - Shutting down connection pool: curAllocated=0 idleQueues.size=0 waitQueue.size=0 maxWaitQueueLimit=256 closed=false 2024-01-11 12:08:46,277 INFO - Shutting down connection pool: curAllocated=0 idleQueues.size=0 waitQueue.size=0 maxWaitQueueLimit=256 closed=false 2024-01-11 12:08:46,279 INFO - Database closed 2024-01-11 12:08:46,279 INFO - Shutting down connection pool: curAllocated=0 idleQueues.size=0 waitQueue.size=0 maxWaitQueueLimit=256 closed=false 2024-01-11 12:08:46,284 INFO - Stream materializer shut down 2024-01-11 12:08:46,287 INFO - WDL HTTP import resolver closed