WARNING: Could not find a final dir among

[sunnycqcn]

Hello Developers,
I am running ipa using corrected data. I met the errors as follows. I tried to fix based on the issues. I still can not figure out it.
INFO: /home/AAFC-AAC/fuf/miniconda3/envs/ipa/bin/ipa local --nthreads 80 --njobs 1 --run-dir BOpb --tmp-dir BO -i /isilon/saskatoon-rdc/users/fuf/BnGenome/genome/polishedgenome/BO.correctedReads.fasta --dry-run --verbose --resume
INFO: ipa.py ipa (wrapper) version=1.3.1 ... Checking dependencies ...
INFO: Dependencies
/home/AAFC-AAC/fuf/miniconda3/envs/ipa/bin/python3
/home/AAFC-AAC/fuf/miniconda3/envs/ipa/bin/ipa2-task
/home/AAFC-AAC/fuf/miniconda3/envs/ipa/bin/falconc
/home/AAFC-AAC/fuf/miniconda3/envs/ipa/bin/minimap2
/home/AAFC-AAC/fuf/miniconda3/envs/ipa/bin/nighthawk
/home/AAFC-AAC/fuf/miniconda3/envs/ipa/bin/pancake
/home/AAFC-AAC/fuf/miniconda3/envs/ipa/bin/pblayout
/home/AAFC-AAC/fuf/miniconda3/envs/ipa/bin/pbmm2
/home/AAFC-AAC/fuf/miniconda3/envs/ipa/bin/racon
/home/AAFC-AAC/fuf/miniconda3/envs/ipa/bin/samtools
/home/AAFC-AAC/fuf/miniconda3/envs/ipa/bin/ipa_purge_dups
/home/AAFC-AAC/fuf/miniconda3/envs/ipa/bin/ipa_purge_dups_split_fa
snakemake version=5.9.1
ipa2-task 1.0.2 (commit 038eb47eb869b3ea55bfbd83a8b4f78873a2c437)
Machine name: 'Linux'
falconc version=1.10.3+git., nim-version=1.2.4
minimap2 version=2.17-r941
Nighthawk 0.1.0 (commit a3ede16*)
pancake 0.2.0 (commit 2a70d3e*)
pblayout 0.1.1 (commit 137e0ad*)
pbmm2 1.4.0 (commit 1.4.0)
racon version=v1.4.13
samtools 1.10
Using htslib 1.10.2
ipa_purge_dups Version: 1.2.3

WARNING: Could not find a final dir among ['config.yaml', 'assembly-results', '.snakemake', 'input.fofn', 'qsub_log', 'ipa.log']. Not symlinking assembly-results directory.

Wrote "BOpb/config.yaml".
advanced_options: ''
coverage: 0
genome_size: 0
m4filt_high_copy_sample_rate: 1.0
max_nchunks: 40
nproc: 80
phase_run: 1
polish_run: 1
purge_dups_calcuts: ''
purge_dups_run: 1
reads_fn: BOpb/input.fofn
tmp_dir: BO

To run this yourself:
/home/AAFC-AAC/fuf/miniconda3/envs/ipa/bin/python3 -m snakemake --dryrun -j 1 -d BOpb -p -s /home/AAFC-AAC/fuf/miniconda3/envs/ipa/etc/ipa.snakefile --configfile BOpb/config.yaml --reason

Starting snakemake --dryrun ...
CWD:/isilon/saskatoon-rdc/users/fuf/BnGenome/genome/rawgenome/BO/BOpb
NPROC:80
NPROC_SERIAL:80
config:{'advanced_options': '', 'coverage': 0, 'genome_size': 0, 'm4filt_high_copy_sample_rate': 1.0, 'max_nchunks': 40, 'nproc': 80, 'phase_run': 1, 'polish_run': 1, 'purge_dups_calcuts': '', 'purge_dups_run': 1, 'reads_fn': 'BOpb/input.fofn', 'tmp_dir': 'BO'}
Building DAG of jobs...
Job counts:
count jobs
1 assemble
1 build_db
1 final
1 finish
1 generate_config
1 ovl_asym_merge
1 ovl_asym_prepare
1 ovl_filter
1 phasing_merge
1 phasing_prepare
1 polish_merge
1 polish_prepare
1 purge_dups_map_merge
1 purge_dups_map_prepare
1 purge_dups_paf
1 separate_p_from_a
16

[Thu Oct 22 22:16:31 2020]
localrule generate_config:
input: /isilon/saskatoon-rdc/users/fuf/BnGenome/genome/rawgenome/BO/BOpb/input.fofn
output: 01-generate_config/generated.config
jobid: 5
reason: Missing output files: 01-generate_config/generated.config

    wd=$(dirname 01-generate_config/generated.config)
    mkdir -p $wd
    cd $wd
    rel=..

    params_advanced_opt=""         params_coverage="0"         params_genome_size="0"         params_polish_run="1"         params_phase_run="1"         params_purge_dups_run="1"         params_purge_dups_calcuts=""         params_purge_dups_get_seqs=""         params_m4filt_high_copy_sample_rate="1.0"         params_log_level="INFO"         params_tmp_dir="BO"         output_fn="generated.config.sh"         sentinel_fn="generated.config"             time ipa2-task generate_config_from_workflow

[Thu Oct 22 22:16:31 2020]
rule build_db:
input: /isilon/saskatoon-rdc/users/fuf/BnGenome/genome/rawgenome/BO/BOpb/input.fofn, 01-generate_config/generated.config
output: 02-build_db/reads.seqdb, 02-build_db/reads.seeddb, 02-build_db/reads.seqdb.0.seq, 02-build_db/reads.seeddb.0.seeds, 02-build_db/input.fofn
jobid: 9
reason: Missing output files: 02-build_db/reads.seqdb; Input files updated by another job: 01-generate_config/generated.config

    wd=$(dirname 02-build_db/reads.seqdb)
    mkdir -p $wd
    cd $wd
    rel=..

    input_reads_fn="/isilon/saskatoon-rdc/users/fuf/BnGenome/genome/rawgenome/BO/BOpb/input.fofn"         params_db_prefix="reads"         params_config_sh_fn="$rel/01-generate_config/generated.config"         params_num_threads=80         params_log_level="INFO"         params_tmp_dir="BO"             time ipa2-task build_db

[Thu Oct 22 22:16:31 2020]
localcheckpoint ovl_asym_prepare:
input: 02-build_db/reads.seqdb
output: 03-ovl_asym_prepare/shards
jobid: 15
reason: Missing output files: 03-ovl_asym_prepare/shards; Input files updated by another job: 02-build_db/reads.seqdb
Downstream jobs will be updated after completion.

    rm -f 03-ovl_asym_prepare/shards
    mkdir -p 03-ovl_asym_prepare/shards
    cd 03-ovl_asym_prepare/shards
    rel=../..

    input_db="$rel/02-build_db/reads.seqdb"         output_shard_ids=./all_shard_ids         output_pwd=./pwd         params_max_nchunks="40"         params_log_level="INFO"         params_tmp_dir="BO"             time ipa2-task ovl_asym_prepare

[Thu Oct 22 22:16:31 2020]
rule ovl_asym_merge:
input: , 01-generate_config/generated.config
output: 05-ovl_asym_merge/ovl.merged.m4, 05-ovl_asym_merge/ovl.nonlocal.m4
jobid: 13
reason: Missing output files: 05-ovl_asym_merge/ovl.nonlocal.m4; Input files updated by another job: 01-generate_config/generated.config, 03-ovl_asym_prepare/shards

    wd=$(dirname 05-ovl_asym_merge/ovl.merged.m4)
    mkdir -p $wd
    cd $wd
    rel=..

    # We must change rel-path names when we chdir.
    for fn in 03-ovl_asym_prepare/shards; do
        echo $rel/$fn
    done >| ./sorted.fofn

    input_fofn=./sorted.fofn         params_num_threads="80"         params_db_prefix=reads         params_config_sh_fn="$rel/01-generate_config/generated.config"         params_log_level="INFO"         params_tmp_dir="BO"             time ipa2-task ovl_asym_merge

[Thu Oct 22 22:16:31 2020]
checkpoint phasing_prepare:
input: 02-build_db/reads.seqdb, 05-ovl_asym_merge/ovl.nonlocal.m4, 01-generate_config/generated.config
output: 06-phasing_prepare/shards
jobid: 14
reason: Missing output files: 06-phasing_prepare/shards; Input files updated by another job: 01-generate_config/generated.config, 05-ovl_asym_merge/ovl.nonlocal.m4, 02-build_db/reads.seqdb
Downstream jobs will be updated after completion.

    rm -f 06-phasing_prepare/shards
    mkdir -p 06-phasing_prepare/shards
    cd 06-phasing_prepare/shards
    rel=../..

    input_m4="$rel/05-ovl_asym_merge/ovl.nonlocal.m4"         output_shard_ids=./all_shard_ids         output_pwd=./pwd.txt         params_config_sh_fn="$rel/01-generate_config/generated.config"         params_max_nchunks="40"         params_log_level="INFO"         params_tmp_dir="BO"             time ipa2-task phasing_prepare

[Thu Oct 22 22:16:31 2020]
rule phasing_merge:
input: 05-ovl_asym_merge/ovl.nonlocal.m4, , 01-generate_config/generated.config
output: 08-phasing_merge/ovl.phased.m4
jobid: 12
reason: Missing output files: 08-phasing_merge/ovl.phased.m4; Input files updated by another job: 01-generate_config/generated.config, 05-ovl_asym_merge/ovl.nonlocal.m4, 06-phasing_prepare/shards

    wd=$(dirname 08-phasing_merge/ovl.phased.m4)
    mkdir -p $wd
    cd $wd
    rel=..

    # Collect all overlap paths to keep.
    for fn in 06-phasing_prepare/shards; do
        # Get the folder name.
        dn=$(cat $rel/$fn)
        echo $dn/ovl.phased.m4
    done >| ./merged.keep.fofn

    # Collect all scraps paths.
    for fn in 06-phasing_prepare/shards; do
        # Get the folder name.
        dn=$(cat $rel/$fn)
        echo $dn/ovl.phased.m4.scraps
    done >| ./merged.scraps.fofn

    input_keep_fofn="./merged.keep.fofn"         input_scraps_fofn="./merged.scraps.fofn"         input_original_m4="$rel/05-ovl_asym_merge/ovl.nonlocal.m4"         output_m4="ovl.phased.m4"         params_num_threads=80         params_config_sh_fn="$rel/01-generate_config/generated.config"         params_log_level="INFO"         params_tmp_dir="BO"             time ipa2-task phasing_merge

[Thu Oct 22 22:16:31 2020]
rule ovl_filter:
input: 08-phasing_merge/ovl.phased.m4, 01-generate_config/generated.config
output: 09-ovl_filter/ovl.final.m4, 09-ovl_filter/ovl.chimerfilt.m4
jobid: 11
reason: Missing output files: 09-ovl_filter/ovl.final.m4; Input files updated by another job: 01-generate_config/generated.config, 08-phasing_merge/ovl.phased.m4

    wd=$(dirname 09-ovl_filter/ovl.final.m4)
    mkdir -p $wd
    cd $wd
    rel=..

    input_m4="$rel/08-phasing_merge/ovl.phased.m4"         output_m4_final=$(basename 09-ovl_filter/ovl.final.m4)         output_m4_chimerfilt=$(basename 09-ovl_filter/ovl.chimerfilt.m4)         params_num_threads=80         params_config_sh_fn="$rel/01-generate_config/generated.config"         params_log_level="INFO"         params_tmp_dir="BO"             time ipa2-task ovl_filter

[Thu Oct 22 22:16:31 2020]
rule assemble:
input: /isilon/saskatoon-rdc/users/fuf/BnGenome/genome/rawgenome/BO/BOpb/input.fofn, 02-build_db/reads.seqdb, 09-ovl_filter/ovl.final.m4, 08-phasing_merge/ovl.phased.m4, 01-generate_config/generated.config
output: 10-assemble/p_ctg.fasta, 10-assemble/a_ctg.fasta, 10-assemble/p_ctg_tiling_path, 10-assemble/a_ctg_tiling_path, 10-assemble/p_ctg.fasta.fai, 10-assemble/a_ctg.fasta.fai, 10-assemble/read_to_contig.csv, 10-assemble/circular_contigs.csv
jobid: 8
reason: Missing output files: 10-assemble/p_ctg.fasta.fai, 10-assemble/read_to_contig.csv, 10-assemble/p_ctg.fasta, 10-assemble/a_ctg.fasta, 10-assemble/a_ctg.fasta.fai; Input files updated by another job: 01-generate_config/generated.config, 09-ovl_filter/ovl.final.m4, 02-build_db/reads.seqdb, 08-phasing_merge/ovl.phased.m4

    wd=$(dirname 10-assemble/p_ctg.fasta)
    mkdir -p $wd
    cd $wd
    rel=..

    input_seqdb="$rel/02-build_db/reads.seqdb"         input_m4="$rel/09-ovl_filter/ovl.final.m4"         input_m4_phasing_merge="$rel/08-phasing_merge/ovl.phased.m4"         input_reads="/isilon/saskatoon-rdc/users/fuf/BnGenome/genome/rawgenome/BO/BOpb/input.fofn"         params_ctg_prefix=ctg.         params_num_threads=80         params_config_sh_fn="$rel/01-generate_config/generated.config"         params_log_level="INFO"         params_tmp_dir="BO"             time ipa2-task assemble

[Thu Oct 22 22:16:31 2020]
checkpoint polish_prepare:
input: 10-assemble/read_to_contig.csv, 01-generate_config/generated.config, 10-assemble/p_ctg.fasta.fai, 10-assemble/a_ctg.fasta.fai
output: 11-polish_prepare/shards
jobid: 10
reason: Missing output files: 11-polish_prepare/shards; Input files updated by another job: 01-generate_config/generated.config, 10-assemble/p_ctg.fasta.fai, 10-assemble/read_to_contig.csv, 10-assemble/a_ctg.fasta.fai
Downstream jobs will be updated after completion.

    rm -f 11-polish_prepare/shards
    mkdir -p 11-polish_prepare/shards
    cd 11-polish_prepare/shards
    rel=../..

    input_read_to_contig="$rel/10-assemble/read_to_contig.csv"         input_p_ctg_fasta_fai="$rel/10-assemble/p_ctg.fasta.fai"         input_a_ctg_fasta_fai="$rel/10-assemble/a_ctg.fasta.fai"         output_shard_ids=./all_shard_ids         output_pwd=./pwd.txt         params_config_sh_fn="$rel/01-generate_config/generated.config"         params_max_nchunks="40"         params_log_level="INFO"         params_tmp_dir="BO"             time ipa2-task polish_prepare

[Thu Oct 22 22:16:31 2020]
rule polish_merge:
input: , 10-assemble/p_ctg.fasta, 10-assemble/a_ctg.fasta, 01-generate_config/generated.config
output: 13-polish_merge/assembly.merged.fasta, 13-polish_merge/assembly.merged.fasta.fai
jobid: 7
reason: Missing output files: 13-polish_merge/assembly.merged.fasta, 13-polish_merge/assembly.merged.fasta.fai; Input files updated by another job: 01-generate_config/generated.config, 11-polish_prepare/shards, 10-assemble/p_ctg.fasta, 10-assemble/a_ctg.fasta

    wd=$(dirname 13-polish_merge/assembly.merged.fasta)
    mkdir -p $wd
    cd $wd
    rel=..

    # We must change rel-path names when we chdir.
    for fn in 11-polish_prepare/shards; do
        echo $rel/$fn
    done >| ./merged.fofn

    for fn in 11-polish_prepare/shards; do
        dn=$(dirname $rel/$fn)
        echo $dn
    done >| ./merged.fodn

    input_fofn=./merged.fofn         input_fodn=./merged.fodn         input_p_ctg_fasta="$rel/10-assemble/p_ctg.fasta"         input_a_ctg_fasta="$rel/10-assemble/a_ctg.fasta"         output_fasta=$(basename 13-polish_merge/assembly.merged.fasta)         output_p_paf="p.paf"         output_a_paf="a.paf"         params_num_threads="80"         params_config_sh_fn="$rel/01-generate_config/generated.config"         params_log_level="INFO"         params_tmp_dir="BO"             time ipa2-task polish_merge

[Thu Oct 22 22:16:31 2020]
localrule separate_p_from_a:
input: 13-polish_merge/assembly.merged.fasta, 13-polish_merge/assembly.merged.fasta.fai, 10-assemble/p_ctg.fasta, 10-assemble/a_ctg.fasta
output: 14-separate/p_ctg.fasta, 14-separate/a_ctg.fasta
jobid: 4
reason: Missing output files: 14-separate/a_ctg.fasta, 14-separate/p_ctg.fasta; Input files updated by another job: 13-polish_merge/assembly.merged.fasta, 13-polish_merge/assembly.merged.fasta.fai, 10-assemble/p_ctg.fasta, 10-assemble/a_ctg.fasta

    wd=$(dirname 14-separate/p_ctg.fasta)
    mkdir -p $wd
    cd $wd
    rel=..

    input_assembly_merged_fasta="$rel/13-polish_merge/assembly.merged.fasta"         input_p_ctg_fasta="$rel/10-assemble/p_ctg.fasta"         input_a_ctg_fasta="$rel/10-assemble/a_ctg.fasta"         output_p_ctg_fasta=$(basename 14-separate/p_ctg.fasta)         output_a_ctg_fasta=$(basename 14-separate/a_ctg.fasta)         params_log_level="INFO"         params_tmp_dir="BO"         params_polish_run="1"             time ipa2-task separate_p_from_a

[Thu Oct 22 22:16:31 2020]
checkpoint purge_dups_map_prepare:
input: 02-build_db/reads.seqdb, 14-separate/p_ctg.fasta, 01-generate_config/generated.config
output: 15-purge_dups_map_prepare/shards
jobid: 6
reason: Missing output files: 15-purge_dups_map_prepare/shards; Input files updated by another job: 01-generate_config/generated.config, 02-build_db/reads.seqdb, 14-separate/p_ctg.fasta
Downstream jobs will be updated after completion.

    rm -f 15-purge_dups_map_prepare/shards
    mkdir -p 15-purge_dups_map_prepare/shards
    cd 15-purge_dups_map_prepare/shards
    rel=../..

    input_reads_db="$rel/02-build_db/reads.seqdb"         input_primary_fasta="$rel/14-separate/p_ctg.fasta"         output_shard_ids=./all_shard_ids         output_pwd=./pwd         params_num_threads="80"         params_max_nchunks="40"         params_log_level="INFO"         params_config_sh_fn="$rel/01-generate_config/generated.config"             time ipa2-task purge_dups_map_prepare

[Thu Oct 22 22:16:31 2020]
rule purge_dups_map_merge:
input: , 01-generate_config/generated.config
output: 17-purge_dups_map_merge/merged.paf.gz
jobid: 3
reason: Missing output files: 17-purge_dups_map_merge/merged.paf.gz; Input files updated by another job: 01-generate_config/generated.config, 15-purge_dups_map_prepare/shards

    wd=$(dirname 17-purge_dups_map_merge/merged.paf.gz)
    mkdir -p $wd
    cd $wd
    rel=..

    # We must change rel-path names when we chdir.
    for fn in 15-purge_dups_map_prepare/shards; do
        echo $rel/$fn
    done >| ./sorted.fofn

    input_fofn=./sorted.fofn         params_num_threads="80"         params_config_sh_fn="$rel/01-generate_config/generated.config"         params_log_level="INFO"         output_paf="$(basename 17-purge_dups_map_merge/merged.paf.gz)"             time ipa2-task purge_dups_map_merge

[Thu Oct 22 22:16:31 2020]
rule purge_dups_paf:
input: 17-purge_dups_map_merge/merged.paf.gz, 14-separate/p_ctg.fasta, 14-separate/a_ctg.fasta, 01-generate_config/generated.config
output: 18-purge_dups/final_purged_primary.fasta, 18-purge_dups/final_purged_haplotigs.fasta
jobid: 2
reason: Missing output files: 18-purge_dups/final_purged_haplotigs.fasta, 18-purge_dups/final_purged_primary.fasta; Input files updated by another job: 01-generate_config/generated.config, 17-purge_dups_map_merge/merged.paf.gz, 14-separate/p_ctg.fasta, 14-separate/a_ctg.fasta

    wd=$(dirname 18-purge_dups/final_purged_primary.fasta)
    mkdir -p $wd
    cd $wd
    rel=..

    input_paf="$rel/17-purge_dups_map_merge/merged.paf.gz"         input_primary_fasta="$rel/14-separate/p_ctg.fasta"         input_haplotigs_fasta="$rel/14-separate/a_ctg.fasta"         params_num_threads="80"         params_log_level="INFO"         params_config_sh_fn="$rel/01-generate_config/generated.config"         output_primary_fasta=$(basename 18-purge_dups/final_purged_primary.fasta)         output_haplotigs_fasta=$(basename 18-purge_dups/final_purged_haplotigs.fasta)             time ipa2-task purge_dups_paf

[Thu Oct 22 22:16:31 2020]
localrule final:
input: 18-purge_dups/final_purged_primary.fasta, 18-purge_dups/final_purged_haplotigs.fasta
output: 19-final/final.p_ctg.fasta, 19-final/final.a_ctg.fasta
jobid: 1
reason: Missing output files: 19-final/final.a_ctg.fasta, 19-final/final.p_ctg.fasta; Input files updated by another job: 18-purge_dups/final_purged_haplotigs.fasta, 18-purge_dups/final_purged_primary.fasta

    wd=$(dirname 19-final/final.p_ctg.fasta)
    mkdir -p $wd
    cd $wd
    rel=..

    input_p_ctg_fasta="$rel/18-purge_dups/final_purged_primary.fasta"         input_a_ctg_fasta="$rel/18-purge_dups/final_purged_haplotigs.fasta"         output_p_ctg_fasta="$rel/19-final/final.p_ctg.fasta"         output_a_ctg_fasta="$rel/19-final/final.a_ctg.fasta"             time ipa2-task final_copy

[Thu Oct 22 22:16:31 2020]
localrule finish:
input: 19-final/final.p_ctg.fasta, 19-final/final.a_ctg.fasta
jobid: 0
reason: Input files updated by another job: 19-final/final.a_ctg.fasta, 19-final/final.p_ctg.fasta

Job counts:
count jobs
1 assemble
1 build_db
1 final
1 finish
1 generate_config
1 ovl_asym_merge
1 ovl_asym_prepare
1 ovl_filter
1 phasing_merge
1 phasing_prepare
1 polish_merge
1 polish_prepare
1 purge_dups_map_merge
1 purge_dups_map_prepare
1 purge_dups_paf
1 separate_p_from_a
16
This was a dry-run (flag -n). The order of jobs does not reflect the order of execution.
Thanks,
Fuyou

[sunnycqcn]

In fact, when I check the directory, the config.yaml and input.fofn are here.
(ipa) [fuf@biocluster BO]$ ls -ltrh BOpb
total 106K
-rwx------ 1 fuf domain users 87 Oct 22 21:32 input.fofn
drwx------ 2 fuf domain users 0 Oct 22 21:32 qsub_log
-rwx------ 1 fuf domain users 214 Oct 22 22:16 config.yaml
lrwx------ 1 fuf domain users 7 Oct 22 22:16 assembly-results -> ipa.log
-rwx------ 1 fuf domain users 1.5K Oct 22 22:16 ipa.log

[pb-cdunn]

% ipa local --run-dir BOpb --tmp-dir BO -i input.fofn --dry-run --verbose --resume
...
This was a dry-run (flag -n). The order of jobs does not reflect the order of execution.
WARNING: Could not find a final dir among ['ipa.log', 'input.fofn', 'config.yaml', '.snakemake', 'qsub_log']. Not symlinking assembly-results directory.

You're right. That warning happens on a dry-run. I guess it shouldn't. Not a bug though.

[sunnycqcn]

ipa local --run-dir BOpb --tmp-dir BO -i input.fofn --dry-run --verbose --resume

Hello,
I did not understand your answer. You mean I should use the command
"ipa local --run-dir BOpb --tmp-dir BO -i input.fofn --dry-run --verbose --resume" .
Or I need delete --dry-run? However, I tried to delete --dry-run. It did not work yet.
Thanks,
Fuyou

[sunnycqcn]

The error is following:

INFO: /home/AAFC-AAC/fuf/miniconda3/envs/ipa/bin/ipa local --nthreads 4 --njobs 20 --run-dir BOpb --tmp-dir BO -i input.fofn --verbose --resume
INFO: ipa.py ipa (wrapper) version=1.3.1 ... Checking dependencies ...
INFO: Dependencies
/home/AAFC-AAC/fuf/miniconda3/envs/ipa/bin/python3
/home/AAFC-AAC/fuf/miniconda3/envs/ipa/bin/ipa2-task
/home/AAFC-AAC/fuf/miniconda3/envs/ipa/bin/falconc
/home/AAFC-AAC/fuf/miniconda3/envs/ipa/bin/minimap2
/home/AAFC-AAC/fuf/miniconda3/envs/ipa/bin/nighthawk
/home/AAFC-AAC/fuf/miniconda3/envs/ipa/bin/pancake
/home/AAFC-AAC/fuf/miniconda3/envs/ipa/bin/pblayout
/home/AAFC-AAC/fuf/miniconda3/envs/ipa/bin/pbmm2
/home/AAFC-AAC/fuf/miniconda3/envs/ipa/bin/racon
/home/AAFC-AAC/fuf/miniconda3/envs/ipa/bin/samtools
/home/AAFC-AAC/fuf/miniconda3/envs/ipa/bin/ipa_purge_dups
/home/AAFC-AAC/fuf/miniconda3/envs/ipa/bin/ipa_purge_dups_split_fa
snakemake version=5.9.1
ipa2-task 1.0.2 (commit 038eb47eb869b3ea55bfbd83a8b4f78873a2c437)
 Machine name: 'Linux'
falconc version=1.10.3+git., nim-version=1.2.4
minimap2 version=2.17-r941
Nighthawk 0.1.0 (commit a3ede16*)
pancake 0.2.0 (commit 2a70d3e*)
pblayout 0.1.1 (commit 137e0ad*)
pbmm2 1.4.0 (commit 1.4.0)
racon version=v1.4.13
samtools 1.10
Using htslib 1.10.2
ipa_purge_dups Version: 1.2.3

ERROR: Command '/home/AAFC-AAC/fuf/miniconda3/envs/ipa/bin/python3 -m snakemake -j 20 -d BOpb -p -s /home/AAFC-AAC/fuf/miniconda3/envs/ipa/etc/ipa.snakefile --configfile BOpb/config.yaml --reason' returned non-zero exit status 1.
Exiting.

Thanks,
Fuyou

[pb-cdunn]

That's not enough information for debugging. You'll have to look for a log-file, look at the run-dir BOpb to see what steps if any actually ran. In some cases, it might be easier to debug running snakemake directly rather than via ipa, and you can see the command-line for that.

[sunnycqcn]

Hello Christopher,
I am much appreciated for your suggestions. I checked ipa.log file. The file is as following:

INFO: /home/AAFC-AAC/fuf/miniconda3/envs/ipa/bin/ipa local --nthreads 4 --njobs 20 --run-dir BOpb --tmp-dir BO -i input.fofn --verbose --resume
INFO: ipa.py ipa (wrapper) version=1.3.1 ... Checking dependencies ...
INFO: Dependencies
/home/AAFC-AAC/fuf/miniconda3/envs/ipa/bin/python3
/home/AAFC-AAC/fuf/miniconda3/envs/ipa/bin/ipa2-task
/home/AAFC-AAC/fuf/miniconda3/envs/ipa/bin/falconc
/home/AAFC-AAC/fuf/miniconda3/envs/ipa/bin/minimap2
/home/AAFC-AAC/fuf/miniconda3/envs/ipa/bin/nighthawk
/home/AAFC-AAC/fuf/miniconda3/envs/ipa/bin/pancake
/home/AAFC-AAC/fuf/miniconda3/envs/ipa/bin/pblayout
/home/AAFC-AAC/fuf/miniconda3/envs/ipa/bin/pbmm2
/home/AAFC-AAC/fuf/miniconda3/envs/ipa/bin/racon
/home/AAFC-AAC/fuf/miniconda3/envs/ipa/bin/samtools
/home/AAFC-AAC/fuf/miniconda3/envs/ipa/bin/ipa_purge_dups
/home/AAFC-AAC/fuf/miniconda3/envs/ipa/bin/ipa_purge_dups_split_fa
snakemake version=5.9.1
ipa2-task 1.0.2 (commit 038eb47eb869b3ea55bfbd83a8b4f78873a2c437)
Machine name: 'Linux'
falconc version=1.10.3+git., nim-version=1.2.4
minimap2 version=2.17-r941
Nighthawk 0.1.0 (commit a3ede16*)
pancake 0.2.0 (commit 2a70d3e*)
pblayout 0.1.1 (commit 137e0ad*)
pbmm2 1.4.0 (commit 1.4.0)
racon version=v1.4.13
samtools 1.10
Using htslib 1.10.2
ipa_purge_dups Version: 1.2.3

ERROR: Command '/home/AAFC-AAC/fuf/miniconda3/envs/ipa/bin/python3 -m snakemake -j 20 -d BOpb -p -s /home/AAFC-AAC/fuf/miniconda3/envs/ipa/etc/ipa.snakefile --configfile BOpb/config.yaml --reason' returned non-zero exit status 1.
Exiting.
And I run snake command as
python3 -m snakemake -j 20 -d BOpb -p -s /home/AAFC-AAC/fuf/miniconda3/envs/ipa/etc/ipa.snakefile --configfile BOpb/config.yaml --reason
However, I still can not run my job. I can not understand what it happens. I think I followed your instructions.
Best,
Fuyou

[sunnycqcn]

Hello Christopher,
I know what it is the reason which snakemake uses bash strict mode!
I will try how to submit the command with qsub.
Best,
Fuyou

[sunnycqcn]

Hello Christopher,
I tried to figure out what happened. I found the error is as following.
/home/AAFC-AAC/fuf/miniconda3/envs/ipa/bin/ipa2-task: line 120: /usr/bin/time: No such file or directory

I do not know it is my system problem or the code bug. Because the code works well if I directly run in my cluster. If I submitted by qsub, I met this problem.
My qsub scripts is:
#!/bin/bash
#$ -N fc-BO
#$ -cwd
#$ -pe smp 1
#$ -S /bin/bash

conda activate ipa

ipa local -i input.fofn --run-dir BOpb --nthreads 20 --njobs 4 --resume
/home/AAFC-AAC/fuf/miniconda3/envs/ipa/bin/python3 -m snakemake -j 4 -d BOpb -p -s /home/AAFC-AAC/fuf/miniconda3/envs/ipa/etc/ipa.snakefile --configfile BOpb/config.yaml --reason.
If you can check your pipeline, I am much appreciated for your help.
Thanks,
Fuyou

[iggyB]

Hej,

Most likely "time" is not installed on your cluster.
Bash has built in "time" command. Therefor it works when you run directly.

Cheers,
Iggy

[sunnycqcn]

Hej,

Most likely "time" is not installed on your cluster.
Bash has built in "time" command. Therefor it works when you run directly.

Cheers,
Iggy

Hello Iggy,
I installed time. And the command can run without submit to server.
Thanks,
Fuyou

[sunnycqcn]

Hi,
I did not know what happened.
I checked my command "time"
/usr/bin/time Usage: /usr/bin/time [-apvV] [-f format] [-o file] [--append] [--verbose] [--portability] [--format=format] [--output=file] [--version] [--help] command [arg...]

This command is presented and working well.
However, I always met this error if I submitted to server cluster.
/usr/bin/time '--format=cmd: %C\nreal_time: %e s\nuser_time: %U s\nsys_time: %S s\nmax_rss: %M kB\nexit_status: %x' -o log.build_db.seqdb_full.memtime pancake seqdb --log-level INFO --compression 0 --block-size 4096 --compression 1 reads input.fofn /home/AAFC-AAC/fuf/miniconda3/envs/ipa/bin/ipa2-task: line 121: /usr/bin/time: No such file or directory
Thanks,
Fuyou

[ctxchris]

I run ipa local and get the same

Could not find a final dir among

error message basically after every stage. Executing the same command with --resume continues the assembly without problem up to the next stage. The assembly finishes without errors in the end, but not having to restart it all the time would be a great plus ;)

[pb-cdunn]

@sunnycqcn , I think we've addressed the problems with UNIX time.

@ctxchris , you might have an old version of snakemake.

[yizhouc]

I have the same problem with the docker image
<8 --njobs 1 --input-fn ASSEMBLY_NAME.fasta --resume

Error in rule build_db:
jobid: 9
output: 02-build_db/reads.seqdb, 02-build_db/reads.seeddb, 02-build_db/reads.seqdb.0.seq, 02-build_db/reads.seeddb.0.seeds, 02-build_db/input.fofn
shell:

    wd=$(dirname 02-build_db/reads.seqdb)
    mkdir -p $wd
    cd $wd
    rel=..

    input_reads_fn="/tmp/RUN/input.fofn"         params_db_prefix="reads"         params_config_sh_fn="$rel/01-generate_config/generated.config"         params_num_threads=8         params_log_level="INFO"         params_tmp_dir="/tmp"             time ipa2-task build_db

    (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

no 02-build_db directory created .