Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error on test code (however, using singularity) #10

Open
gailrosen opened this issue Jun 6, 2023 · 10 comments
Open

Error on test code (however, using singularity) #10

gailrosen opened this issue Jun 6, 2023 · 10 comments

Comments

@gailrosen
Copy link

Hi,

Getting all this to work in singularity is a nightmare since there are a lot of wrapper scripts around the docker. Singularity can pull a docker and work as long as everything is self-contained in the docker. Would love for that to be the case, where all these wrapper are inside the docker. I am having trouble running things -- so now I have tried the test.

I am running:
singularity run -B /ifs/groups/rosenMRIGrp/NIST_datasets/fastvifi_depdata/kraken_datasets:/home/kraken2-db -B .:/test ../fastvifi_latest.sif python /test/run_kraken_vifi_pipeline.py --output-dir /test/fastvifi_output_files --input-file /test/test_reads_1.fq --input-file-2 /test/test_reads_2.fq --level sample-level-validation-intermediate --kraken-path /home/kraken2/kraken2 --kraken-db-path /home/kraken2-db --vifi-path /home/ViFi/scripts/run_vifi.py --virus hpv --human-chr-list /test/human_chr_list.txt --skip-bwa-filter --keep-intermediate-files

But the BAMs are not output -- do you know why? (see below even when i try to get rid of the internal docker flag). Could you make this a little more singularity friendly and have the pipelines contained within the container?

Here is the output:

Warning! output directory already exists. Rewriting previous outputs.
Loading database information... done.
19 sequences (0.00 Mbp) processed in 0.006s (193.4 Kseq/m, 30.53 Mbp/m).
19 sequences classified (100.00%)
0 sequences unclassified (0.00%)
Command being timed: "/home/kraken2/kraken2 --use-names --report /test/fastvifi_output_files/kraken_report_first_level_for_virus_hp
v --db /home/kraken2-db/Kraken2StandardDB_k_25_hpv_hg --threads 1 --paired --f-threshold 0.4 --keep-unmapped-reads --unmapped-threshold 0.8
--classified-out /test/fastvifi_output_files/reads_passing_kraken_first_level_for_virus_hpv#.fq /test/fastvifi_output_files/input_file_1.f
q /test/fastvifi_output_files/input_file_2.fq --output /test/fastvifi_output_files/kraken_output_classify_first_level_for_virus_hpv"
User time (seconds): 0.01
System time (seconds): 2.43
Percent of CPU this job got: 98%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.50
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 4166152
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 30
Minor (reclaiming a frame) page faults: 1042986
Voluntary context switches: 495
Involuntary context switches: 85
Swaps: 0
File system inputs: 90402
File system outputs: 32
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
Loading database information... done.
10 sequences (0.00 Mbp) processed in 0.004s (143.3 Kseq/m, 42.99 Mbp/m).
10 sequences classified (100.00%)
0 sequences unclassified (0.00%)
Command being timed: "/home/kraken2/kraken2 --use-names --report /test/fastvifi_output_files/kraken_report_final_level_for_virus_hpv --db /home/kraken2-db/Kraken2StandardDB_k_22_hpv --threads 1 --paired --f-threshold 0.5 --unmapped-threshold 0.9 --classified-out /test/fastvifi_output_files/reads_passing_kraken_filter_for_virus_hpv#.fq /test/fastvifi_output_files/reads_passing_kraken_first_level_for_virus_hpv_1.fq /test/fastvifi_output_files/reads_passing_kraken_first_level_for_virus_hpv_2.fq --output /test/fastvifi_output_files/kraken_output_classify_final_level_for_virus_hpv"
User time (seconds): 0.01
System time (seconds): 0.01
Percent of CPU this job got: 70%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.03
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 9796
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 6
Minor (reclaiming a frame) page faults: 3175
Voluntary context switches: 25
Involuntary context switches: 3
Swaps: 0
File system inputs: 14
File system outputs: 32
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
sh: 1: docker: not found
Command being timed: "python /home/ViFi/scripts/run_vifi.py --docker -f /test/fastvifi_output_files/reads_passing_kraken_filter_for_virus_hpv_1.fq -r /test/fastvifi_output_files/reads_passing_kraken_filter_for_virus_hpv_2.fq -o /test/fastvifi_output_files --virus hpv -c 1 --prefix output_hpv"
User time (seconds): 0.01
System time (seconds): 0.03
Percent of CPU this job got: 94%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.05
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 11156
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 15
Minor (reclaiming a frame) page faults: 1880
Voluntary context switches: 25
Involuntary context switches: 7
Swaps: 0
File system inputs: 274
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
[E::hts_open_format] Failed to open file "/test/fastvifi_output_files/output_hpv.viral.bam" : No such file or directory
Traceback (most recent call last):
File "/test/run_kraken_vifi_pipeline.py", line 468, in
run_pipeline(args)
File "/test/run_kraken_vifi_pipeline.py", line 459, in run_pipeline
bwa_filtered_fq_filename_2=bwa_filtered_fq_filename_2)
File "/test/run_kraken_vifi_pipeline.py", line 326, in run_kraken_vifi
vifi_input_fq_1=vifi_input_fq_1)
File "/test/run_kraken_vifi_pipeline.py", line 358, in write_viral_reads_report
viral_aligned_file = pysam.AlignmentFile(os.path.join(args.output_dir, "{}.viral.bam".format(prefix)), 'rb')
File "pysam/libcalignmentfile.pyx", line 751, in pysam.libcalignmentfile.AlignmentFile.cinit
File "pysam/libcalignmentfile.pyx", line 950, in pysam.libcalignmentfile.AlignmentFile._open
IOError: [Errno 2] could not open alignment file /test/fastvifi_output_files/output_hpv.viral.bam: No such file or directory
[gailr@node040 vifi_test]$ singularity run -B /ifs/groups/rosenMRIGrp/NIST_datasets/fastvifi_depdata/kraken_datasets:/home/kraken2-db -B .:/test ../fastvifi_latest.sif python /test/run_kraken_vifi_pipeline.py --output-dir /test/fastvifi_output_files --input-file /test/test_reads_1.fq --input-file-2 /test/test_reads_2.fq --level sample-level-validation-intermediate --kraken-path /home/kraken2/kraken2 --kraken-db-path /home/kraken2-db --vifi-path /home/ViFi/scripts/run_vifi.py --virus hpv --human-chr-list /test/human_chr_list.txt --skip-bwa-filter --keep-intermediate-files |more
Loading database information... done.
19 sequences (0.00 Mbp) processed in 0.006s (191.5 Kseq/m, 30.24 Mbp/m).
19 sequences classified (100.00%)
0 sequences unclassified (0.00%)
Command being timed: "/home/kraken2/kraken2 --use-names --report /test/fastvifi_output_files/kraken_report_first_level_for_virus_hpv --db /home/kraken2-db/Kraken2StandardDB_k_25_hpv_hg --threads 1 --paired --f-threshold 0.4 --keep-unmapped-reads --unmapped-threshold 0.8 --classified-out /test/fastvifi_output_files/reads_passing_kraken_first_level_for_virus_hpv#.fq /test/fastvifi_output_files/input_file_1.fq /test/fastvifi_output_files/input_file_2.fq --output /test/fastvifi_output_files/kraken_output_classify_first_level_for_virus_hpv"
User time (seconds): 0.01
System time (seconds): 2.41
Percent of CPU this job got: 98%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.46
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 4166256
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 30
Minor (reclaiming a frame) page faults: 1042987
Voluntary context switches: 418
Involuntary context switches: 84
Swaps: 0
File system inputs: 60410
File system outputs: 32
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
Loading database information... done.
10 sequences (0.00 Mbp) processed in 0.005s (119.9 Kseq/m, 35.96 Mbp/m).
10 sequences classified (100.00%)
0 sequences unclassified (0.00%)
Command being timed: "/home/kraken2/kraken2 --use-names --report /test/fastvifi_output_files/kraken_report_final_level_for_virus_hpv --db /home/kraken2-db/Kraken2StandardDB_k_22_hpv --threads 1 --paired --f-threshold 0.5 --unmapped-threshold 0.9 --classified-out /test/fastvifi_output_files/reads_passing_kraken_filter_for_virus_hpv#.fq /test/fastvifi_output_files/reads_passing_kraken_first_level_for_virus_hpv_1.fq /test/fastvifi_output_files/reads_passing_kraken_first_level_for_virus_hpv_2.fq --output /test/fastvifi_output_files/kraken_output_classify_final_level_for_virus_hpv"
User time (seconds): 0.01
System time (seconds): 0.00
Percent of CPU this job got: 67%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.03
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 9796
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 4
Minor (reclaiming a frame) page faults: 3174
Voluntary context switches: 25
Involuntary context switches: 5
Swaps: 0
File system inputs: 14
File system outputs: 32
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
sh: 1: docker: not found
Command being timed: "python /home/ViFi/scripts/run_vifi.py --docker -f /test/fastvifi_output_files/reads_passing_kraken_filter_for_virus_hpv_1.fq -r /test/fastvifi_output_files/reads_passing_kraken_filter_for_virus_hpv_2.fq -o /test/fastvifi_output_files --virus hpv -c 1 --prefix output_hpv"
User time (seconds): 0.01
System time (seconds): 0.03
Percent of CPU this job got: 94%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.05
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 11156
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 15
Minor (reclaiming a frame) page faults: 1880
Voluntary context switches: 25
Involuntary context switches: 5
Swaps: 0
File system inputs: 274
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
[E::hts_open_format] Failed to open file "/test/fastvifi_output_files/output_hpv.viral.bam" : No such file or directory
Traceback (most recent call last):
File "/test/run_kraken_vifi_pipeline.py", line 468, in
run_pipeline(args)
File "/test/run_kraken_vifi_pipeline.py", line 459, in run_pipeline
bwa_filtered_fq_filename_2=bwa_filtered_fq_filename_2)
File "/test/run_kraken_vifi_pipeline.py", line 326, in run_kraken_vifi
vifi_input_fq_1=vifi_input_fq_1)
File "/test/run_kraken_vifi_pipeline.py", line 358, in write_viral_reads_report
viral_aligned_file = pysam.AlignmentFile(os.path.join(args.output_dir, "{}.viral.bam".format(prefix)), 'rb')
File "pysam/libcalignmentfile.pyx", line 751, in pysam.libcalignmentfile.AlignmentFile.cinit
File "pysam/libcalignmentfile.pyx", line 950, in pysam.libcalignmentfile.AlignmentFile._open
IOError: [Errno 2] could not open alignment file /test/fastvifi_output_files/output_hpv.viral.bam: No such file or directory
Warning! output directory already exists. Rewriting previous outputs.

I tried getting rid of the docker flag in the run pipeline script and got:

[gailr@node040 vifi_test]$ singularity run -B /ifs/groups/rosenMRIGrp/NIST_datasets/fastvifi_depdata/kraken_datasets:/home/kraken2-db -B .:
/test ../fastvifi_latest.sif python /test/run_kraken_vifi_pipeline.py --output-dir /test/fastvifi_output_files --input-file /test/test_read
s_1.fq --input-file-2 /test/test_reads_2.fq --level sample-level-validation-intermediate --kraken-path /home/kraken2/kraken2 --kraken-db-pa
th /home/kraken2-db --vifi-path /home/ViFi/scripts/run_vifi.py --virus hpv --human-chr-list /test/human_chr_list.txt --skip-bwa-filter --ke
ep-intermediate-files
Warning! output directory already exists. Rewriting previous outputs.
Loading database information... done.
19 sequences (0.00 Mbp) processed in 0.007s (155.5 Kseq/m, 24.56 Mbp/m).
19 sequences classified (100.00%)
0 sequences unclassified (0.00%)
Command being timed: "/home/kraken2/kraken2 --use-names --report /test/fastvifi_output_files/kraken_report_first_level_for_virus_hp
v --db /home/kraken2-db/Kraken2StandardDB_k_25_hpv_hg --threads 1 --paired --f-threshold 0.4 --keep-unmapped-reads --unmapped-threshold 0.8
--classified-out /test/fastvifi_output_files/reads_passing_kraken_first_level_for_virus_hpv#.fq /test/fastvifi_output_files/input_file_1.f
q /test/fastvifi_output_files/input_file_2.fq --output /test/fastvifi_output_files/kraken_output_classify_first_level_for_virus_hpv"
User time (seconds): 0.01
System time (seconds): 2.35
Percent of CPU this job got: 87%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.70
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 4166328
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 30
Minor (reclaiming a frame) page faults: 1042985
Voluntary context switches: 175
Involuntary context switches: 104
Swaps: 0
File system inputs: 60074
File system outputs: 32
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
Loading database information... done.
10 sequences (0.00 Mbp) processed in 0.005s (117.1 Kseq/m, 35.12 Mbp/m).
10 sequences classified (100.00%)
0 sequences unclassified (0.00%)
Command being timed: "/home/kraken2/kraken2 --use-names --report /test/fastvifi_output_files/kraken_report_final_level_for_virus_hpv --db /home/kraken2-db/Kraken2StandardDB_k_22_hpv --threads 1 --paired --f-threshold 0.5 --unmapped-threshold 0.9 --classified-out /test/fastvifi_output_files/reads_passing_kraken_filter_for_virus_hpv#.fq /test/fastvifi_output_files/reads_passing_kraken_first_level_for_virus_hpv_1.fq /test/fastvifi_output_files/reads_passing_kraken_first_level_for_virus_hpv_2.fq --output /test/fastvifi_output_files/kraken_output_classify_final_level_for_virus_hpv"
User time (seconds): 0.01
System time (seconds): 0.01
Percent of CPU this job got: 70%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.04
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 9604
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 12
Minor (reclaiming a frame) page faults: 2155
Voluntary context switches: 24
Involuntary context switches: 4
Swaps: 0
File system inputs: 14
File system outputs: 32
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
[E::bwa_idx_load_from_disk] fail to locate the index files
Traceback (most recent call last):
File "/home/ViFi/scripts/get_trans_new.py", line 106, in
bamFile = pysam.Samfile(opts.dataName[0], 'rb')
File "pysam/libcalignmentfile.pyx", line 751, in pysam.libcalignmentfile.AlignmentFile.cinit
File "pysam/libcalignmentfile.pyx", line 1000, in pysam.libcalignmentfile.AlignmentFile._open
ValueError: file has no sequences defined (mode='rb') - is it SAM/BAM format? Consider opening with check_sq=False
ls: cannot access '/home/repo/data/hpv/hmms/*.hmmbuild': No such file or directory
[E::hts_open_format] Failed to open file "/test/fastvifi_output_files/output_hpv.unknown.bam" : No such file or directory
Traceback (most recent call last):
File "/home/ViFi/scripts/run_hmms.py", line 112, in
run_pipeline(options)
File "/home/ViFi/scripts/run_hmms.py", line 37, in run_pipeline
prepare_unmapped_sequences(options)
File "/home/ViFi/scripts/run_hmms.py", line 95, in prepare_unmapped_sequences
bam = pysam.Samfile(options.bamfile, 'rb')
File "pysam/libcalignmentfile.pyx", line 751, in pysam.libcalignmentfile.AlignmentFile.cinit
File "pysam/libcalignmentfile.pyx", line 950, in pysam.libcalignmentfile.AlignmentFile._open
IOError: [Errno 2] could not open alignment file /test/fastvifi_output_files/output_hpv.unknown.bam: No such file or directory
[E::hts_open_format] Failed to open file "/test/fastvifi_output_files/output_hpv.trans.bam" : No such file or directory
Traceback (most recent call last):
File "/home/ViFi/scripts/merge_viral_reads.py", line 117, in
transFile = pysam.Samfile(args.transName[0], 'rb')
File "pysam/libcalignmentfile.pyx", line 751, in pysam.libcalignmentfile.AlignmentFile.cinit
File "pysam/libcalignmentfile.pyx", line 950, in pysam.libcalignmentfile.AlignmentFile._open
IOError: [Errno 2] could not open alignment file /test/fastvifi_output_files/output_hpv.trans.bam: No such file or directory
[E::hts_open_format] Failed to open file /test/fastvifi_output_files/output_hpv.fixed.trans.bam
samtools sort: can't open "/test/fastvifi_output_files/output_hpv.fixed.trans.bam": No such file or directory
samtools index: "/test/fastvifi_output_files/output_hpv.fixed.trans.cs.bam" is in a format that cannot be usefully indexed
[E::hts_open_format] Failed to open file /test/fastvifi_output_files/output_hpv.viral.bam
samtools sort: can't open "/test/fastvifi_output_files/output_hpv.viral.bam": No such file or directory
samtools index: "/test/fastvifi_output_files/output_hpv.viral.cs.bam" is in a format that cannot be usefully indexed
WARNING:root:#TIME 0.042 Unable to find reference in $AA_DATA_REPO/reference.txt. Setting to working directory.
WARNING:root:#TIME 0.044 Unable to find reference in $AA_DATA_REPO/REF/file_list.txt. Setting to empty.
WARNING:root:#TIME 0.044 Unable to open fasta file: "/home/data_repo//". Reference sequences will be set to N.
WARNING:root:#TIME 0.044 Unable to open chromosome lengths file: "/home/data_repo//"
WARNING:root:#TIME 0.044 interval_list: Unable to open interval file "/home/data_repo//".
WARNING:root:#TIME 0.044 interval_list: Unable to open interval file "/home/data_repo//".
WARNING:root:#TIME 0.044 interval_list: Unable to open interval file "/home/data_repo//".
WARNING:root:#TIME 0.044 interval_list: Unable to open interval file "/home/data_repo//".
WARNING:root:#TIME 0.044 interval_list: Unable to open interval file "/home/data_repo//".
Traceback (most recent call last):
File "/home/ViFi/scripts/cluster_trans_new.py", line 38, in
bamFile = pysam.Samfile(args.dataName[0], 'rb')
File "pysam/libcalignmentfile.pyx", line 751, in pysam.libcalignmentfile.AlignmentFile.cinit
File "pysam/libcalignmentfile.pyx", line 956, in pysam.libcalignmentfile.AlignmentFile._open
ValueError: file does not contain alignment data
Command being timed: "python /home/ViFi/scripts/run_vifi.py -f /test/fastvifi_output_files/reads_passing_kraken_filter_for_virus_hpv_1.fq -r /test/fastvifi_output_files/reads_passing_kraken_filter_for_virus_hpv_2.fq -o /test/fastvifi_output_files --virus hpv -c 1 --prefix output_hpv"
User time (seconds): 0.14
System time (seconds): 0.17
Percent of CPU this job got: 67%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.47
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 23808
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 96
Minor (reclaiming a frame) page faults: 17524
Voluntary context switches: 171
Involuntary context switches: 90
Swaps: 0
File system inputs: 4512
File system outputs: 8
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
[E::hts_open_format] Failed to open file "/test/fastvifi_output_files/output_hpv.viral.bam" : No such file or directory
Traceback (most recent call last):
File "/test/run_kraken_vifi_pipeline.py", line 469, in
run_pipeline(args)
File "/test/run_kraken_vifi_pipeline.py", line 460, in run_pipeline
bwa_filtered_fq_filename_2=bwa_filtered_fq_filename_2)
File "/test/run_kraken_vifi_pipeline.py", line 327, in run_kraken_vifi
vifi_input_fq_1=vifi_input_fq_1)
File "/test/run_kraken_vifi_pipeline.py", line 359, in write_viral_reads_report
viral_aligned_file = pysam.AlignmentFile(os.path.join(args.output_dir, "{}.viral.bam".format(prefix)), 'rb')
File "pysam/libcalignmentfile.pyx", line 751, in pysam.libcalignmentfile.AlignmentFile.cinit
File "pysam/libcalignmentfile.pyx", line 950, in pysam.libcalignmentfile.AlignmentFile._open
IOError: [Errno 2] could not open alignment file /test/fastvifi_output_files/output_hpv.viral.bam: No such file or directory
@sara-javadzadeh
Copy link
Owner

Hi Gail,

Thanks for reaching out. This is a good point. Let me look into this and get back to you.

@sara-javadzadeh
Copy link
Owner

Hi Gail,

I have added the option to run FastViFi with Singularity. Please refer to section "Running FastViFi with Singularity" of the manual, as you need to run run_kraken_vifi_container.py instead of run_kraken_vifi_pipeline.py.

I tested it with singularity version 3 or above installed. Please check the version of singularity by running singularity --version. Please let me know if you have any questions.

Sara

@gailrosen
Copy link
Author

gailrosen commented Aug 11, 2023 via email

@gailrosen
Copy link
Author

gailrosen commented Oct 3, 2023 via email

@gailrosen
Copy link
Author

gailrosen commented Oct 3, 2023 via email

@gailrosen
Copy link
Author

gailrosen commented Oct 3, 2023 via email

@gailrosen
Copy link
Author

just for reference, it is trying to open a file that was never made. Here are the files that were made:
(pysam) [gailr@node042 FastViFi]$ ls -l test
total 416
-rw-r--r-- 1 gailr rosenMRIGrp 4360 Oct 3 13:38 hmms.txt
-rw-r--r-- 1 gailr rosenMRIGrp 138 Oct 2 19:23 human_chr_list.txt
-rw-r--r-- 1 gailr rosenMRIGrp 6129 Oct 3 13:38 input_file_1.fq
-rw-r--r-- 1 gailr rosenMRIGrp 6129 Oct 3 13:38 input_file_2.fq
-rw-r--r-- 1 gailr rosenMRIGrp 1303 Oct 3 13:38 log_fastvifi.txt
-rw-r--r-- 1 gailr rosenMRIGrp 321 Oct 3 13:38 log_fastvifi_verbose.txt
-rw-r--r-- 1 gailr rosenMRIGrp 62 Oct 3 13:38 output_hpv.bam
-rw-r--r-- 1 gailr rosenMRIGrp 0 Oct 3 13:38 output_hpv.fixed.trans.cs.bam
-rw-r--r-- 1 gailr rosenMRIGrp 0 Oct 3 13:38 output_hpv.viral.cs.bam
-rw-r--r-- 1 gailr rosenMRIGrp 3648 Oct 3 13:38 reads_passing_kraken_filter_for_virus_hpv_1.fq
-rw-r--r-- 1 gailr rosenMRIGrp 3648 Oct 3 13:38 reads_passing_kraken_filter_for_virus_hpv_2.fq
-rw-r--r-- 1 gailr rosenMRIGrp 3438 Oct 3 13:38 reads_passing_kraken_first_level_for_virus_hpv_1.fq
-rw-r--r-- 1 gailr rosenMRIGrp 3438 Oct 3 13:38 reads_passing_kraken_first_level_for_virus_hpv_2.fq
-rw-r--r-- 1 gailr rosenMRIGrp 1184 Oct 2 19:23 test_fastvifi.sh
-rw-r--r-- 1 gailr rosenMRIGrp 6129 Oct 2 19:23 test_reads_1.fq
-rw-r--r-- 1 gailr rosenMRIGrp 6129 Oct 2 19:23 test_reads_2.fq
drwxrwsrwx 4 gailr rosenMRIGrp 44 Oct 3 13:25 tmp

@gailrosen
Copy link
Author

for your reference, here is the full output

(pysam) [gailr@node042 FastViFi]$ python3 run_kraken_vifi_container.py --singularity --skip-bwa
-filter --input-file /ifs/groups/rosenMRIGrp/gailr/FastViFi/test/test_reads_1.fq --input-file-2 /if
s/groups/rosenMRIGrp/gailr/FastViFi/test/test_reads_2.fq --output-dir test --virus hpv --kr
aken-db-path /ifs/groups/rosenMRIGrp/gailr/FastViFi/kraken_datasets --vifi-viral-ref-dir /ifs/group
s/rosenMRIGrp/gailr/FastViFi/viral_data --human-chr-list /ifs/groups/rosenMRIGrp/gailr/FastViFi/tes
t/human_chr_list.txt --vifi-human-ref-dir /ifs/groups/rosenMRIGrp/gailr/FastViFi/data_repo > vifi_e
rrors
Loading database information... done.
19 sequences (0.00 Mbp) processed in 0.005s (248.7 Kseq/m, 39.27 Mbp/m).
19 sequences classified (100.00%)
0 sequences unclassified (0.00%)
Command being timed: "/home/kraken2/kraken2 --use-names --report /dev/null --db /home/kraken2-d
b/Kraken2StandardDB_k_25_hpv_hg --threads 1 --paired --f-threshold 0.4 --keep-unmapped-reads --unmapped
-threshold 0.8 --classified-out /home/output/reads_passing_kraken_first_level_for_virus_hpv#.fq /home/o
utput/input_file_1.fq /home/output/input_file_2.fq --output /dev/null"
User time (seconds): 0.01
System time (seconds): 1.29
Percent of CPU this job got: 98%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:01.32
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 4167580
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 19
Minor (reclaiming a frame) page faults: 4558
Voluntary context switches: 145
Involuntary context switches: 60
Swaps: 0
File system inputs: 4094
File system outputs: 16
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
Loading database information... done.
10 sequences (0.00 Mbp) processed in 0.004s (144.8 Kseq/m, 43.43 Mbp/m).
10 sequences classified (100.00%)
0 sequences unclassified (0.00%)
Command being timed: "/home/kraken2/kraken2 --use-names --report /dev/null --db /home/kraken2-db/Kraken2StandardDB_k_22_hpv --threads 1 --paired --f-threshold 0.5 --unmapped-threshold 0.9 --classified-out /home/output/reads_passing_kraken_filter_for_virus_hpv#.fq /home/output/reads_passing_kraken_first_level_for_virus_hpv_1.fq /home/output/reads_passing_kraken_first_level_for_virus_hpv_2.fq --output /dev/null"
User time (seconds): 0.01
System time (seconds): 0.00
Percent of CPU this job got: 66%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.02
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 9556
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 2085
Voluntary context switches: 18
Involuntary context switches: 0
Swaps: 0
File system inputs: 0
File system outputs: 16
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
[E::bwa_idx_load_from_disk] fail to locate the index files
Traceback (most recent call last):
File "/home/ViFi/scripts/get_trans_new.py", line 104, in
bamFile = pysam.Samfile(opts.dataName[0], 'rb')
File "pysam/libcalignmentfile.pyx", line 747, in pysam.libcalignmentfile.AlignmentFile.cinit
File "pysam/libcalignmentfile.pyx", line 996, in pysam.libcalignmentfile.AlignmentFile._open
ValueError: file has no sequences defined (mode='rb') - is it SAM/BAM format? Consider opening with check_sq=False
[E::hts_open_format] Failed to open file "/home/output/output_hpv.unknown.bam" : No such file or directory
Traceback (most recent call last):
File "/home/ViFi/scripts/run_hmms.py", line 115, in
run_pipeline(options)
File "/home/ViFi/scripts/run_hmms.py", line 37, in run_pipeline
prepare_unmapped_sequences(options)
File "/home/ViFi/scripts/run_hmms.py", line 96, in prepare_unmapped_sequences
bam = pysam.Samfile(options.bamfile, 'rb')
File "pysam/libcalignmentfile.pyx", line 747, in pysam.libcalignmentfile.AlignmentFile.cinit
File "pysam/libcalignmentfile.pyx", line 946, in pysam.libcalignmentfile.AlignmentFile._open
FileNotFoundError: [Errno 2] could not open alignment file /home/output/output_hpv.unknown.bam: No such file or directory
[E::hts_open_format] Failed to open file "/home/output/output_hpv.trans.bam" : No such file or directory
Traceback (most recent call last):
File "/home/ViFi/scripts/merge_viral_reads.py", line 117, in
transFile = pysam.Samfile(args.transName[0], 'rb')
File "pysam/libcalignmentfile.pyx", line 747, in pysam.libcalignmentfile.AlignmentFile.cinit
File "pysam/libcalignmentfile.pyx", line 946, in pysam.libcalignmentfile.AlignmentFile._open
FileNotFoundError: [Errno 2] could not open alignment file /home/output/output_hpv.trans.bam: No such file or directory
[E::hts_open_format] Failed to open file /home/output/output_hpv.fixed.trans.bam
samtools sort: can't open "/home/output/output_hpv.fixed.trans.bam": No such file or directory
samtools index: "/home/output/output_hpv.fixed.trans.cs.bam" is in a format that cannot be usefully indexed
[E::hts_open_format] Failed to open file /home/output/output_hpv.viral.bam
samtools sort: can't open "/home/output/output_hpv.viral.bam": No such file or directory
samtools index: "/home/output/output_hpv.viral.cs.bam" is in a format that cannot be usefully indexed
WARNING:root:#TIME 0.061 Unable to find reference in $AA_DATA_REPO/reference.txt. Setting to working directory.
WARNING:root:#TIME 0.061 Unable to find reference in $AA_DATA_REPO/REF/file_list.txt. Setting to empty.
[E::fai_build3_core] Failed to open the file /home/data_repo//
WARNING:root:#TIME 0.061 Unable to open fasta file: "/home/data_repo//". Reference sequences will be set to N.
WARNING:root:#TIME 0.061 Unable to open chromosome lengths file: "/home/data_repo//"
WARNING:root:#TIME 0.061 interval_list: Unable to open interval file "/home/data_repo//".
WARNING:root:#TIME 0.061 interval_list: Unable to open interval file "/home/data_repo//".
WARNING:root:#TIME 0.061 interval_list: Unable to open interval file "/home/data_repo//".
WARNING:root:#TIME 0.061 interval_list: Unable to open interval file "/home/data_repo//".
WARNING:root:#TIME 0.061 interval_list: Unable to open interval file "/home/data_repo//".
Traceback (most recent call last):
File "/home/ViFi/scripts/cluster_trans_new.py", line 37, in
bamFile = pysam.Samfile(args.dataName[0], 'rb')
File "pysam/libcalignmentfile.pyx", line 747, in pysam.libcalignmentfile.AlignmentFile.cinit
File "pysam/libcalignmentfile.pyx", line 952, in pysam.libcalignmentfile.AlignmentFile._open
ValueError: file does not contain alignment data
Command being timed: "python /home/ViFi/scripts/run_vifi.py -f /home/output/reads_passing_kraken_filter_for_virus_hpv_1.fq -r /home/output/reads_passing_kraken_filter_for_virus_hpv_2.fq -o /home/output --virus hpv -c 1 --prefix output_hpv"
User time (seconds): 0.27
System time (seconds): 0.12
Percent of CPU this job got: 93%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.42
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 21024
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 48
Minor (reclaiming a frame) page faults: 17487
Voluntary context switches: 373
Involuntary context switches: 35
Swaps: 0
File system inputs: 10072
File system outputs: 24
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
[E::hts_open_format] Failed to open file "/home/output/output_hpv.viral.bam" : No such file or directory
Traceback (most recent call last):
File "/home/fastvifi/run_kraken_vifi_pipeline.py", line 491, in
run_pipeline(args)
File "/home/fastvifi/run_kraken_vifi_pipeline.py", line 482, in run_pipeline
bwa_filtered_fq_filename_2=bwa_filtered_fq_filename_2)
File "/home/fastvifi/run_kraken_vifi_pipeline.py", line 348, in run_kraken_vifi
vifi_input_fq_1=vifi_input_fq_1)
File "/home/fastvifi/run_kraken_vifi_pipeline.py", line 380, in write_viral_reads_report
viral_aligned_file = pysam.AlignmentFile(os.path.join(args.output_dir, "{}.viral.bam".format(prefix)), 'rb')
File "pysam/libcalignmentfile.pyx", line 747, in pysam.libcalignmentfile.AlignmentFile.cinit
File "pysam/libcalignmentfile.pyx", line 946, in pysam.libcalignmentfile.AlignmentFile._open
FileNotFoundError: [Errno 2] could not open alignment file /home/output/output_hpv.viral.bam: No such file or directory
Traceback (most recent call last):
File "run_kraken_vifi_container.py", line 95, in
call_fastvifi_pipeline(args)
File "run_kraken_vifi_container.py", line 88, in call_fastvifi_pipeline
shell_output = subprocess.check_output(
File "/ifs/opt/python/anaconda3/lib/python3.8/subprocess.py", line 411, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
File "/ifs/opt/python/anaconda3/lib/python3.8/subprocess.py", line 512, in run
raise CalledProcessError(retcode, process.args,
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command ' singularity run --bind /ifs/groups/rosenMRIGrp/gailr/FastViFi/test:/home/input/ --bind /ifs/groups/rosenMRIGrp/gailr/FastViFi/test/test_reads_2.fq:/home/input/test_reads_2.fq --bind /ifs/groups/rosenMRIGrp/gailr/FastViFi/kraken_datasets:/home/kraken2-db --bind /ifs/groups/rosenMRIGrp/gailr/FastViFi/data_repo:/home/data_repo/ --bind /ifs/groups/rosenMRIGrp/gailr/FastViFi/viral_data:/home/repo/data/ --bind test:/home/output/ --bind ./run_kraken_vifi_pipeline.py:/home/fastvifi/run_kraken_vifi_pipeline.py fastvifi_v1.1.sif python /home/fastvifi/run_kraken_vifi_pipeline.py --kraken-path /home/kraken2/kraken2 --vifi-path /home/ViFi/scripts/run_vifi.py --output /home/output --human-chr-list /home/data_repo/GRCh38/chrom_list.txt --kraken-db-path /home/kraken2-db --docker --virus hpv --input-file /home/input/test_reads_1.fq --input-file-2 /home/input/test_reads_2.fq --skip-bwa-filter ' returned non-zero exit status 1.
(pysam) [gailr@node042 FastViFi]$

@sara-javadzadeh
Copy link
Owner

sara-javadzadeh commented Oct 12, 2023 via email

@gailrosen
Copy link
Author

gailrosen commented Oct 24, 2023 via email

@v-mukhina v-mukhina mentioned this issue Mar 5, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@gailrosen @sara-javadzadeh