Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Attempt to run on Colab CPU - .bam files missing! #365

Open
nickkolok opened this issue Jan 28, 2025 · 0 comments
Open

Attempt to run on Colab CPU - .bam files missing! #365

nickkolok opened this issue Jan 28, 2025 · 0 comments

Comments

@nickkolok
Copy link

Dear AidenLab,

The amount of effort you put in juicer looks really incredible. Being a total newbie, I try to run juicer on Google Colab in CPU mode. And yeah, Google Colab is really a gold standard now, so I've supposed there would be no entanglements - just as there were no problems with the HiCCUPS colab referenced in the wiki (thanks!).

As you can infer, I was wrong.

Here is my Colab code:

# Install dependencies (we have Linux Ubuntu here)
!apt install samtools bwa

# Clone the repository
!git clone https://github.com/aidenlab/juicer.git

# Arrange generic folder structure
!mkdir references fastq restriction_sites aligned hic_results

# Get the examples
# Note the -P trick:
# it's not that easy to perform a long-playing `cd` in colab!
!wget -P references https://s3.amazonaws.com/juicerawsmirror/opt/juicer/references/Homo_sapiens_assembly19.fasta
!wget -P references https://s3.amazonaws.com/juicerawsmirror/opt/juicer/references/Homo_sapiens_assembly19.fasta.amb
!wget -P references https://s3.amazonaws.com/juicerawsmirror/opt/juicer/references/Homo_sapiens_assembly19.fasta.ann
!wget -P references https://s3.amazonaws.com/juicerawsmirror/opt/juicer/references/Homo_sapiens_assembly19.fasta.bwt
!wget -P references https://s3.amazonaws.com/juicerawsmirror/opt/juicer/references/Homo_sapiens_assembly19.fasta.pac
!wget -P references https://s3.amazonaws.com/juicerawsmirror/opt/juicer/references/Homo_sapiens_assembly19.fasta.sa
!wget -P restriction_sites https://s3.amazonaws.com/juicerawsmirror/opt/juicer/restriction_sites/hg19_MboI.txt

# In Colab, we run in CPU mode
!ln -s juicer/CPU scripts

# Do I need to add `/common/` ?..
!cd scripts; wget https://hicfiles.tc4ga.com/public/juicer/juicer_tools.1.9.9_jcuda.0.8.jar
!cd scripts; ln -s juicer_tools.1.9.9_jcuda.0.8.jar  juicer_tools.jar

# It insists...
!rm -rf aligned

!./scripts/juicer.sh -D `pwd` -g "hg19"

It is based on https://github.com/aidenlab/juicer/wiki/Installation

I get the following output:

Details
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  libhts3 libhtscodecs2
Suggested packages:
  cwltool
The following NEW packages will be installed:
  bwa libhts3 libhtscodecs2 samtools
0 upgraded, 4 newly installed, 0 to remove and 49 not upgraded.
Need to get 1,158 kB of archives.
After this operation, 2,736 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 bwa amd64 0.7.17-6 [195 kB]
Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libhtscodecs2 amd64 1.1.1-3 [53.2 kB]
Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libhts3 amd64 1.13+ds-2build1 [390 kB]
Get:4 http://archive.ubuntu.com/ubuntu jammy/universe amd64 samtools amd64 1.13-4 [520 kB]
Fetched 1,158 kB in 2s (480 kB/s)
Selecting previously unselected package bwa.
(Reading database ... 124788 files and directories currently installed.)
Preparing to unpack .../bwa_0.7.17-6_amd64.deb ...
Unpacking bwa (0.7.17-6) ...
Selecting previously unselected package libhtscodecs2:amd64.
Preparing to unpack .../libhtscodecs2_1.1.1-3_amd64.deb ...
Unpacking libhtscodecs2:amd64 (1.1.1-3) ...
Selecting previously unselected package libhts3:amd64.
Preparing to unpack .../libhts3_1.13+ds-2build1_amd64.deb ...
Unpacking libhts3:amd64 (1.13+ds-2build1) ...
Selecting previously unselected package samtools.
Preparing to unpack .../samtools_1.13-4_amd64.deb ...
Unpacking samtools (1.13-4) ...
Setting up libhtscodecs2:amd64 (1.1.1-3) ...
Setting up libhts3:amd64 (1.13+ds-2build1) ...
Setting up bwa (0.7.17-6) ...
Setting up samtools (1.13-4) ...
Processing triggers for man-db (2.10.2-1) ...
Processing triggers for libc-bin (2.35-0ubuntu3.4) ...
/sbin/ldconfig.real: /usr/local/lib/libhwloc.so.15 is not a symbolic link

/sbin/ldconfig.real: /usr/local/lib/libur_adapter_level_zero.so.0 is not a symbolic link

/sbin/ldconfig.real: /usr/local/lib/libtcm_debug.so.1 is not a symbolic link

/sbin/ldconfig.real: /usr/local/lib/libtbbmalloc.so.2 is not a symbolic link

/sbin/ldconfig.real: /usr/local/lib/libur_adapter_opencl.so.0 is not a symbolic link

/sbin/ldconfig.real: /usr/local/lib/libtbbbind_2_5.so.3 is not a symbolic link

/sbin/ldconfig.real: /usr/local/lib/libur_loader.so.0 is not a symbolic link

/sbin/ldconfig.real: /usr/local/lib/libtcm.so.1 is not a symbolic link

/sbin/ldconfig.real: /usr/local/lib/libtbbbind_2_0.so.3 is not a symbolic link

/sbin/ldconfig.real: /usr/local/lib/libtbbmalloc_proxy.so.2 is not a symbolic link

/sbin/ldconfig.real: /usr/local/lib/libumf.so.0 is not a symbolic link

/sbin/ldconfig.real: /usr/local/lib/libtbb.so.12 is not a symbolic link

/sbin/ldconfig.real: /usr/local/lib/libtbbbind.so.3 is not a symbolic link

Cloning into 'juicer'...
remote: Enumerating objects: 2245, done.
remote: Counting objects: 100% (33/33), done.
remote: Compressing objects: 100% (23/23), done.
remote: Total 2245 (delta 15), reused 19 (delta 10), pack-reused 2212 (from 2)
Receiving objects: 100% (2245/2245), 79.16 MiB | 21.84 MiB/s, done.
Resolving deltas: 100% (1305/1305), done.
--2025-01-27 23:48:29--  https://s3.amazonaws.com/juicerawsmirror/opt/juicer/references/Homo_sapiens_assembly19.fasta
Resolving s3.amazonaws.com (s3.amazonaws.com)... 52.216.212.120, 54.231.132.112, 54.231.133.232, ...
Connecting to s3.amazonaws.com (s3.amazonaws.com)|52.216.212.120|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3140756381 (2.9G) [text/plain]
Saving to: ‘references/Homo_sapiens_assembly19.fasta’

Homo_sapiens_assemb 100%[===================>]   2.92G  16.9MB/s    in 3m 14s  

2025-01-27 23:51:44 (15.4 MB/s) - ‘references/Homo_sapiens_assembly19.fasta’ saved [3140756381/3140756381]

--2025-01-27 23:51:44--  https://s3.amazonaws.com/juicerawsmirror/opt/juicer/references/Homo_sapiens_assembly19.fasta.amb
Resolving s3.amazonaws.com (s3.amazonaws.com)... 52.216.217.56, 52.216.170.229, 16.15.193.182, ...
Connecting to s3.amazonaws.com (s3.amazonaws.com)|52.216.217.56|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 6597 (6.4K) [text/plain]
Saving to: ‘references/Homo_sapiens_assembly19.fasta.amb’

Homo_sapiens_assemb 100%[===================>]   6.44K  --.-KB/s    in 0s      

2025-01-27 23:51:45 (142 MB/s) - ‘references/Homo_sapiens_assembly19.fasta.amb’ saved [6597/6597]

--2025-01-27 23:51:45--  https://s3.amazonaws.com/juicerawsmirror/opt/juicer/references/Homo_sapiens_assembly19.fasta.ann
Resolving s3.amazonaws.com (s3.amazonaws.com)... 52.216.217.56, 52.216.170.229, 16.15.193.182, ...
Connecting to s3.amazonaws.com (s3.amazonaws.com)|52.216.217.56|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 6901 (6.7K) [text/plain]
Saving to: ‘references/Homo_sapiens_assembly19.fasta.ann’

Homo_sapiens_assemb 100%[===================>]   6.74K  --.-KB/s    in 0s      

2025-01-27 23:51:46 (196 MB/s) - ‘references/Homo_sapiens_assembly19.fasta.ann’ saved [6901/6901]

--2025-01-27 23:51:46--  https://s3.amazonaws.com/juicerawsmirror/opt/juicer/references/Homo_sapiens_assembly19.fasta.bwt
Resolving s3.amazonaws.com (s3.amazonaws.com)... 52.216.217.56, 52.216.170.229, 16.15.193.182, ...
Connecting to s3.amazonaws.com (s3.amazonaws.com)|52.216.217.56|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3101976644 (2.9G) [application/octet-stream]
Saving to: ‘references/Homo_sapiens_assembly19.fasta.bwt’

Homo_sapiens_assemb 100%[===================>]   2.89G  16.6MB/s    in 3m 3s   

2025-01-27 23:54:50 (16.1 MB/s) - ‘references/Homo_sapiens_assembly19.fasta.bwt’ saved [3101976644/3101976644]

--2025-01-27 23:54:50--  https://s3.amazonaws.com/juicerawsmirror/opt/juicer/references/Homo_sapiens_assembly19.fasta.pac
Resolving s3.amazonaws.com (s3.amazonaws.com)... 52.216.10.149, 3.5.30.166, 52.217.192.64, ...
Connecting to s3.amazonaws.com (s3.amazonaws.com)|52.216.10.149|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 775494142 (740M) [application/octet-stream]
Saving to: ‘references/Homo_sapiens_assembly19.fasta.pac’

Homo_sapiens_assemb 100%[===================>] 739.57M  17.1MB/s    in 44s     

2025-01-27 23:55:35 (16.9 MB/s) - ‘references/Homo_sapiens_assembly19.fasta.pac’ saved [775494142/775494142]

--2025-01-27 23:55:35--  https://s3.amazonaws.com/juicerawsmirror/opt/juicer/references/Homo_sapiens_assembly19.fasta.sa
Resolving s3.amazonaws.com (s3.amazonaws.com)... 52.217.191.8, 16.15.179.202, 52.217.171.200, ...
Connecting to s3.amazonaws.com (s3.amazonaws.com)|52.217.191.8|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1550988336 (1.4G) [application/octet-stream]
Saving to: ‘references/Homo_sapiens_assembly19.fasta.sa’

Homo_sapiens_assemb 100%[===================>]   1.44G  16.3MB/s    in 92s     

2025-01-27 23:57:08 (16.1 MB/s) - ‘references/Homo_sapiens_assembly19.fasta.sa’ saved [1550988336/1550988336]

--2025-01-27 23:57:08--  https://s3.amazonaws.com/juicerawsmirror/opt/juicer/restriction_sites/hg19_MboI.txt
Resolving s3.amazonaws.com (s3.amazonaws.com)... 52.216.215.56, 3.5.21.69, 52.217.116.176, ...
Connecting to s3.amazonaws.com (s3.amazonaws.com)|52.216.215.56|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 65953989 (63M) [text/plain]
Saving to: ‘restriction_sites/hg19_MboI.txt’

hg19_MboI.txt       100%[===================>]  62.90M  15.9MB/s    in 5.2s    

2025-01-27 23:57:14 (12.2 MB/s) - ‘restriction_sites/hg19_MboI.txt’ saved [65953989/65953989]

--2025-01-27 23:57:14--  https://hicfiles.tc4ga.com/public/juicer/juicer_tools.1.9.9_jcuda.0.8.jar
Resolving hicfiles.tc4ga.com (hicfiles.tc4ga.com)... 172.67.219.232, 104.21.24.187, 2606:4700:3034::6815:18bb, ...
Connecting to hicfiles.tc4ga.com (hicfiles.tc4ga.com)|172.67.219.232|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 32048230 (31M) [application/x-www-form-urlencoded]
Saving to: ‘juicer_tools.1.9.9_jcuda.0.8.jar’

juicer_tools.1.9.9_ 100%[===================>]  30.56M  9.43MB/s    in 3.2s    

2025-01-27 23:57:19 (9.43 MB/s) - ‘juicer_tools.1.9.9_jcuda.0.8.jar’ saved [32048230/32048230]

(-: Aligning files matching /content/fastq/*_R*.fastq*
 to genome /content/references/Homo_sapiens_assembly19.fasta with no fragment delimited maps.
[E::hts_open_format] Failed to open file "/content/splits/*.bam" : No such file or directory
samtools merge: fail to open "/content/splits/*.bam": No such file or directory
***! Some problems occurred somewhere in creating sorted align files.

What am I doing wrong?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant