GATK4 first round without MuTect1 and indel realignment #607

szilvajuhos · 2018-07-17T09:35:49Z

Also have a look at the new container structure. I am trying to accommodate nf-core guidelines. alleleCount and ASCAT needs new bioconda recipes, but most of the other tools are in a collated (relatively big) container including GATK4, igvtools, etc.

…l/29/files

…ow error. Maybe NXF bug?

maxulysse · 2018-07-17T09:54:14Z

annotate.nf

@@ -73,23 +73,23 @@ vcfToAnnotate = Channel.create()
 vcfNotToAnnotate = Channel.create()

 if (annotateVCF == []) {
+// by default we annotate both germline and somatic results that we can find in the VariantCalling directory


In fact, we annote all available vcfs by default, so it's not really a question of germline/somatic, but really more a question of which tools was run

maxulysse · 2018-07-17T10:06:16Z

containers/sarek/Dockerfile


 LABEL \
-  author="Maxime Garcia" \
+  authors="Maxime.Gracia@scilifelab.se, Szilveszter.Juhos@scilifelab.se" \


typo in my name.

maxulysse · 2018-07-17T10:08:02Z

containers/sarek/Dockerfile

  description="Image with tools used in Sarek" \
-  maintainer="maxime.garcia@scilifelab.se"
+  maintainers="Maxime.Gracia@scilifelab.se, Szilveszter.Juhos@scilifelab.se"


If I remember well, we can use whichever label we want, but maintainer is meant to stay that way, because it's a port of the deprecated instruction MAINTAINER.
I do think we can use:

maintainer="Maxime Garcia <maxime.garcia@scilifelab.se>, Szilveszter Juhos <Szilveszter.Juhos@scilifelab.se>"

maxulysse · 2018-07-17T10:09:23Z

containers/vcfanno/to_build

+docker build -t szilvajuhos/sarek-vcfanno:latest .
+docker images
+docker push szilvajuhos/sarek-vcfanno:latest
+singularity pull docker://szilvajuhos/sarek-vcfanno:latest


Do we really need this script in this repo ?

No, of course not

maxulysse · 2018-07-17T10:10:18Z

germlineVC.nf

+      -L ${intervalBed} \
+  		--dbsnp ${dbsnp} \
+  		-O ${intervalBed.baseName}_${idSample}.g.vcf 	\
+			--emit-ref-confidence GVCF


strange indentation

maxulysse · 2018-07-17T10:12:25Z

lib/QC.groovy

@@ -59,14 +59,14 @@ class QC {
 // Get GATK version
  static def getVersionGATK() {
    """
-    echo "GATK version"\$(java -jar \$GATK_HOME/GenomeAnalysisTK.jar --version 2>&1) > v_gatk.txt
+		gatk-launch ApplyBQSR --help 2>&1| awk -F/ '/java/{for(i=1;i<=NF;i++){if(\$i~/gatk4/){sub("gatk4-","",\$i);print \$i>"v_gatk.txt"}}}'


I think we can work out the regex in the Python script instead of doing it here, it'll make more sense

the gatk-launch is GATK-provided, I do not want to fiddle with that. OTOH it would be nice if they would have a --version option :/

I'll look more if there's something similar with the new GATK

My feeling is that it is still the easiest way to have the version :/

I was thinking more of removing the awk part, and do the regex in the python script

I see, sure, we can refactor it for the rest of the software later also.

maxulysse · 2018-07-17T10:22:51Z

Quite an impressive work.
Well done @szilvajuhos \o/

maxulysse · 2018-07-19T10:32:32Z

containers/sarek/environment.yml

+  - conda-forge::openjdk=8.0.144 # Needed for FastQC docker - see bioconda/bioconda-recipes#5026
+  - fastqc=0.11.7
+  - freebayes=1.2.0
+  - gatk4=4.0.3.0


You can use the 4.0.4.0, the executable is back to being gatk and not gatck-launch anymore

Fine, will change the name in processes as well. In fact we have 4.0.6.0 also

Even better ;-)

maxulysse · 2018-07-19T10:34:56Z

containers/sarek/environment.yml

+  - fastqc=0.11.7
+  - freebayes=1.2.0
+  - gatk4=4.0.3.0
+  - htslib=1.7


You should use the 1.8 here.
If I remember well, htslib, bcftools and samtools can all have the same version

OK, done but will check since I got a feeling that 1.8 has compatibility issues

OK, strange, but good to know if you can confirm that

There is even 1.9 out already! You could already skip 1.8 ...
https://github.com/samtools/samtools/releases/

maxulysse · 2018-07-19T10:36:01Z

containers/sarek/environment.yml

+  - gatk4=4.0.3.0
+  - htslib=1.7
+  - igvtools=2.3.93
+  - manta=1.3.0


Since we're updating, we can try the 1.4.0

maxulysse · 2018-08-01T07:50:38Z

configuration/uppmax-localhost.config

@@ -14,7 +14,7 @@ env {
 params {
  genome_base = params.genome == 'GRCh37' ? '/sw/data/uppnex/ToolBox/ReferenceAssemblies/hg38make/bundle/2.8/b37' : params.genome == 'GRCh38' ? '/sw/data/uppnex/ToolBox/hg38bundle' : 'References/smallGRCh37'
  singleCPUMem  = 8.GB
-  totalMemory = 104.GB // change to 240 on irma
+  totalMemory = 92.GB // change to 240 on irma


Why did you change to 92?

maxulysse · 2018-08-01T07:59:52Z

containers/sarek/Dockerfile

-  build.sh
+COPY environment.yml /
+RUN conda env update -n root -f /environment.yml && conda clean -a
+ENV PATH /opt/conda/bin:$PATH


we can get rid of this last line:
https://gitter.im/nf-core/Lobby?at=5b59f41bd2f0934551d30d5d

tried, and was not working as expected, so I prefer to leave it as it is now, and improve when needed

we will keep now with the ENV

maxulysse · 2018-08-01T08:00:44Z

containers/sarek/environment.yml

@@ -0,0 +1,24 @@
+# You can use this file to create a conda environment for this pipeline:
+# conda env create -f environment.yml
+name: sarek-core


We should specify a version here
so I would go for sarek-core-dev or sarek-core-2.1

can we just leave as sarek ?

GATK4 first round without MuTect1 and indel realignment

Szilveszter Juhos and others added 18 commits July 5, 2018 09:21

QR for posters

c9b2cd9

Dockerand singularity images for vcfanno

428c893

removing superfluous routines, adding comments

6f438ad

logging improvements

e00872b

nfcore based image file containing vcfanno and qc modules also

f6c60bf

on a way to unify to nf-core and GATK4

a3f102d

It works without indelrealignment until CreateRecalibrationTable

576c171

new target directories for Prerocessing, works to the end of ApplyBQSR

0aea380

HaplotypeCaller works with GATK4

72a1b50

path and version fixed for both manta and strelka

c6137bf

Mutect2 GATK4 initial dumb version

bba73f1

Added genome index file

70a3584

Changed Dockerfile to follow https://github.com/nf-core/methylseq/pul…

a836057

…l/29/files

removed -Xmx4g because it was ignored anyway and made a syntax/workfl…

733e2c4

…ow error. Maybe NXF bug?

alphabetical order and added freebayes and new htslib version

449f939

freebayes and igvtools in the core container

3fb6dc6

set -euo pipefail needed also for germline

0c71f0a

die mutect1

da3be40

maxulysse reviewed Jul 17, 2018

View reviewed changes

Szilveszter Juhos added 3 commits July 17, 2018 20:15

commenting out AFs as it will be for a new sprint

4db0d5d

vcfanno is merged to the core

efd11c6

removed igvtools (as went to core)

e81e100

maxulysse reviewed Jul 19, 2018

View reviewed changes

Merge remote-tracking branch 'upstream/master'

0876a89

szilvajuhos added the do not merge label Jul 27, 2018

Szilveszter Juhos and others added 5 commits July 30, 2018 11:53

Not using the log util, as it is not working

7cf77b6

fixed indent

e2cacd7

gatk-launch rename

735463b

MarkDuplicates got all the memory

a64dd3c

Merge branch 'master' of github.com:szilvajuhos/Sarek

c14fa8e

maxulysse reviewed Aug 1, 2018

View reviewed changes

Szilveszter Juhos and others added 12 commits August 1, 2018 17:19

Latest versions

5ab4491

MarkDuplicates MAX_RECORDS_IN_RAM decreased to 50000

537faa8

Picard removed from buildReferences.nf

24b1ffb

adding all the cpus and mem to MarkDuplicates

a5b3f7c

update submodule Sarek-data a.k.a. #609

6ebed7a

s/gatk-launch/gatk/

58bab13

removing realign parts

12826fd

removing realign parts

b21dc68

small review changes

db137ce

Merge branch 'master' of github.com:szilvajuhos/Sarek

cb7e228

adding singularty environments for tests

97d3fc5

VEP fancy parametrisation

98cf879

szilvajuhos removed the do not merge label Aug 14, 2018

maxulysse merged commit 924b90a into SciLifeLab:master Aug 14, 2018

This was referenced Aug 14, 2018

Remove MuTecT1 #565

Closed

Rewrote docker containers using bioconda environment.yml ? #551

Closed

Szilveszter Juhos added 2 commits August 14, 2018 20:27

memory adjustments for samtools and MarkDuplicates

393da8b

s/sarek-core/sarek/ also deleting build.sh

9d20c68

jherrero referenced this pull request in UCL-BLIC/Sarek_v2.2.1 Apr 11, 2019

Merge pull request nf-core#607 from szilvajuhos/master

cbadab7

GATK4 first round without MuTect1 and indel realignment

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GATK4 first round without MuTect1 and indel realignment #607

GATK4 first round without MuTect1 and indel realignment #607

szilvajuhos commented Jul 17, 2018

maxulysse Jul 17, 2018

maxulysse Jul 17, 2018

maxulysse Jul 17, 2018

maxulysse Jul 17, 2018

szilvajuhos Jul 17, 2018

maxulysse Jul 17, 2018

maxulysse Jul 17, 2018

szilvajuhos Jul 17, 2018

maxulysse Jul 17, 2018

szilvajuhos Jul 31, 2018

maxulysse Jul 31, 2018

szilvajuhos Aug 1, 2018

maxulysse commented Jul 17, 2018

maxulysse Jul 19, 2018

szilvajuhos Jul 31, 2018

maxulysse Jul 31, 2018

maxulysse Jul 19, 2018

szilvajuhos Jul 31, 2018

maxulysse Jul 31, 2018

apeltzer Jul 31, 2018

maxulysse Jul 19, 2018

szilvajuhos Jul 31, 2018

maxulysse Aug 1, 2018

szilvajuhos Aug 2, 2018

maxulysse Aug 1, 2018

szilvajuhos Aug 2, 2018

szilvajuhos Aug 14, 2018

maxulysse Aug 1, 2018

szilvajuhos Aug 2, 2018

GATK4 first round without MuTect1 and indel realignment #607

GATK4 first round without MuTect1 and indel realignment #607

Conversation

szilvajuhos commented Jul 17, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maxulysse commented Jul 17, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment