fix: remove stat files and add count files for scout upload VCFs

[khurrammaqbool]

This PR:

Cleans up vep rules and adds stats files for all endpoint VCFs to be useful for automatic uploads of VCF files.

Added: *_counts files for each end point VCF file including *.all.filtered.pass.vcf.gz
Removed: *_stats files from vep rules

Review and tests:

• Tests pass
• Code review
• New code is executed and covered by tests, and test approve

[codecov]

Codecov Report

Merging #899 (6bb9976) into develop (cf98015) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff            @@
##           develop     #899   +/-   ##
========================================
  Coverage    99.54%   99.54%           
========================================
  Files           29       29           
  Lines         1775     1775           
========================================
  Hits          1767     1767           
  Misses           8        8           

Flag	Coverage Δ
unittests	99.54% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cf98015...6bb9976. Read the comment docs.

[khurrammaqbool]

@ivadym , I added stats files for all the filtered.passed VCF files and also added varcaller names to the output files.

[khurrammaqbool]

@ashwini06 and @ivadym, bcftools counts is added for all end point VCF files and bcftools stats is removed from vep rules.
This pull request is ready for review!

[ashwini06]

I added the comments about the count files for which we are not uploading to scout. It's up to you to either address it here or @ivadym if would like to take it in his upcoming PR. You both can decide on it :)

I wonder if this .all.filtered.pass.stats are correctly generated. Check with some dry runs. If not include the required file in

BALSAMIC/BALSAMIC/workflows/balsamic.smk

Line 262 in 164e3ae

#Gather all the filtered and PASSed variants post annotation

[khurrammaqbool]

@keyvanelhami could you please share the updated list of end-point VCF files to be uploaded to scout, we can use to generate counts for?

[keyvanelhami]

@khurrammaqbool We are using below flags and files for Scout upload. You can also look into an example of a yaml file for Scout: /home/proj/production/housekeeper-bundles/ablepoodle/2022-04-07/scout_load.yaml

vcf_cancer: /home/proj/production/housekeeper-bundles/ablepoodle/2022-04-07/SNV.somatic.ablepoodle.vardict.all.filtered.pass.vcf.gz
vcf_cancer_sv: /home/proj/production/housekeeper-bundles/ablepoodle/2022-04-07/SV.somatic.ablepoodle.manta.all.filtered.pass.vcf.gz

[khurrammaqbool]

/home/proj/production/housekeeper-bundles/ablepoodle/2022-04-07/scout_load.yaml

The list seems to include manta. We have decided to upload svdb instead. Isn't it an old list?

[keyvanelhami]

have decided to upload svdb instead. Isn't it an old list?

But is the svdb PR merged and deployed in production? If so we should open an issue so cg points to the svdb vcf instead

[khurrammaqbool]

have decided to upload svdb instead. Isn't it an old list?

But is the svdb PR merged and deployed in production? If so we should open an issue so cg points to the svdb vcf instead

It will be merged in develop and is linked to the delivery of files @ivadym is currently working with.

[ivadym]

Looks good to me 👍

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
No Duplication information