refactor: Update the list of files to be stored and delivered

[ivadym]

This PR:

Updates the list of files that are generated and stored by BALSAMIC.

• Removed from .hk file:

  • fastp rule:
    • _1.fp.fastq.gz
    • _2.fp.fastq.gz
    • _fastp.json
    • _fastp.html
  • bcftools_filter_tnscope_tumor_only rule:
    • .tnscope_isec.all.filtered.vcf.gz
  • bcftools_filter_tnhaplotyper_tumor_normal and bcftools_filter_tnhaplotyper_tumor_only rules:
    • .tnhaplotyper.all.filtered.vcf.gz
    • .tnhaplotyper.all.filtered.pass.vcf.gz

• Made temporal:

  • fastp rule:
    • _1.fp.fastq.gz
    • _2.fp.fastq.gz
  • fastp_umi rule:
    • _1.umi_optimized.fastq.gz
    • _2.umi_optimized.fastq.gz

Hermes PR: Clinical-Genomics/hermes#44

Review and tests:

• Tests pass
• Code review
• New code is executed and covered by tests, and test approve

[codecov]

Codecov Report

Merging #915 (62c590f) into develop (8142fac) will decrease coverage by 0.28%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##           develop     #915      +/-   ##
===========================================
- Coverage    99.54%   99.25%   -0.29%     
===========================================
  Files           29       29              
  Lines         1749     1749              
===========================================
- Hits          1741     1736       -5     
- Misses           8       13       +5     

Flag	Coverage Δ
unittests	99.25% <ø> (-0.29%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
BALSAMIC/constants/workflow_rules.py	100.00% <ø> (ø)
BALSAMIC/utils/rule.py	94.01% <0.00%> (-4.28%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8142fac...62c590f. Read the comment docs.

[ashwini06]

@ivadym : Can you also please link the related Hermes PR to this PR.

[ivadym]

Linking the Hermes PR: Clinical-Genomics/hermes#44

[ashwini06]

@ivadym : Is it possible to fix the tests for few of these missing lines? https://app.codecov.io/gh/Clinical-Genomics/BALSAMIC/compare/915/changes

[ivadym]

@ivadym : Is it possible to fix the tests for few of these missing lines? https://app.codecov.io/gh/Clinical-Genomics/BALSAMIC/compare/915/changes

I was trying to address it and I added an additional test which covers the lines which codecov was complaining about.

BALSAMIC/tests/commands/report/test_deliver.py

Lines 48 to 108 in 9ade4bf

	def test_deliver_tumor_only_panel_files(
	invoke_cli,
	environ,
	tumor_only_config,
	helpers,
	sentieon_install_dir,
	sentieon_license,
	caplog,
	):
	# GIVEN a tumor-normal config file
	helpers.read_config(tumor_only_config)
	actual_delivery_report = Path(helpers.delivery_dir, helpers.case_id + ".hk")
	# GIVEN a list of expected files to be included in the report
	delivery_files = [
	"sample_tumor_only.json",
	"sample_tumor_only_report.html",
	"sample_tumor_only_BALSAMIC_" + balsamic_version + "_graph.pdf",
	"multiqc_data.json",
	"sample_tumor_only_metrics_deliverables.yaml",
	]
	# GIVEN a compound tag that should be added to the report
	compound_tags = [
	["balsamic-config"],
	["balsamic-report"],
	["balsamic-dag"],
	["multiqc-json", "json"],
	["qc-metrics-yaml", "yaml"],
	]
	with mock.patch.dict(
	environ,
	{
	"SENTIEON_LICENSE": sentieon_license,
	"SENTIEON_INSTALL_DIR": sentieon_install_dir,
	},
	), caplog.at_level(logging.DEBUG):
	# WHEN running analysis
	result = invoke_cli(
	[
	"report",
	"deliver",
	"--sample-config",
	tumor_only_config,
	"--disable-variant-caller",
	"cnvkit",
	]
	)
	# THEN it should run without any errors
	assert result.exit_code == 0
	assert actual_delivery_report.is_file()
	with open(actual_delivery_report, "r") as report:
	report = json.load(report)["files"]
	assert report
	# THEN all the expected files and tags should be included in the report
	for file in report:
	assert os.path.basename(file["path"]) in delivery_files
	assert file["tag"] in compound_tags or file["tag"][::-1] in compound_tags

To test it explicitly we would need to call get_rule_output and provide a snakemake.common.Rules objectas an input. Any ideas of how to create a mock Rules object?

[ivadym]

Quality control results:

/qc/sample_tumor_normal_wgs_metrics_deliverables.yaml
/qc/multiqc_report.html
/qc/multiqc_data/multiqc_data.json

Analysis specific results:

• TN-WGS

        /vep/SNV.germline.tumor.dnascope.vcf.gz
        /vep/SNV.germline.normal.dnascope.vcf.gz
        /vep/SV.germline.tumor.manta_germline.vcf.gz
        /vep/SV.germline.normal.manta_germline.vcf.gz
        /vcf/SNV.somatic.sample_tumor_normal_wgs.tnscope.vcf.gz
        /vcf/SNV.somatic.sample_tumor_normal_wgs.tnhaplotyper.vcf.gz
        /vcf/SV.somatic.sample_tumor_normal_wgs.svdb.vcf.gz
        /vep/SNV.somatic.sample_tumor_normal_wgs.tnscope.all.filtered.pass.vcf.gz
        /vep/SNV.somatic.sample_tumor_normal_wgs.tnhaplotyper.all.filtered.pass.vcf.gz
        /vep/SV.somatic.sample_tumor_normal_wgs.svdb.all.filtered.pass.vcf.gz
        /vep/SNV.somatic.sample_tumor_normal_wgs.tnscope.balsamic_stat
        /vep/SNV.somatic.sample_tumor_normal_wgs.tnhaplotyper.balsamic_stat
        /vcf/CNV.somatic.sample_tumor_normal_wgs.ascat.output.pdf
        /vcf/CNV.somatic.sample_tumor_normal_wgs.ascat.copynumber.txt.gz

• TN-TGA

        /vep/SNV.germline.tumor.haplotypecaller.vcf.gz
        /vep/SNV.germline.normal.haplotypecaller.vcf.gz
        /vep/SNV.germline.tumor.dnascope.vcf.gz
        /vep/SNV.germline.normal.dnascope.vcf.gz
        /vep/SV.germline.tumor.manta_germline.vcf.gz
        /vep/SV.germline.normal.manta_germline.vcf.gz
        /vcf/SNV.somatic.sample_tumor_normal.TNscope_umi.vcf.gz
        /vcf/SNV.somatic.sample_tumor_normal.vardict.vcf.gz
        /vcf/SNV.somatic.sample_tumor_normal.tnhaplotyper.vcf.gz
        /vcf/SV.somatic.sample_tumor_normal.svdb.vcf.gz
        /vep/SNV.somatic.sample_tumor_normal.TNscope_umi.all.filtered.pass.vcf.gz
        /vep/SNV.somatic.sample_tumor_normal.vardict.all.filtered.pass.vcf.gz
        /vep/SNV.somatic.sample_tumor_normal.tnhaplotyper.all.filtered.pass.vcf.gz
        /vep/SV.somatic.sample_tumor_normal.svdb.all.filtered.pass.vcf.gz
        /vep/SNV.somatic.sample_tumor_normal.vardict.balsamic_stat
        /vep/SNV.somatic.sample_tumor_normal.tnhaplotyper.balsamic_stat
        /vep/SNV.somatic.sample_tumor_normal.TNscope_umi.balsamic_stat
        /cnv/tumor.merged.cns
        /cnv/tumor.merged-diagram.pdf
        /cnv/tumor.merged-scatter.pdf
        /cnv/sample_tumor_normal.gene_metrics
        /vcf/CNV.somatic.sample_tumor_normal.cnvkit.vcf2cytosure.cgh
        /vep/SNV.somatic.sample_tumor_normal.vardict.all.filtered.pass.ranked.vcf.gz
        /qc/umi_qc/concatenated_normal_XXXXXX_R.umi.mean_family_depth
        /qc/umi_qc/concatenated_tumor_XXXXXX_R.umi.mean_family_depth
        /qc/umi_qc/sample_tumor_normal.TNscope_umi.AFtable.txt

[ivadym]

Files and tags are parsed and generated as expected:

• WGS_TN

• TGA_TN

[ashwini06]

Looks good 🎇

[ashwini06]

Files and tags are parsed and generated as expected:

• WGS_TN

• TGA_TN

@ivadym What does the warning message "Cannot deliver step (rule) ..." refers to?

[ivadym]

@ivadym What does the warning message "Cannot deliver step (rule) ..." refers to?

It means that the rule it tries to deliver is not among the list of all the rules executed during the analysis.

If you notice we are going through all the rules that we are going to deliver. However, among all of them we can find some that are specific to the analysis type. Therefore, for TN_WGS, for example, you cannot find the rules associated with the UMIs.

BALSAMIC/BALSAMIC/workflows/balsamic.smk

Lines 299 to 304 in 8244f83

	for my_rule in set(rules_to_deliver):
	try:
	housekeeper_id = getattr(rules, my_rule).params.housekeeper_id
	except (ValueError, AttributeError, RuleException, WorkflowError) as e:
	LOG.warning("Cannot deliver step (rule) {}: {}".format(my_rule, e))
	continue

[khurrammaqbool]

👍 .
As we discussed, let's refactor to decouple delivery after this.

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication