Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: Remove Balsamic container 👋 👋 #1220

Merged
merged 70 commits into from
Aug 10, 2023
Merged
Changes from 1 commit
Commits
Show all changes
70 commits
Select commit Hold shift + click to select a range
9e5bff2
Remove unused imports analysis models
ivadym Jul 18, 2023
fc5f85e
Create Snakemake model
ivadym Jul 20, 2023
10e721c
Update init workflow
ivadym Jul 21, 2023
2134fa1
Remove empty new line
ivadym Jul 21, 2023
c2e7257
Snakemake model integrity tests
ivadym Jul 24, 2023
5b1eae3
Models methods pytests
ivadym Jul 25, 2023
55ba76b
Remove unused pytests
ivadym Jul 25, 2023
edf56c0
Add benchmark as a model option
ivadym Jul 25, 2023
de0fe16
Update run analysis
ivadym Jul 25, 2023
c34d40c
Update deliver.py
ivadym Jul 25, 2023
393df1c
Rename fixtures
ivadym Jul 25, 2023
08214d0
Merge branch 'develop' of github.com:Clinical-Genomics/BALSAMIC into …
ivadym Jul 25, 2023
161803f
Rename command method name
ivadym Jul 25, 2023
033248c
Add additional method for config files
ivadym Jul 25, 2023
a919bab
Split subprocess submission command
ivadym Jul 25, 2023
e104ab5
CHANGELOG
ivadym Jul 25, 2023
a56da8d
1211
ivadym Jul 25, 2023
b4933f3
Pytest mail user
ivadym Jul 26, 2023
91ac79b
Test pon cnn
ivadym Jul 26, 2023
fc39962
Undo PON
ivadym Jul 26, 2023
753341b
Push containers
ivadym Jul 26, 2023
06bab07
Undo push
ivadym Jul 26, 2023
45f04d1
Merge develop
ivadym Jul 28, 2023
88e832a
Update CHANGELOG.rst
ivadym Jul 28, 2023
d5f4804
Update CHANGELOG.rst
ivadym Jul 28, 2023
4115fda
Review: Typehint comments singularity_bind_path
ivadym Jul 28, 2023
160d0ca
Move cache utils to utils :p
ivadym Jul 28, 2023
15cd6ce
Review: move methods
ivadym Jul 28, 2023
4d0f0d5
Merge branch 'refactor-bind-path-singularity' of github.com:Clinical-…
ivadym Jul 28, 2023
51ed5ba
pon pon pon poooon
ivadym Jul 28, 2023
efbdbd5
Missing capture kit key
ivadym Jul 28, 2023
ec44f43
TEST 0: bind path
ivadym Jul 28, 2023
105fc76
TEST 0: completed
ivadym Jul 28, 2023
cbd3cc0
Remove extra space :p
ivadym Jul 28, 2023
788a39c
Merge branch 'develop' into refactor-bind-path-singularity
ivadym Aug 1, 2023
8ca2356
Update dependency versions
ivadym Aug 4, 2023
948c3b6
Update balsamic container
ivadym Aug 4, 2023
24530cc
Update docs
ivadym Aug 4, 2023
aeb15d5
CHANGELOG
ivadym Aug 4, 2023
5dbc49d
Update conda environment
ivadym Aug 4, 2023
88c71ce
Push containers :rocket
ivadym Aug 4, 2023
1d8bfea
Add command to snakemake
ivadym Aug 4, 2023
19e1eaa
uppercase
ivadym Aug 4, 2023
3686747
Update GitHub Actions & docs python version
ivadym Aug 4, 2023
01c5f59
Fixed version
ivadym Aug 4, 2023
9e325e6
Update pytests
ivadym Aug 4, 2023
4607748
Fix
ivadym Aug 7, 2023
a7dbda0
CHANGELOG
ivadym Aug 7, 2023
b50e182
Merge branch 'fix-rm-temp' of github.com:Clinical-Genomics/BALSAMIC i…
ivadym Aug 7, 2023
1eb45d3
Merge branch 'fix-rm-temp' into refactor-bind-path-singularity
ivadym Aug 7, 2023
e20645a
Merge branch 'refactor-bind-path-singularity' into fix-snakemake-rerun
ivadym Aug 7, 2023
33ca244
Merge branch 'fix-snakemake-rerun' into update-python
ivadym Aug 7, 2023
9b04af9
Review
ivadym Aug 7, 2023
9c13b9b
Merge branch 'fix-rm-temp' into refactor-bind-path-singularity
ivadym Aug 7, 2023
63cb8e1
Merge branch 'develop' into refactor-bind-path-singularity
ivadym Aug 7, 2023
598a66e
Merge branch 'refactor-bind-path-singularity' into fix-snakemake-rerun
ivadym Aug 7, 2023
48d6158
Merge develop
ivadym Aug 7, 2023
0697500
Merge branch 'fix-snakemake-rerun' into update-python
ivadym Aug 7, 2023
77e8fc8
Undo push to docker
ivadym Aug 7, 2023
51fa769
Merge branch 'develop' of github.com:Clinical-Genomics/BALSAMIC into …
ivadym Aug 7, 2023
77875b5
Remove Balsamic container
ivadym Aug 8, 2023
a1d9e24
Update conftest
ivadym Aug 8, 2023
4458276
CHANGELOG
ivadym Aug 8, 2023
7f4a00f
Kommatecken
ivadym Aug 8, 2023
9fadf04
Update .github/workflows/docker_build_push.yml
ivadym Aug 9, 2023
6f84080
Update .github/workflows/docker_build_push_release.yml
ivadym Aug 9, 2023
3399301
Review
ivadym Aug 9, 2023
fd98ed2
Review: update comment
ivadym Aug 9, 2023
859b7bd
Merge branch 'update-python' into remove-balsamic-container
ivadym Aug 9, 2023
d7dea91
Merge develop
ivadym Aug 10, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Remove unused pytests
ivadym committed Jul 25, 2023
commit 55ba76b942dee7a901e65b1d333c1663eb4c12fc
1 change: 0 additions & 1 deletion BALSAMIC/utils/cli.py
Original file line number Diff line number Diff line change
@@ -3,7 +3,6 @@
import re
import subprocess
import sys
import zlib
from distutils.spawn import find_executable
from io import StringIO
from pathlib import Path
14 changes: 10 additions & 4 deletions tests/models/test_snakemake_models.py
Original file line number Diff line number Diff line change
@@ -176,7 +176,10 @@ def test_get_snakemake_options_command(snakemake: Snakemake):


def test_get_snakemake_command(
snakemake: Snakemake, session_tmp_path: Path, reference_file: Path
snakemake: Snakemake,
case_id_tumor_only: str,
session_tmp_path: Path,
reference_file: Path,
):
"""Test retrieval of the snakemake command to be submitted to Slurm."""

@@ -192,7 +195,7 @@ def test_get_snakemake_command(
f"--snakefile {reference_file.as_posix()} "
f"--configfiles {reference_file.as_posix()} {reference_file.as_posix()} "
f"--use-singularity --singularity-args ' --cleanenv --bind {session_tmp_path.as_posix()}:/' --quiet "
f"--immediate-submit -j {MAX_JOBS} --jobname BALSAMIC.sample_tumor_only.{{rulename}}.{{jobid}}.sh "
f"--immediate-submit -j {MAX_JOBS} --jobname BALSAMIC.{case_id_tumor_only}.{{rulename}}.{{jobid}}.sh "
f"--cluster-config {reference_file.as_posix()} --cluster '{sys.executable} {SCHEDULER_PATH} "
f"--sample-config {reference_file.as_posix()} --profile {ClusterProfile.SLURM.value} "
f"--account {ClusterAccount.DEVELOPMENT.value} --qos {QOS.HIGH.value} --log-dir {session_tmp_path} "
@@ -214,7 +217,10 @@ def test_get_snakemake_config_options(snakemake: Snakemake):


def test_get_snakemake_cluster_options(
snakemake: Snakemake, session_tmp_path: Path, reference_file: Path
snakemake: Snakemake,
case_id_tumor_only: str,
session_tmp_path: Path,
reference_file: Path,
):
"""Test formatting of the snakemake cluster options."""

@@ -226,7 +232,7 @@ def test_get_snakemake_cluster_options(
# THEN the expected format should be returned
assert (
snakemake_cluster_options
== f"--immediate-submit -j {MAX_JOBS} --jobname BALSAMIC.sample_tumor_only.{{rulename}}.{{jobid}}.sh "
== f"--immediate-submit -j {MAX_JOBS} --jobname BALSAMIC.{case_id_tumor_only}.{{rulename}}.{{jobid}}.sh "
f"--cluster-config {reference_file.as_posix()} --cluster '{sys.executable} {SCHEDULER_PATH.as_posix()} "
f"--sample-config {reference_file.as_posix()} --profile {ClusterProfile.SLURM.value} "
f"--account {ClusterAccount.DEVELOPMENT.value} --qos {QOS.HIGH.value} --log-dir {session_tmp_path} "
63 changes: 0 additions & 63 deletions tests/utils/test_utils.py
Original file line number Diff line number Diff line change
@@ -15,7 +15,6 @@
from BALSAMIC.constants.cluster import ClusterConfigType
from BALSAMIC.constants.paths import CONTAINERS_DIR
from BALSAMIC.utils.cli import (
SnakeMake,
CaptureStdout,
get_snakefile,
createDir,
@@ -232,68 +231,6 @@ def test_get_file_extension_known_ext():
assert file_extension == actual_extension


def test_snakemake_local():
# GIVEN required params
snakemake_local = SnakeMake()
snakemake_local.working_dir = "this_path/snakemake"
snakemake_local.snakefile = "workflow/variantCalling_paired"
snakemake_local.configfile = "sample_config.json"
snakemake_local.run_mode = "local"
snakemake_local.use_singularity = True
snakemake_local.singularity_bind = ["path_1", "path_2"]
snakemake_local.forceall = True

# WHEN calling the build command
shell_command = snakemake_local.build_cmd()

# THEN it will contruct the snakemake command to run
assert isinstance(shell_command, str)
assert "workflow/variantCalling_paired" in shell_command
assert "sample_config.json" in shell_command
assert "this_path/snakemake" in shell_command
assert "--dryrun" in shell_command
assert "--forceall" in shell_command


def test_snakemake_slurm():
# GIVEN required params
snakemake_slurm = SnakeMake()
snakemake_slurm.case_name = "test_case"
snakemake_slurm.working_dir = "this_path/snakemake"
snakemake_slurm.snakefile = "worflow/variantCalling_paired"
snakemake_slurm.configfile = "sample_config.json"
snakemake_slurm.run_mode = "cluster"
snakemake_slurm.cluster_config = "cluster_config.json"
snakemake_slurm.scheduler = "sbatch.py"
snakemake_slurm.log_path = "logs/"
snakemake_slurm.script_path = "scripts/"
snakemake_slurm.result_path = "results/"
snakemake_slurm.qos = "normal"
snakemake_slurm.account = "development"
snakemake_slurm.profile = "slurm"
snakemake_slurm.mail_type = "FAIL"
snakemake_slurm.mail_user = "john.doe@example.com"
snakemake_slurm.sm_opt = ("containers",)
snakemake_slurm.quiet = True
snakemake_slurm.use_singularity = True
snakemake_slurm.singularity_bind = ["path_1", "path_2"]
snakemake_slurm.run_analysis = True

# WHEN calling the build command
shell_command = snakemake_slurm.build_cmd()

# THEN constructing snakecommand for slurm runner
assert isinstance(shell_command, str)
assert "worflow/variantCalling_paired" in shell_command
assert "sample_config.json" in shell_command
assert "this_path/snakemake" in shell_command
assert "--dryrun" not in shell_command
assert "sbatch.py" in shell_command
assert "test_case" in shell_command
assert "containers" in shell_command
assert "--quiet" in shell_command


def test_get_script_path():
# GIVEN list of scripts
custom_scripts = ["refseq_sql.awk"]