updates sv manager and cluster creation scripts to utilize dataproc #3579
Conversation
Codecov Report
@@ Coverage Diff @@
## master #3579 +/- ##
==============================================
+ Coverage 79.937% 79.94% +0.003%
Complexity 18005 18005
==============================================
Files 1206 1206
Lines 65578 65578
Branches 10245 10245
==============================================
+ Hits 52421 52423 +2
+ Misses 9061 9059 -2
Partials 4096 4096
|
There was a problem hiding this comment.
It looks good to me. I haven't personally run tests but it looks so simple that I can't imagine it doesn't work. I might be good to also include a value for ‑‑max‑age since as is the cluster isn't good for running on more than one bam. I'd have the slight worry that some rogue process would be running so the cluster would never be idle. max-age could be set to a conservative value like 4 hours.
To be clear, the current changes are an improvement, so this is more a suggestion for a follow-up than a request for changes to the current pull request.
scripts/sv/manage_sv_pipeline.sh
Outdated
| @@ -206,8 +213,10 @@ while true; do | |||
| INIT_ARGS="${INIT_SCRIPT} gs://${GCS_SAVE_PATH}/init/" | |||
| fi | |||
|
|
|||
| echo "create_cluster.sh ${GATK_DIR} ${PROJECT_NAME} ${CLUSTER_NAME} ${GCS_REFERENCE_DIR} ${GCS_BAM_DIR} ${INIT_ARGS} 2>&1 | tee -a ${LOCAL_LOG_FILE}" | tee -a ${LOCAL_LOG_FILE} | |||
| create_cluster.sh ${GATK_DIR} ${PROJECT_NAME} ${CLUSTER_NAME} ${GCS_REFERENCE_DIR} ${GCS_BAM_DIR} ${INIT_ARGS} 2>&1 | tee -a ${LOCAL_LOG_FILE} | |||
| MAX_IDLE_MINUTES=${MAX_IDLE_MINUTES:-60m} | |||
There was a problem hiding this comment.
It might be good to move this to near the top of the file and then add in a new override field (e.g. -t|--timeout) with documentation.
…luster timed self-termination feature
SHuang-Broad
force-pushed
the
sh_dataproc_timeout
branch
from
f061d17
to
0abd064
Compare
|
Thanks @TedBrookings ! I did a creation and self-termination test run (without actually running the analysis to save time), and it worked as expected. Also implemented the suggested changes:
|
cluster timed self-termination (beta) feature.
Fixes #3574 considering that several people are suggesting beta from gcloud is safe enough.