Freezeman

.gitignore

@@ -1,3 +1,6 @@
+*.swp
+*.swo
+*.swn
+
 .nextflow*
 work
-outputs

.gitmodules

@@ -0,0 +1,4 @@
+[submodule "genpipes"]
+	path = genpipes
+	url = https://bitbucket.org/mugqic/genpipes.git
+	branch = run_processing

README.md

@@ -0,0 +1,161 @@
+Run Processing Monitor
+======================
+
+This nextflow workflow manages the monitoring and automates launch of run
+processing jobs following the completion of a run provided by the sequencing
+laboratory at McGill Genome Center. Launches rely on Freezeman for their inputs
+and outputs. There is also a some legacy code that managed the same but from
+runs that were managed through Clarity (Illumina's proprietary Freezeman
+equivalent).
+
+Install
+-------
+
+Make sure to clone the repository along with its submodule.
+
+```
+git clone --recurse-submodules git@github.com:c3g/monitor.git
+
+or
+
+git clone --recurse-submodules https://github.com/c3g/monitor.git
+```
+
+Usage
+-----
+
+The monitor is dependant on the local McGill University cluster, Abacus. Make
+sure to load the required modules before launching it.
+
+```
+module purge && module load mugqic/java/openjdk-jdk-17.0.1 mugqic/nextflow/22.10.6 mugqic/python/3.10.2
+```
+
+The simplest usage of the monitor is to run `main.nf` along with an entrypoint.
+To reduce the ressources usage, limiting the entrypoint to Freezeman monitoring
+is recommended, particularly knowing that Clarity support is deprecated.
+
+Profiles are used to set configurations, they change behaviors such as emailing
+notifications, filepath globs to watch for sequencing files, remote access
+details and more.
+
+The production profile should not be changed lightly and is only meant to be
+run by the freezeman-lims user on Abacus.
+
+```
+nextflow run main.nf -profile [production,debug,dev] -entry [Freezeman][Monitor,Launch,MonitorAndLaunch]   
+```
+
+Redirecting the logs is recommended.
+
+```
+nextflow -log [filepath].log run main.nf -profile [production,debug,dev] -entry [Freezeman][Monitor,Launch,MonitorAndLaunch]
+```
+
+Remember to keep the nextflow logs tidy with
+
+```
+nextflow log
+
+and 
+
+nexflow clean
+```
+
+Notes
+-----
+
+### Launch delays
+
+The Launch part of the monitor is particularly slow to be ready to receive
+NovaSeq runs runinfofile. Even with the fzmn-child-process child config file
+that reduce the glob pattern to check for `RTAComplete.txt` files, the launch
+for Illumina takes at least 25 mins.
+
+*The current version in production requires ~10 mins to be ready with MGI files
+and a good 8 hours for Illumina files.*
+
+Not only is the repo relying on Abacus' system and filesystem, the trigger that
+launches the run processing relies on files dropped in the
+`freezeman-lims-run-info` folder. In the current case, that is performed by a
+5min cron job under freezeman-[lims,qc,dev] users that drops them in a folder
+under freezeman-[lims,qc,dev] access.
+
+I put some efforts on the bloating of the monitor and I will not put anymore.
+Removing useless steps had an underwhelming impact on the overall monitor both
+on time to run and memory usage. Reduced from ~40GB RAM to ~30GB in production
+and similar impact on the launch of nextflow until all channels are up to
+monitor. The only impactful changes would be to reduce the number of open
+channels or limit the size of glob patterns and wildcards to match fewer
+filepaths in the filesystem. Doing so would require to revisit the way
+runinfofiles, RTAcomplete.txt and checkpoint files are monitored at the
+filesystem level by the watchPatch channels.
+
+**Most of the time lost is spent managing the database** which is used to match
+runs to their metadata. Their are columns that are used to get filepaths,
+timestamps and such, but unfortunately **the entire run info json file is store
+in a cell for each run**. Parsing these is the time consuming step. Therefore
+the minimal launch time can be obtained with the smallest database, I recommend
+to move/remove the previous database before launch. Be weary that the monitor
+won't monitor runs that were process prior to this move/remove.
+
+Typical "short" launch:
+
+```
+module load mugqic/java/openjdk-jdk-17.0.1 mugqic/nextflow/22.10.6 mugqic/python/3.10.2 && mv /nb/Research/freezeman-processing/monitor.db /nb/Research/freezeman-processing/db.bu/monitor.db.240704 && nextflow -log log/log run main.nf -profile production -entry FreezemanMonitorAndLaunch -bg
+```
+
+### Particular set-up
+
+The monitor is highly dependant on a number of softwares, env variables, ssh
+keys, paths and network access that are only avaible on Abacus. It was not
+designed to be run outside of the freezeman-[lims,qc,dev] users environment and
+relies on some of their specific set-up. Notable examples of required set-up:
+
+1. The runinfofiles are dumped by the Freezeman interface when an experiment is
+   "Launched" to be processed and reingested to be added as a "Dataset" for run
+   validation. They are copied via an rsync 5min cron job that relies on a
+   unique ssh-key for each freezeman-[lims,qc,dev] to access the virtual
+   machine under the "intermediary" user. A different user will have to move or
+   copy these files themselves inside the directories set in `nextflow.config`
+   under `neweventpath` & `newruninfopath`
+
+2. Genpipes run_processing.py is using some software that are part of
+   $MUGQIC_INSTALL_HOME_PRIVATE, such as bcl2fastq. To access these, one must
+   set their environment using:
+
+   ```
+   export MUGQIC_INSTALL_HOME_PRIVATE=/lb/project/mugqic/analyste_private
+   module use $MUGQIC_INSTALL_HOME_PRIVATE/modulefiles
+   ```
+
+   User freezeman-lims have $MUGQIC_INSTALL_HOME_PRIVATE set-up in its
+   `~.bash_profile`.
+
+3. The user running the monitor should also have access to the run_processing
+   directories: `/nb/Research/<platform>/<run_folder>`
+
+4. Review the content of nextflow.config before launch. Paths listed in the
+   configs should exist since the monitor will try to read from them in the
+   early steps. There is also a final copy of the run_processing output files
+   that is targeting a directory listed in the config parameter `custom_ini`
+   files that should target one of the provided `.ini` files in the `assets`
+   folder. Make sure that this folder exists before launching. In the `.ini`,
+   search for
+
+   ```
+   [copy]
+   destination_folder=/lb/project/mugqic/projects/[...]
+   ```
+
+### What is dependent on the monitor
+
+Even though this monitor needs to be able to take care of Illumina & MGI
+run_processing, at the moment, most run_processing is managed by Haig
+Djambazian's pipeline which I believe is a set of bash scripts. However, it
+seems that some of his processing relies on this monitor for complementary
+steps, notably MultiQC and reporting emails. They are part of WatchCheckpoints
+workflow in monitor.nf that manages the interface between Haig's stuff and this
+monitor. It used to run from the main branch of the repo, under the bravolims
+user, but it hasn't been restarted since the last Abacus outages (2023/07/08 &
+2023/08/05).

assets/debug.ini

@@ -0,0 +1,8 @@
+[DEFAULT]
+cluster_other_arg = -W umask=0002 -l qos=f2
+
+[basecall]
+cluster_other_arg=-m a -M $JOB_MAIL -W umask=0002 -l nodes=1:gpus=1
+
+[fastq_t7]
+tmp_dir=/lb/scratch/$USER/tmp.$PBS_JOBID

assets/dev.ini

@@ -0,0 +1,8 @@
+[DEFAULT]
+cluster_other_arg = -W umask=0002 -l qos=normal
+
+[basecall]
+cluster_other_arg=-m a -M $JOB_MAIL -W umask=0002 -l nodes=1:gpus=1
+
+[fastq_t7]
+tmp_dir=/lb/scratch/$USER/tmp.$PBS_JOBID

assets/email_MGI_run_finish.groovy → assets/email_run_finish.groovy

@@ -35,21 +35,40 @@ yieldUnescaped '<!DOCTYPE html>'
 html(lang:'en') {
     head {
         meta('http-equiv':'"Content-Type" content="text/html; charset=utf-8"')
-        title("MGI Run finished: ${run.run}")
+        title("${platform} Run finished: ${run.run}")
     }
     body {
         div(style:"font-family: Helvetica, Arial, sans-serif; padding: 30px; max-width: 900px; margin: 0 auto;") {
-            h3 "Run: ${run.run} (${run.flowcell})"
+            h3 "Run: ${event.data.run_name} (${run.flowcell})"
+            h3 "Folder: [${run.analysis_dir}]"
             p {
                 span "Run processing finished. Full report attached to this email, but also available "
-                a ( href:"https://datahub-297-p25.p.genap.ca/MGI_validation/2023/${run.run}.report.html", "on GenAP" )
+                a ( href:"https://datahub-297-p25.p.genap.ca/Freezeman_validation/${event.year}/${event.data.run_name}.report.html", "on GenAP" )
                 span "."
             }
             ul {
                 li "Spread: ${run.headerInfo.Spreads}"
                 li "Yield assigned to samples: ${nfGeneral.format(yield)} bp"
                 li "Instrument: ${run.headerInfo.Instrument}"
             }
+            table(style:"box-shadow: 0 0 30px rgba(0, 0, 0, 0.05);margin: 25px 0;font-size: 0.9em;border-collapse: collapse;") {
+                tbody {
+                    tr(style:"background-color: #009879;color: #ffffff;text-align: left;") {
+                        th(style:'padding: 5px 10px', "Project Name")
+                        th(style:'padding: 5px 10px', "Samples")
+                    }
+                    samples.countBy{it.ProjectName ? it.ProjectName : it.project_name}.each { projectname, count -> 
+                        tr(style:'border: 2px solid #dddddd;') {
+                            td style:"padding: 5px 10px; font-weight: bold", projectname
+                            td style:"padding: 5px 10px;", count == 1 ? "${count} sample" : "${count} samples"
+                        }
+                    }
+                    tr(style:'background-color: #666666;color: #ffffff;text-align:left;') {
+                        td "Total"
+                        td  samples.size() == 1 ? "${samples.size()} sample" : "${samples.size()} samples"
+                    }
+                }
+            }
             table(style:"box-shadow: 0 0 30px rgba(0, 0, 0, 0.05);margin: 25px 0;font-size: 0.9em;border-collapse: collapse;") {
                 thead {
                     tr(style:"background-color: #009879;color: #ffffff;text-align: left;") {
@@ -83,6 +102,11 @@ html(lang:'en') {
                 span workflow.commitId ? "Email generated at ${dateFormat(now)} using monitor at commit ${workflow.commitId}." : "Email generated at ${dateFormat(now)}."
             }
             p(style:"color: #999999; font-size: 12px", "C3G Run Processing.")
+            // p {
+            //     span(class:"apple-link", style:"color: #999999; font-size: 12px; text-align: center;") {
+            //         a(href:"https://c3g.ca/", style:"text-decoration: none; color: #999999; font-size: 12px; text-align: center;", "C3G Run Processing")
+            //     }
+            // }
         }
     }
 }

assets/email_run_start.groovy

@@ -0,0 +1,126 @@
+// == Introduction == //
+// This template expects three variables to be defined - `flowcell`, `samples` and `platofrm`.
+// The template is instantiated (in the `EmailAlertStart` process) like so:
+// email_fields = [flowcell: eventfile.flowcell, samples: rows, platform: platform]
+// def html_template = engine.createTemplate(html).make(email_fields)
+
+// == Testing == //
+// To test modifications to this template, there is the `Debug` workflow, which looks for changes to
+// this file and upon detecting a change, regenerates the template using an example multiqc json file.
+// The demo HTML is written to outputs/testing/email/email_MGI_run_finish.html. I'd recommend setting up
+// a live watcher so that you can just save the template and the page gets updated in-browser instantly.
+
+// == Helpers == //
+// Here we define a couple of helpful variables/function to make the rest of the
+// template a little cleaner.
+Date now = new Date()
+
+def dateFormat(date) {
+    date.format('yyyy-MM-dd HH:mm:ss z')
+}
+
+// == HTML Template == //
+// The actual temlate code.
+yieldUnescaped '<!DOCTYPE html>'
+html(lang:'en') {
+    head {
+        meta('http-equiv':'"Content-Type" content="text/html; charset=utf-8"')
+        title("${platform} Run started - Flowcell ${flowcell}")
+    }
+    body {
+        div(style:"font-family: Helvetica, Arial, sans-serif; padding: 30px; max-width: 900px; margin: 0 auto;") {
+            h2 "New run processing started : ${flowcell}"
+            p {
+                span "This is an automated message sent from the run processing event monitor."
+            }
+            p {
+                span "Run processing has started for new ${platform} run on flowcell: ${flowcell}. "
+            }
+            p {
+                span "Freezeman Run Info file used is attached."
+            }
+            table(style:"box-shadow: 0 0 30px rgba(0, 0, 0, 0.05);margin: 25px 0;font-size: 0.9em;border-collapse: collapse;") {
+                tbody {
+                    tr(style:"background-color: #009879;color: #ffffff;text-align: left;") {
+                        th(style:'padding: 5px 10px', "Project Name")
+                        th(style:'padding: 5px 10px', "Samples")
+                    }
+                    samples.countBy{it.ProjectName ? it.ProjectName : it.project_name}.each { projectname, count -> 
+                        tr(style:'border: 2px solid #dddddd;') {
+                            td style:"padding: 5px 10px; font-weight: bold", projectname
+                            td style:"padding: 5px 10px;", count == 1 ? "${count} sample" : "${count} samples"
+                        }
+                    }
+                    tr(style:'background-color: #666666;color: #ffffff;text-align:left;') {
+                        td "Total"
+                        td  samples.size() == 1 ? "${samples.size()} sample" : "${samples.size()} samples"
+                    }
+                }
+            }
+            p(style:"color: #999999; font-size: 12px") {
+                span workflow.commitId ? "Email generated at ${dateFormat(now)} using monitor at commit ${workflow.commitId}." : "Email generated at ${dateFormat(now)}."
+            }
+            p {
+                span(class:"apple-link", style:"color: #999999; font-size: 12px; text-align: center;") {
+                    a(href:"https://c3g.ca/", style:"text-decoration: none; color: #999999; font-size: 12px; text-align: center;", "C3G Run Processing")
+                }
+            }
+        }
+    }
+}
+
+
+// <html>
+
+//  <head>
+//      <meta charset="utf-8">
+//      <meta http-equiv="X-UA-Compatible" content="IE=edge">
+//      <meta name="viewport" content="width=device-width, initial-scale=1">
+
+//      <meta name="description" content="New run notification.">
+//      <title>T7 Flowcell ${flowcell}</title>
+//  </head>
+
+//  <body>
+//      <div style="font-family: Helvetica, Arial, sans-serif; padding: 30px; max-width: 800px; margin: 0 auto;">
+
+//          <h2>New run processing started</h2>
+//          <h3>Flowcell: <span style='font-weight: bold'>$flowcell</span></h3>
+
+//          <table
+//              style="width:100%; max-width:100%; border-spacing: 0; border-collapse: collapse; border:0; margin-bottom: 30px;">
+//              <tbody style="border-bottom: 1px solid #ddd;">
+//                  <tr>
+//                      <th
+//                          style='text-align:left; padding: 8px 0; line-height: 1.42857143; vertical-align: top; border-top: 1px solid #ddd; border-bottom: 1px solid #ddd;'>
+//                          Project Name</th>
+//                      <th
+//                          style='text-align:left; padding: 8px; line-height: 1.42857143; vertical-align: top; border-top: 1px solid #ddd; border-bottom: 1px solid #ddd;'>
+//                          Samples</th>
+//                  </tr>
+//                  <% samples.countBy{it.ProjectName}.each { projectname, count -> %>
+//                      <tr style='line-height: 1.42857143; vertical-align: top; '>
+//                          <td>$projectname</td>
+//                          <td>$count ${count == 1 ? "sample" : "samples"}</td>
+//                      </tr>
+//                      <% } %>
+//                          <tr
+//                              style='text-align:left; padding: 8px; line-height: 1.42857143; vertical-align: top; border-top: 1px solid #ddd; border-bottom: 1px solid #ddd;'>
+//                              <td>Total</td>
+//                              <td>${samples.size()} ${samples.size() == 1 ? "sample" : "samples"}</td>
+//                          </tr>
+//              </tbody>
+//          </table>
+
+//          <p>
+//              <span class="apple-link" style="color: #999999; font-size: 12px; text-align: center;"><a
+//                      href="https://c3g.ca/"
+//                      style="text-decoration: none; color: #999999; font-size: 12px; text-align: center;">C3G
+//                      Run Processing</a>
+//          </p>
+
+//      </div>
+
+//  </body>
+
+//  </html>