Quandenser-pipeline v0.081

Changes in the image:

Major changes:

• Updated the boxcar merge method. It is now both faster (about 100s per file) and more precise, using a new "momentum" method. Currently testing whether the merge method gives better results.

Minor changes:

• Fixed a bug which prevented copy and paste with the clipboard (ctrl + c and ctrl + v). This fix allows you to copy to and from excel or other programs and also import old file_lists

• Updated the "auto labeler" with new method. It now works with regex instead of labeling everything from 1 and up. Right click on "label" header in first tab to open. Example what it can do is found here. This should make it way faster to label large data sets.

• Sometimes, quandenser_parallel_1 and msconvert crashes, which collapses the whole pipeline. I added an "error strategy", so the pipeline retries the process once if it crashes.

Quandenser-pipeline v0.08-beta

Changes in the shell script:

Major changes:

• Due to a new update on singularity hub, the API get requests are limited, meaning the it will not be possible to determine version number by using the Singularity hub API, thus preventing updates from the shellscript. Therefore, the new .sh file will use github's api to fetch the version number instead. This means that both the tag name of each release in github and the version number in the image needs to be updated on each release while before, only the version number in the image was needed to update. THIS MEANS THAT USERS BEFORE 0.08 NEEDS TO UPDATE TO THE NEWEST VERSION OF THE SHELL SCRIPT, SINCE THE OLD VERSION WILL NOT AUTOMATICALLY FIND UPDATES. Please download the newest version of the shell script to fix this issue.

Changes in building the image:

Minor changes:

• New version of nextflow requires the user to run it in the background:
  This breaks several things which will require some rework. With the new nextflow version, run_quandenser.sh will suspend all processes if the tag "-bf" is not added. Adding the "background" tag will break the ability to kill processes in the GUI, since the .sh script will no longer have nextflow as its child-process. Some functions used in the pipeline will be depreciated in future nextflow versions, which might cause further problems. Thus, the update for nextflow 19.01 --> 19.07 causes a lot of issues, which might not be worth fixing at the moment. Nextflow version 19.01 will be the default, static version for now.

• Added some permission for the new quandenser version, so users can run it without any problems.

Changes in the image:

Major changes:

• A new version of quandenser has been installed (v0.02) which includes updated versions of submodules and huge improvements to speed of parallelization and the stability of the processing.

• A new triqler version has been added (v0.03, which has several improvements, such as optimizations for very large datasets.

Minor changes:

• Removed the ability to choose max missing value directly from the GUI. The reason is that few users know what the the option does and that it should be changed depending on the amount of files available. The default value in quandenser takes this into account, but is overwritten by the GUI value. The value can still be changed by adding "--max-missing" into quandenser's optional parameters. This should prevent confusion of new users, but still allow users to change the values to their liking.

• Removed parallel step 5, by merging 4 and 5. The last step is not necessary and only adds to the total calculation time.

• Fixed some bugs in the nextflow pipeline when running quandenser v0.02

• Fixed some bugs where elements of the GUI are hidden if the window size is too small.

Quandenser-pipeline v0.071-beta

Changes in building the image:

• Added psutil to python 3.6 and gcc

Changes in the image:

Minor changes:

• Fixed a bug where the kill button would in some rare cases kill the GUI window. Using psutil, everything should be killed properly, without touching the GUI

• Removed redundant crash checks when starting the GUI

• The pipeline now prints out all parameters during start, for better debugging process.

• Added so the GUI checks more thoroughly if a process has crashed or not

• Fixed publish path issue with boxcar convert

• The last directory when choosing files will now be remembered between sessions

• Added so the GUI will only check runs without a label (aka COMPLETED, FAILED... etc). This will make the GUI more responsive during updates

Quandenser-pipeline v0.07-beta

Changes in shell script

• Added tqdm in pip install

• Added the new boxcar conversion script

Changes in the image:

Major changes:

• Added so running jobs are checked in a different thread (no freezing) and only checks non completed jobs (which will make the GUI more responsive)

• Added Boxcar scan conversion, so boxcar scans can be used in the pipeline (can be enabled in tab 2).

Minor changes:

• Fixed a bug where the kill button did not kill all processes, creating "zombie" processes.

• The file list will now be added to the publish directory

• Added stdout capturing to triqler process

• Changed some defaults in nf.config

Quandenser-pipeline v0.061-beta (hotfix)

Changes in shell script

• You can now read readme of new updates when updating from the shell script

• Quick edit: It now also checks if SingularityHub is not reachable/the image is updating on singularity hub

Changes in the image:

Minor changes:

• Missing conditional statement makes the "regular" quandenser processing crash. Sorry about that.

Quandenser-pipeline v0.06-beta

Changes in shell script

• Added so the shell script checks for newer versions of the pipeline. This will overwrite the older image and download the newest version of the shell script. This should be compatible with all releases in the future

• Improved help description

• Added license of Proteowizard

• Added short commands to every option

Changes in building the image:

• Added "zip" and "tkinter for python3.6" in the singularity image. Zip will allow for triqler sending files via email and tkinter will prevent python leakages to the host.

Changes in the image:

Major changes:

• You can now send the triqler files with email when the pipeline is done! Just set "send files" in the advanced tab to "true".

• Changed so msconvert can output any format, not just mzML

• Changed so python uses only the image modules and will no longer "leak"
  into the host computer, i.e use the hosts python modules will not be loaded if there both the host and the image has the same module installed.

• Added option to ignore tree parallelization, so you only parallelize only the first part. Disabling the tree parallelization should prevent bugs related to issue #15 . Quandenser has also been slightly modified to make this work

Minor changes:

• Improved defaults when running on cluster. Also condensed the in the workflow, putting the cpu spinboxes besides the time boxes

• Changed back ' to " in nf.config, since mixing the quotes is not compatible with some commands. The user can only use ' in the additional commands

• Added some additional commands in the config parser and fixed some minor bugs.

• Removed a crash warning if the directory is not created within 2 seconds. Sometimes, it is just slow. If that is the case, it will still say quandenser started

• Added experimental raw intensities and distributions output. These are a work in progress, and should not really be used, but they are available.

• Fixed limits for spinboxes related to issue #11

• Improved how tooltips are loaded. This will not affect the users, but it will allow me to much easier document and add tooltips to the GUI :)

• Improved About tab in the GUI. A neat feature about the About tab is that it is a mini browser, so you can now click the links inside and read about the software used! (Requires open-gl enabled)

• The workflow has been improved in the edit workflow tab. It is now much clearer how the workflow works. (Requires open-gl enabled)

Edit: Updated binary slightly, spotted parenthesis was wrong

Quandenser-pipeline v0.05-beta

Changes in building the image:

The quandenser version is prepackaged in the "dependencies" directory,
which uses the modified quandenser on this branch:
https://github.com/statisticalbiotechnology/quandenser/tree/quandenser-pipeline

The triqler version is now using another branch and not the pip version.
The modified version includes outputting raw intensities, and will be
used to add additional post processing steps
https://github.com/statisticalbiotechnology/triqler/tree/quandenser-pipeline

Changes in the image:

Major changes:

Increased parallelization functionality by saving and loading
parameters. For a data set of about 80 files, this gives 7.5 hours
faster calculation time (!) There are still many more areas where you
can speed up the processing, but for now it is enough. This MIGHT yield
strange bugs when running, which I am working on to check if that is true.

Fixed bug where path to converted files were wrong,
which made RAW files not able to process due to the wrong paths.
(Sorry about that)

Fixed bug where maracluster freezes if you run large cohorts,
which is now included in the new version of quandenser

Minor changes:

Auto-labeling is now using numbers instead of characters
(related to issue)
#4

Improved check for crashes during start of run. It now waits 2 seconds
to check if nextflow managed to run or not.

Increased limit for spinboxes, so you are now not limited to 99

Changed "" to '' in nf.config so you can use " in additional arguments

Added basic copy and paste on the file table
(This is not saved to the clipboard at the moment, since Qclipboard
doesnt seem to work. The copy and paste are used internally). This will
be improved in the future

Pressing enter at the end of the file choosing table now creates a new row

Fixed minor bugs with empty rows (they should not be happening anymore)

Added right-click on the "file" column to remove empty rows

The new quandenser file packaged comes from a different branch of
quandenser, to minimize the work to repack the file each time

Added option to output raw intensities from triqler (WIP)

Added defaults slightly to more accurately be compatible with HPC
running time

Quandenser-pipeline v0.04-beta. Stable

Quandenser-pipeline v0.04-beta

Changelog:

Added MSconvert to the pipeline

Added option to parallelize msconvert and quandenser, with limits on how
many processes you can do simultaneously

Added a modified quandenser, which can handle parallelization

Improved the channels to prevent name collisions

Config location changed from /var/tmp/quandenser_pipeline_$USER to
/home/$USER/.quandenser_pipeline The reason why is that in clusters,
there are usually several login nodes. Nextflow is started from
one of the login nodes and will use that node's environment. When
logging in to say rackham, there are 3 possible login nodes that you
might be logged in at. Each node has it's own /var/tmp folder, meaning
if you logged in at rackham1 and then to rackham2, the settings and
everything like that would be different. However, the home directory
is shared between nodes which makes any files in home persist.
Another problem with login nodes are that if you login to rackham1 and
run a script there, it will not be accessible to rackham2. This prevents
quandenser-pipeline from interacting with that process.

Added tooltips to some options

Fixed problem with links with msconvert

Added option to resume from a previous directory (aka all previous files
are loaded in when running the next run). This will be useful if the
workflow crashes or you run out of calculation time on the cluster, so
you don't lose all your data.

Added so you can get an email when the pipeline has finished. Only works
with gmail accounts (this includes scilifelab emails). If you have 2
factor authorization, go to:
https://security.google.com/settings/security/apppasswords
to generate a token, to use instead of a password.
Note: Since all files you create are only readable by you (chmod 700),
your password is secure in the nf.config file

Added "labels" to the output folder. The purpose is that if you want to
run many analyzes in the same folder, you add a label (ex "ecoli_gut"),
quandenser-pipeline will create a new directory for you (ecoli_gut_1)
that will prevent outputs from overwriting one another. All runs will
use the same work directory, without collisions between them.

Added right-click on label field on tab1 to automatically assign an
unique label for each file that is unlabeled

Improved robustness with the pipe script, which prevents crashes on
clusters. Note that some issues are still prevalent, please go to Readme
in the github page for known issues.

Fixed some typos that crashed the nextflow pipeline in some cases

Fixed a lot of weird bugs in the UI. All parameters should now write
properly to their assigned location

Improved defaults for slurm cluster computation