Parallelization is sometimes slow

[TimothyOlsson]

For very large data sets, each parallel_3 process is quite frankly, slow.

Adding placeholders and sorting the values is what makes the overhead calculations slow. A lot of work has already been done by Matthew, the creator of Quandenser to make it faster.

Changing the loop so the values are stored in .dat files, like what was done previously might solve this issue

[MatthewThe]

This sounds like it shouldn't be too hard, I will look into this sometime next week.

[TimothyOlsson]

This sounds like it shouldn't be too hard, I will look into this sometime next week.

Thank you! The work you have already done has been extremely helpful. The overhead calculations have been significantly improved after some testing and the "stability issues" I had with the pipeline (#15 types of crashes) are pretty much gone, from what I have seen.

Edit: addFeatureLinks could also possibly be a bottleneck

[MatthewThe]

I have added intermediate writing of features between alignments to your Quandenser branch:
statisticalbiotechnology/quandenser@df79378

This skips all the matching of the previous alignments. I've only tested it on the iPRG2016 set, where it resulted in a limited speed up, but I imagine it should give you a considerable speed-up on large sets. Please give it a try! :)

[TimothyOlsson]

I have added intermediate writing of features between alignments to your Quandenser branch:
statisticalbiotechnology/quandenser@df79378

This skips all the matching of the previous alignments. I've only tested it on the iPRG2016 set, where it resulted in a limited speed up, but I imagine it should give you a considerable speed-up on large sets. Please give it a try! :)

I have been testing it and it seems to really speed up the second step. I tested on an old data set on cyanobacteria (10 files), and got 45 minutes instead of 1h 21m. I am going to test it further, but results so far seems very promising. This will be very useful, thank you!