Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improvement for pileup memory usage #188

Merged
merged 4 commits into from
Jan 23, 2020
Merged

Conversation

athos
Copy link
Member

@athos athos commented Jan 22, 2020

cljam.algo.pileup tends to consume a very large amount of memory and in the worst case it ends up with OOME.

This PR mitigates the situation by improving memory efficiency for the pileup data structure. The primary changes are as follows:

  • Use more space-efficient data structures (eg. primitive arrays, dedicated deftypes) to represent a pileup result
  • Drop unnecessary intermediate data (:seqs-at-ref / :quals-at-ref ) from the pileup result
  • Unchunk a pileup result (which was a chunked sequence) and return a lazy sequence instead to make each element easier to consume and deallocate one by one

In my small examination, the changes reduced the memory usage by ~45% at peak time with almost no impact on time efficiency:

Memory usage (before change) Memory usage (after change)

See here for the detailed raw profiling data for the above result.

@athos athos requested review from alumi, yito88 and a team January 22, 2020 09:05
@ghost ghost removed their request for review January 22, 2020 09:05
@codecov
Copy link

codecov bot commented Jan 22, 2020

Codecov Report

Merging #188 into master will increase coverage by <.01%.
The diff coverage is 96%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #188      +/-   ##
==========================================
+ Coverage   86.67%   86.68%   +<.01%     
==========================================
  Files          76       76              
  Lines        6051     6055       +4     
  Branches      501      501              
==========================================
+ Hits         5245     5249       +4     
  Misses        305      305              
  Partials      501      501
Impacted Files Coverage Δ
src/cljam/algo/pileup.clj 96.66% <96%> (+0.06%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e98b889...ea0d300. Read the comment docs.

@athos athos force-pushed the spike/space-efficient-pileup branch from 6e2f54d to ccc4eb0 Compare January 23, 2020 01:54
@athos
Copy link
Member Author

athos commented Jan 23, 2020

Rebased the patch onto the master branch.

Copy link
Member

@alumi alumi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the investigation and PR! Added a suggestion comment.

src/cljam/algo/pileup.clj Outdated Show resolved Hide resolved
Co-Authored-By: alumi <alumi@users.noreply.github.com>
Copy link
Member

@alumi alumi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍

Copy link
Contributor

@yito88 yito88 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for improving that!
LGTM 👍

@alumi alumi merged commit 42835b9 into master Jan 23, 2020
@alumi alumi deleted the spike/space-efficient-pileup branch January 23, 2020 03:22
@athos
Copy link
Member Author

athos commented Jan 23, 2020

Thanks for the review! 🙌

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants