Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add na.rm to aggregating functions #34

Open
wants to merge 7 commits into
base: master
Choose a base branch
from

Conversation

TuomasBorman
Copy link

This PR adds na.rm option to following functions: sumCountsAcrossFeatures, numDetectedAcrossFeatures, summarizeAssayByGroup, numDetectedAcrossCells.

This PR is related to this issue: #33

@LTLA
Copy link
Owner

LTLA commented Nov 12, 2024

Thanks for the PR. I'll address some immediate points first, but having slept on your problem, I would recommend a much simpler solution.

  • value == NA_INTEGER is not correct if the matrix was double-precision, as the NA_INTEGER value (-2^31) is a valid double-precision number. You'll need to check the input matrix type and only perform that check if the input matrix is integer or logical.
  • Conversely, std::isnan() is unnecessary when the input matrix is integer or logical, as IEEE specials wouldn't fit inside a 32-bit integer anyway.
  • All of the new checks are unnecessary when na.rm=FALSE. To avoid having a potential optimization-breaking if statement in the tight inner while loop, hoist it out and duplicate the for (auto s : runs) loop for each NA'ness status (na.rm=TRUE with double, na.rm=TRUE with int, na.rm=FALSE) with only the relevant NA checks in each one.
  • You have too many comments - don't touch the lines that you don't need to.
  • No pipes in my packages.

You can see that it is quite tedious to protect against NAs while preserving performance for the vast majority of (single-cell) use cases that don't need it. So perhaps you might consider a much simpler approach:

y <- rsparsematrix(10000, 5, density=0.1)
z <- sample(LETTERS[1:5], nrow(y), replace=TRUE)
y[1] <- NA

y <- DelayedArray(y)
is.okay <- !is.na(y)
type(is.okay) <- "double"
DelayedArray::rowsum(y, z, na.rm=TRUE) / DelayedArray::rowsum(is.okay, z)

4 lines and done. No copying involved as the DelayedArray() defers the is.na and type coercion operations. Maybe a bit slower than the C++ code but I doubt it matters much.

@TuomasBorman
Copy link
Author

Thank you for the feedback! I'm not very experienced with C++, so I jumped straight into the deep end. Your suggestions seems to be working and it is much simpler. I would still tend to think that scuttle is the right place to fix this na.rm issue. Would it be ok, if I

  1. revert changes affecting the C++ code
  2. add na.rm to .sum_across_features: if na.rm, then run your 4 lines instead of C++ code

Moreover, summarizeAssayByGroup used loops (lapply) to aggregate data. I guess this approach could also improve the efficiency of that function.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants