Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix!: statistical functions should return null when provided a vector of only null values #5606

Merged
merged 19 commits into from
Jun 19, 2024

Conversation

lbooker42
Copy link
Contributor

Updated Numeric functions and tests:

  • avg
  • absAvg
  • var
  • wvar
  • std
  • wstd
  • ste
  • wste
  • tstat
  • wtstat
  • median - had incorrect behavior when null values present
  • percentile - had incorrect behavior when null values present
  • cov
  • cor

Updated UpdateBy aggregations and tests:

  • RollingAvg
  • RollingStd

Updated AggBy aggregations:

  • avg
  • var
  • std
  • ReVar
  • ReAvg

Will close #5564

@lbooker42 lbooker42 added this to the June 2024 milestone Jun 12, 2024
@lbooker42 lbooker42 self-assigned this Jun 12, 2024
engine/function/src/templates/Numeric.ftl Outdated Show resolved Hide resolved
engine/function/src/templates/Numeric.ftl Outdated Show resolved Hide resolved
return 0.5 * (copy[n / 2 - 1] + copy[n / 2]);
else return copy[n / 2];
// NULL values sorted to beginning of the array.
${pt.primitive}[] copy = sort(values.toArray());
Copy link
Member

@rcaudy rcaudy Jun 14, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not ideal. The implementation of sort is sub-optimal:

  1. We make one toArray() call internally.
  2. Then we make an array of boxed values.
  3. Then we sort using a comparator.
  4. Then we make a third copy wherein we unbox the values.

Sorting on the vector at least let's us skip one of the copies.

We should write a separate ticket for updating the sort kernels in io.deephaven.engine.table.impl.sort to have a version that does not bother with a valuesToPermute chunk, and use that version for the implementation of sort. This way we make exactly one copyToArray() and we're done. Note copyToArray(), not toArray(): we mustn't sort our backing array.

engine/function/src/templates/Numeric.ftl Outdated Show resolved Hide resolved
@chipkent chipkent added the devrel-watch DevRel team is watching label Jun 17, 2024
Copy link
Member

@chipkent chipkent left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I gave this a look, and the stuff that jumped out at me is the stuff @rcaudy already commented on.

engine/function/src/templates/Numeric.ftl Outdated Show resolved Hide resolved
engine/function/src/templates/Numeric.ftl Outdated Show resolved Hide resolved
engine/function/src/templates/Numeric.ftl Outdated Show resolved Hide resolved
engine/function/src/templates/Numeric.ftl Outdated Show resolved Hide resolved
Copy link
Member

@chipkent chipkent left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes look good. We just need a resolution on the 2 array copies.

engine/function/src/templates/Numeric.ftl Show resolved Hide resolved
engine/function/src/templates/Numeric.ftl Outdated Show resolved Hide resolved
engine/function/src/templates/Numeric.ftl Outdated Show resolved Hide resolved
engine/function/src/templates/Numeric.ftl Outdated Show resolved Hide resolved
engine/function/src/templates/Numeric.ftl Outdated Show resolved Hide resolved
engine/function/src/templates/Numeric.ftl Outdated Show resolved Hide resolved
engine/function/src/templates/Numeric.ftl Outdated Show resolved Hide resolved
@lbooker42 lbooker42 requested a review from chipkent June 18, 2024 21:19
engine/function/src/templates/Numeric.ftl Outdated Show resolved Hide resolved
engine/function/src/templates/Numeric.ftl Outdated Show resolved Hide resolved
rcaudy
rcaudy previously approved these changes Jun 18, 2024
Copy link
Member

@rcaudy rcaudy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor comments, but approved regardless.

engine/function/src/templates/Numeric.ftl Outdated Show resolved Hide resolved
engine/function/src/templates/Numeric.ftl Outdated Show resolved Hide resolved
Accepting PR suggestions

Co-authored-by: Ryan Caudy <rcaudy@gmail.com>
Accepting PR suggestions

Co-authored-by: Ryan Caudy <rcaudy@gmail.com>
@lbooker42 lbooker42 merged commit 83f62db into deephaven:main Jun 19, 2024
15 checks passed
@github-actions github-actions bot locked and limited conversation to collaborators Jun 19, 2024
@deephaven-internal
Copy link
Contributor

Labels indicate documentation is required. Issues for documentation have been opened:

Community: deephaven/deephaven-docs-community#239

@lbooker42 lbooker42 deleted the lab-numeric-harmony branch June 26, 2024 19:56
@chipkent chipkent removed the devrel-watch DevRel team is watching label Aug 5, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Harmonize Numeric/Agg/UpdateBy calculation ouput
4 participants