Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow moving_function to access percentile sketches #49452

Closed
polyfractal opened this issue Nov 21, 2019 · 1 comment · Fixed by #55441
Closed

Allow moving_function to access percentile sketches #49452

polyfractal opened this issue Nov 21, 2019 · 1 comment · Fixed by #55441
Assignees
Labels
:Analytics/Aggregations Aggregations >enhancement Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo)

Comments

@polyfractal
Copy link
Contributor

If the moving_function pipeline agg could access and merge the internal percentiles sketch (HDRHisto or TDigest), it would allow users to construct moving windows of percentiles. E.g. bucket the results at 1 minute intervals and apply a percentiles agg per-bucket, then using a moving_function over 5 buckets and inside the script merge the sketches to produce a single sketch for that window of time.

When plotted, this gives a percentiles output of the last five minutes, every minute.

This is similar to what the cumulative cardinality agg does, except merging windows of percentiles together instead of cumulatively merging a cardinality sketch.

I think the main difficulty will be exposing this functionality in a safe/simple manner to painless. I think we would first need to allow percentiles objects as a valid input (instead of just numerics/multi-valued stats) and then build a dedicated painless function similar to the existing moving averages (e.g. MovingFunctions.percentiles(values))

If we can't find a good way to do this, we could always create a specialized "moving percentiles" pipeline agg but it would probably be better to extend moving_function first if possible.

@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-analytics-geo (:Analytics/Aggregations)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Analytics/Aggregations Aggregations >enhancement Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo)
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants