Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Numeric.percentile bug for empty input #2611

Closed
mattrunyon opened this issue Jun 30, 2022 · 1 comment · Fixed by #3023
Closed

Numeric.percentile bug for empty input #2611

mattrunyon opened this issue Jun 30, 2022 · 1 comment · Fixed by #3023
Assignees
@chipkent
Labels
bug Something isn't working query engine
Milestone
Jul 2022

Comments

@mattrunyon
Copy link
Contributor

Description

During the nightly docs test which runs all code snippets in the docs for validity, the following error occurred while executing this Python snippet

import deephaven.perfmon as pm

qup = pm.query_update_performance(1)
qp = pm.query_performance(1)
qop = pm.query_operation_performance(1)
ss = pm.server_state()
  heduler-Concurrent-3 | .b.BarrageMessageProducer | Internal Error 'e5c7be57-7583-4e82-8a0a-43d94c913ee8' io.deephaven.UncheckedDeephavenException: Failure to execute snapshot function with unchanged clock
          at io.deephaven.engine.table.impl.remote.ConstructSnapshot.callDataSnapshotFunction(ConstructSnapshot.java:1160)
          at io.deephaven.engine.table.impl.remote.ConstructSnapshot.callDataSnapshotFunction(ConstructSnapshot.java:1019)
          at io.deephaven.engine.table.impl.remote.ConstructSnapshot.constructBackplaneSnapshotInPositionSpace(ConstructSnapshot.java:599)
          at io.deephaven.server.barrage.BarrageMessageProducer.getSnapshot(BarrageMessageProducer.java:2227)
          at io.deephaven.server.barrage.BarrageMessageProducer.updateSubscriptionsSnapshotAndPropagate(BarrageMessageProducer.java:1344)
          at io.deephaven.server.barrage.BarrageMessageProducer$UpdatePropagationJob.run(BarrageMessageProducer.java:1019)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
          at java.util.concurrent.FutureTask.run(FutureTask.java:264)
          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
          at io.deephaven.server.runner.DeephavenApiServerModule$ThreadFactory.lambda$newThread$0(DeephavenApiServerModule.java:160)
          at java.lang.Thread.run(Thread.java:829)
  caused by:
  io.deephaven.engine.table.impl.select.FormulaEvaluationException: In formula: UGPCycleP90Secs = divide(percentile(0.9, IntervalUGPCyclesTimeMicros), (multiply(1000, 1000.0)))
          at io.deephaven.temp.cm336573424987228486v55_0.Formula.applyFormulaPerItem(Formula.java:161)
          at io.deephaven.temp.cm336573424987228486v55_0.Formula.lambda$fillChunkHelper$4(Formula.java:150)
          at io.deephaven.engine.rowset.RowSequence.lambda$forAllRowKeys$0(RowSequence.java:179)
          at io.deephaven.engine.rowset.impl.RowSequenceRowKeysChunkImpl.forEachRowKey(RowSequenceRowKeysChunkImpl.java:206)
          at io.deephaven.engine.rowset.RowSequence.forAllRowKeys(RowSequence.java:178)
          at io.deephaven.temp.cm336573424987228486v55_0.Formula.fillChunkHelper(Formula.java:148)
          at io.deephaven.temp.cm336573424987228486v55_0.Formula.fillChunk(Formula.java:125)
          at io.deephaven.engine.table.impl.sources.ViewColumnSource.fillChunk(ViewColumnSource.java:254)
          at io.deephaven.engine.table.impl.sources.RedirectedColumnSource$FillContext.doOrderedFillAscending(RedirectedColumnSource.java:717)
          at io.deephaven.engine.table.impl.sources.RedirectedColumnSource.doFillChunk(RedirectedColumnSource.java:431)
          at io.deephaven.engine.table.impl.sources.RedirectedColumnSource.fillChunk(RedirectedColumnSource.java:409)
          at io.deephaven.engine.table.impl.remote.ConstructSnapshot.getSnapshotDataAsChunkList(ConstructSnapshot.java:1462)
          at io.deephaven.engine.table.impl.remote.ConstructSnapshot.serializeAllTable(ConstructSnapshot.java:1344)
          at io.deephaven.engine.table.impl.remote.ConstructSnapshot.lambda$constructBackplaneSnapshotInPositionSpace$2(ConstructSnapshot.java:594)
          at io.deephaven.engine.table.impl.remote.ConstructSnapshot.callDataSnapshotFunction(ConstructSnapshot.java:1087)
          at io.deephaven.engine.table.impl.remote.ConstructSnapshot.callDataSnapshotFunction(ConstructSnapshot.java:1019)
          at io.deephaven.engine.table.impl.remote.ConstructSnapshot.constructBackplaneSnapshotInPositionSpace(ConstructSnapshot.java:599)
          at io.deephaven.server.barrage.BarrageMessageProducer.getSnapshot(BarrageMessageProducer.java:2227)
          at io.deephaven.server.barrage.BarrageMessageProducer.updateSubscriptionsSnapshotAndPropagate(BarrageMessageProducer.java:1344)
          at io.deephaven.server.barrage.BarrageMessageProducer$UpdatePropagationJob.run(BarrageMessageProducer.java:1019)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
          at java.util.concurrent.FutureTask.run(FutureTask.java:264)
          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
          at io.deephaven.server.runner.DeephavenApiServerModule$ThreadFactory.lambda$newThread$0(DeephavenApiServerModule.java:160)
          at java.lang.Thread.run(Thread.java:829)
  caused by:
  java.lang.ArrayIndexOutOfBoundsException: Index -1 out of bounds for length 0
          at io.deephaven.function.Numeric.percentile(Numeric.java:12816)
          at io.deephaven.function.Numeric.percentile(Numeric.java:12792)
          at io.deephaven.temp.cm336573424987228486v55_0.Formula.applyFormulaPerItem(Formula.java:159)
          at io.deephaven.temp.cm336573424987228486v55_0.Formula.lambda$fillChunkHelper$4(Formula.java:150)
          at io.deephaven.engine.rowset.RowSequence.lambda$forAllRowKeys$0(RowSequence.java:179)
          at io.deephaven.engine.rowset.impl.RowSequenceRowKeysChunkImpl.forEachRowKey(RowSequenceRowKeysChunkImpl.java:206)
          at io.deephaven.engine.rowset.RowSequence.forAllRowKeys(RowSequence.java:178)
          at io.deephaven.temp.cm336573424987228486v55_0.Formula.fillChunkHelper(Formula.java:148)
          at io.deephaven.temp.cm336573424987228486v55_0.Formula.fillChunk(Formula.java:125)
          at io.deephaven.engine.table.impl.sources.ViewColumnSource.fillChunk(ViewColumnSource.java:254)
          at io.deephaven.engine.table.impl.sources.RedirectedColumnSource$FillContext.doOrderedFillAscending(RedirectedColumnSource.java:717)
          at io.deephaven.engine.table.impl.sources.RedirectedColumnSource.doFillChunk(RedirectedColumnSource.java:431)
          at io.deephaven.engine.table.impl.sources.RedirectedColumnSource.fillChunk(RedirectedColumnSource.java:409)
          at io.deephaven.engine.table.impl.remote.ConstructSnapshot.getSnapshotDataAsChunkList(ConstructSnapshot.java:1462)
          at io.deephaven.engine.table.impl.remote.ConstructSnapshot.serializeAllTable(ConstructSnapshot.java:1344)
          at io.deephaven.engine.table.impl.remote.ConstructSnapshot.lambda$constructBackplaneSnapshotInPositionSpace$2(ConstructSnapshot.java:594)
          at io.deephaven.engine.table.impl.remote.ConstructSnapshot.callDataSnapshotFunction(ConstructSnapshot.java:1087)
          at io.deephaven.engine.table.impl.remote.ConstructSnapshot.callDataSnapshotFunction(ConstructSnapshot.java:1019)
          at io.deephaven.engine.table.impl.remote.ConstructSnapshot.constructBackplaneSnapshotInPositionSpace(ConstructSnapshot.java:599)
          at io.deephaven.server.barrage.BarrageMessageProducer.getSnapshot(BarrageMessageProducer.java:2227)
          at io.deephaven.server.barrage.BarrageMessageProducer.updateSubscriptionsSnapshotAndPropagate(BarrageMessageProducer.java:1344)
          at io.deephaven.server.barrage.BarrageMessageProducer$UpdatePropagationJob.run(BarrageMessageProducer.java:1019)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
          at java.util.concurrent.FutureTask.run(FutureTask.java:264)
          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
          at io.deephaven.server.runner.DeephavenApiServerModule$ThreadFactory.lambda$newThread$0(DeephavenApiServerModule.java:160)
          at java.lang.Thread.run(Thread.java:829)

Additional details and attachments

When I run the code locally, it does not error. It's possible the state only happened from the docs snapshots running ~75 other examples before this and resetting the workspace just by clearing the symbols added to the python globals scope and local scope. I am unable to reproduce, but figured the out of bounds exception is an uncaught edge case in some code somewhere.

Versions
@mattrunyon mattrunyon added bug Something isn't working triage labels Jun 30, 2022
@rcaudy
Copy link
Member

rcaudy commented Jun 30, 2022 •
edited
Loading

I think this is just a bug in Numeric.percentile: it's not currently safe to call with an empty input. Empty input will result in trying to dereference the array's -1th or 0th element, depending on the percentile used.

The code in question:

    /**
     * Returns the percentile.
     *
     * @param percentile percentile to compute.
     * @param values values.
     * @return percentile.
     */
    public static double percentile(double percentile, IntVector values) {
        if(values == null){
            return NULL_DOUBLE;
        }

        if (percentile < 0 || percentile > 1) {
            throw new IllegalArgumentException("Invalid percentile = " + percentile);
        }

        int n = values.intSize("percentile");
        int[] copy = values.toArray();
        Arrays.sort(copy);

        int idx = (int) Math.round(percentile * (n - 1));
        return copy[idx];
    }

I think on values.isEmpty() we should be returning NULL_DOUBLE or NAN_DOUBLE.

@rcaudy rcaudy changed the title Failed to execute snapshot with unchanged clock on performance queries Numerics.percentile bug for empty input Jun 30, 2022
@rcaudy rcaudy changed the title Numerics.percentile bug for empty input Numeric.percentile bug for empty input Jun 30, 2022
@rcaudy rcaudy added this to the Jul 2022 milestone Jun 30, 2022
@rcaudy rcaudy removed the triage label Jun 30, 2022
chipkent added a commit to chipkent/deephaven-core that referenced this issue Oct 24, 2022
@chipkent
Fix percentile error.
168c7cf 
Resolves deephaven#2611
@chipkent chipkent mentioned this issue Oct 24, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@chipkent chipkent

Labels
bug Something isn't working query engine
Projects
None yet
Milestone
Jul 2022
3 participants
@chipkent @rcaudy @mattrunyon