Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Allow DataIndex deferred materialization to parallelize when nested inside a parallel operation #5736

Closed
rcaudy opened this issue Jul 9, 2024 · 0 comments · Fixed by #5789
Closed

Allow DataIndex deferred materialization to parallelize when nested inside a parallel operation #5736

rcaudy opened this issue Jul 9, 2024 · 0 comments · Fixed by #5789
Assignees
@rcaudy
Labels
bug Something isn't working core Core development tasks query engine
Milestone
0.36.0

Comments

@rcaudy
Copy link
Member

rcaudy commented Jul 9, 2024

See:

  • MergedDataIndex.buildTable()
  • TableBackedDataIndex.computeTable() // Nothing to be gained here, but might as well address for future-proofing
  • TransformedDataIndex.table()

All have been written to allow parallelization where possible, but nested parallelization is normally not permitted, which creates performance issues under some circumstances.
@rcaudy rcaudy added bug Something isn't working triage labels Jul 9, 2024
@rcaudy rcaudy self-assigned this Jul 9, 2024
@rcaudy rcaudy added this to the 0.36.0 milestone Jul 9, 2024
@rcaudy rcaudy added query engine core Core development tasks and removed triage labels Jul 9, 2024
@rcaudy rcaudy mentioned this issue Jul 17, 2024
Merged
rcaudy added a commit to rcaudy/deephaven-core that referenced this issue Jul 18, 2024
@rcaudy
perf: Add ForkJoinPool.commonPool()-based OperationInitializer for ne…
22c5c61 
…sted parallel DataIndex building (deephaven#5789)

Introduce a very simple `ForkJoinPool`-based `OperationInitializer`,
wrapping the common pool, and use that to ensure parallelism in
`MergedDataIndex` and `TransformedDataIndex` deferred initialization.

Fixes deephaven#5736
@rcaudy rcaudy mentioned this issue Jul 18, 2024
Merged
rcaudy added a commit that referenced this issue Jul 18, 2024
@rcaudy
perf: Add ForkJoinPool.commonPool()-based OperationInitializer for ne…
4bf035b 
…sted parallel DataIndex building (#5802)

Introduce a very simple `ForkJoinPool`-based `OperationInitializer`,
wrapping the common pool, and use that to ensure parallelism in
`MergedDataIndex` and `TransformedDataIndex` deferred initialization.

Fixes #5736
Cherry pick of #5789
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rcaudy rcaudy

Labels
bug Something isn't working core Core development tasks query engine
Projects
None yet
Milestone
0.36.0
1 participant
@rcaudy