[bugfix] Optimize testcase queries over time period #3256
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
vkarak
added
prio: important
bugfix
reporting
vkarak
changed the title
vkarak
force-pushed
the
bugfix/optimize-testcase-queries
branch
from
1b40352 to
e23a5e4
Compare
teojgo
approved these changes
Sep 6, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a follow up fix on #3227 and build on top of #3253.
This PR eliminates the JOIN between the
testcasesandsessionsarrays as is this very detrimental to performance for sessions that run many testcases. The problem with the current implementation is that the JOIN will replicate the whole sessionjson_blobfor every testcase record bloating the memory consumption and hurting performance significantly!The approach taken in this PR is replacing the single JOIN query with 2+1 SELECT. First we get the distinct session UUIDs that correspond to the test cases matching the selection criteria. Then we use those session UUIDs to extract their blobs from the
sessionstable decoding them and indexing them in memory. Finally, we get the testcase UUIDs matching the criteria selection and from this we are able to quickly pick the JSON data of each selected test case.I tested this impelmentation a DB of 163 sessions and 38K test cases on two systems and we get 6-10x performance improvements on the
--performance-compare/--performance-reportoptions (from 20s to 2s and from 27s down to 4s). Now, the DB query is no more the limiting performance factor, but rather the JSON decoding and the aggregation of the results.