Improve pouchDB performances #1537
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The `view_update_changes_batch_size` parameter is used to determine the docs batch size when computing view updates. PouchDB default is 50, which is quite low, and badly hurt performancs when dealing with thousands of docs and the first view computation. With a 44K files database, in a react-native env: **Before:** Indexing time: 53 868 ms **After:** Indexing time: 31 833 ms
The react-native PouchDB adapter does not handle partialIndexes: it means that any predicate in partialIndex will not be evaluated at indexing time, but at querying time. When dealing with huge database, this can be quite impactful because it will result in full-scan of huge indexes. Therefore, we now force the indexing of any field in partialIndex, to guarantee a doc will be indexed if and only if it has all the necessary attributes. We measured performances for this query: https://github.com/cozy/mespapiers/blob/ e456faaa00dcefef13c6eee88459caffcac4f2d4/src/queries/index.js#L64-L105 **Before:** Query time: 51 499 ms **After:** Query time: 1 854 ms
Ldoppea
reviewed
Oct 3, 2024
| indexedFields = Array.from( | ||
| new Set([...indexedFields, ...Object.keys(partialFilter)]) | ||
| ) | ||
| // FIXME: should properly handle n-level attributes |
There was a problem hiding this comment.
What's the impact for now? Should we handle this soon? Or it is for an hypothetical future?
There was a problem hiding this comment.
For now it means that if we have queries with partialIndex having attributes not on the root level, i.e. with $and: [{...}], we won't benefit from this fix. Thus slow queries.
So hopefully there is no regression regarding the current offline, but we should test it carefully
Ldoppea
approved these changes
Oct 3, 2024
Ldoppea
added a commit
to cozy/cozy-flagship-app
that referenced
this pull request
Oct 3, 2024
`cozy-pouch-link` has been upgraded to `49.1.0` to retrieve performance improvements on Pouch queries Related PR: cozy/cozy-client#1537
|
Nice! 30s to index 40k documents is still huge :( |
It is indeed. While this time would be dramatically reduced on a more powerful device, this is disappointing, but not so surprising regarding previous performances measures, that tend to show massive performances differences depending on the adapter and document size : |
Ldoppea
added a commit
to cozy/cozy-flagship-app
that referenced
this pull request
Nov 12, 2024
`cozy-pouch-link` has been upgraded to `49.1.0` to retrieve performance improvements on Pouch queries Related PR: cozy/cozy-client#1537
Ldoppea
added a commit
to cozy/cozy-flagship-app
that referenced
this pull request
Dec 19, 2024
`cozy-pouch-link` has been upgraded to `49.1.0` to retrieve performance improvements on Pouch queries Related PR: cozy/cozy-client#1537
Ldoppea
added a commit
to cozy/cozy-flagship-app
that referenced
this pull request
Dec 20, 2024
`cozy-pouch-link` has been upgraded to `49.1.0` to retrieve performance improvements on Pouch queries Related PR: cozy/cozy-client#1537
Ldoppea
added a commit
to cozy/cozy-flagship-app
that referenced
this pull request
Dec 20, 2024
`cozy-pouch-link` has been upgraded to `49.1.0` to retrieve performance improvements on Pouch queries Related PR: cozy/cozy-client#1537
Ldoppea
added a commit
to cozy/cozy-flagship-app
that referenced
this pull request
Dec 20, 2024
`cozy-pouch-link` has been upgraded to `49.1.0` to retrieve performance improvements on Pouch queries Related PR: cozy/cozy-client#1537
Ldoppea
added a commit
to cozy/cozy-flagship-app
that referenced
this pull request
Dec 20, 2024
`cozy-pouch-link` has been upgraded to `49.1.0` to retrieve performance improvements on Pouch queries Related PR: cozy/cozy-client#1537
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We noticed severe performances issues when using pouchdb in a react-native environment.
See commit descriptions for more insights
BEFORE
AFTER