Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve pouchDB performances #1537

Merged
merged 2 commits into from
Oct 3, 2024
Merged

Improve pouchDB performances #1537

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f0b5a81
feat: Increase PouchdDB default batch size for view updates
paultranvan Oct 3, 2024
cf9ec39
feat: Improve PouchDB query performances with partialIndex
paultranvan Oct 3, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 9 additions & 9 deletions docs/api/cozy-pouch-link/classes/PouchLink.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -132,7 +132,7 @@ CozyLink.constructor

*Defined in*

[CozyPouchLink.js:689](https://github.com/cozy/cozy-client/blob/master/packages/cozy-pouch-link/src/CozyPouchLink.js#L689)
[CozyPouchLink.js:702](https://github.com/cozy/cozy-client/blob/master/packages/cozy-pouch-link/src/CozyPouchLink.js#L702)

***

Expand All @@ -152,7 +152,7 @@ CozyLink.constructor

*Defined in*

[CozyPouchLink.js:650](https://github.com/cozy/cozy-client/blob/master/packages/cozy-pouch-link/src/CozyPouchLink.js#L650)
[CozyPouchLink.js:663](https://github.com/cozy/cozy-client/blob/master/packages/cozy-pouch-link/src/CozyPouchLink.js#L663)

***

Expand Down Expand Up @@ -199,7 +199,7 @@ Create the PouchDB index if not existing

*Defined in*

[CozyPouchLink.js:693](https://github.com/cozy/cozy-client/blob/master/packages/cozy-pouch-link/src/CozyPouchLink.js#L693)
[CozyPouchLink.js:706](https://github.com/cozy/cozy-client/blob/master/packages/cozy-pouch-link/src/CozyPouchLink.js#L706)

***

Expand All @@ -219,7 +219,7 @@ Create the PouchDB index if not existing

*Defined in*

[CozyPouchLink.js:678](https://github.com/cozy/cozy-client/blob/master/packages/cozy-pouch-link/src/CozyPouchLink.js#L678)
[CozyPouchLink.js:691](https://github.com/cozy/cozy-client/blob/master/packages/cozy-pouch-link/src/CozyPouchLink.js#L691)

***

Expand All @@ -241,7 +241,7 @@ Create the PouchDB index if not existing

*Defined in*

[CozyPouchLink.js:620](https://github.com/cozy/cozy-client/blob/master/packages/cozy-pouch-link/src/CozyPouchLink.js#L620)
[CozyPouchLink.js:633](https://github.com/cozy/cozy-client/blob/master/packages/cozy-pouch-link/src/CozyPouchLink.js#L633)

***

Expand All @@ -261,7 +261,7 @@ Create the PouchDB index if not existing

*Defined in*

[CozyPouchLink.js:558](https://github.com/cozy/cozy-client/blob/master/packages/cozy-pouch-link/src/CozyPouchLink.js#L558)
[CozyPouchLink.js:571](https://github.com/cozy/cozy-client/blob/master/packages/cozy-pouch-link/src/CozyPouchLink.js#L571)

***

Expand Down Expand Up @@ -701,7 +701,7 @@ Emits pouchlink:sync:stop event

*Defined in*

[CozyPouchLink.js:715](https://github.com/cozy/cozy-client/blob/master/packages/cozy-pouch-link/src/CozyPouchLink.js#L715)
[CozyPouchLink.js:728](https://github.com/cozy/cozy-client/blob/master/packages/cozy-pouch-link/src/CozyPouchLink.js#L728)

***

Expand All @@ -721,7 +721,7 @@ Emits pouchlink:sync:stop event

*Defined in*

[CozyPouchLink.js:655](https://github.com/cozy/cozy-client/blob/master/packages/cozy-pouch-link/src/CozyPouchLink.js#L655)
[CozyPouchLink.js:668](https://github.com/cozy/cozy-client/blob/master/packages/cozy-pouch-link/src/CozyPouchLink.js#L668)

***

Expand All @@ -741,7 +741,7 @@ Emits pouchlink:sync:stop event

*Defined in*

[CozyPouchLink.js:660](https://github.com/cozy/cozy-client/blob/master/packages/cozy-pouch-link/src/CozyPouchLink.js#L660)
[CozyPouchLink.js:673](https://github.com/cozy/cozy-client/blob/master/packages/cozy-pouch-link/src/CozyPouchLink.js#L673)

***

Expand Down
13 changes: 13 additions & 0 deletions packages/cozy-pouch-link/src/CozyPouchLink.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -539,6 +539,19 @@ class PouchLink extends CozyLink {

if (!indexedFields) {
indexedFields = getIndexFields(options)
} else if (partialFilter) {
// Some pouch adapters does not support partialIndex, e.g. with websql in react-native
// Therefore, we need to force the indexing the partialIndex fields to ensure they will be
// included in the actual index. Thanks to this, docs with missing fields will be excluded
// from the index.
// Note the $exists: false case should be handled in-memory.
indexedFields = Array.from(
new Set([...indexedFields, ...Object.keys(partialFilter)])
)
// FIXME: should properly handle n-level attributes
Copy link
Member

@Ldoppea Ldoppea Oct 3, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the impact for now? Should we handle this soon? Or it is for an hypothetical future?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For now it means that if we have queries with partialIndex having attributes not on the root level, i.e. with $and: [{...}], we won't benefit from this fix. Thus slow queries.
So hopefully there is no regression regarding the current offline, but we should test it carefully

indexedFields = indexedFields.filter(
field => field !== '$and' && field !== '$or'
)
}

const indexName = getIndexNameFromFields(indexedFields, partialFilter)
Expand Down
39 changes: 36 additions & 3 deletions packages/cozy-pouch-link/src/CozyPouchLink.spec.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -608,9 +608,10 @@ describe('CozyPouchLink', () => {
expect(spy).toHaveBeenCalled()
expect(spy).toHaveBeenCalledWith({
index: {
ddoc: 'by_myIndex_filter_(SOME_FIELD_$exists_true)',
fields: ['myIndex'],
indexName: 'by_myIndex_filter_(SOME_FIELD_$exists_true)',
ddoc: 'by_myIndex_and_SOME_FIELD_filter_(SOME_FIELD_$exists_true)',
fields: ['myIndex', 'SOME_FIELD'],
indexName:
'by_myIndex_and_SOME_FIELD_filter_(SOME_FIELD_$exists_true)',
partial_filter_selector: {
SOME_FIELD: {
$exists: true
Expand All @@ -620,6 +621,38 @@ describe('CozyPouchLink', () => {
})
})

it('should exclude $and and $or operators from fields with partialIndex', async () => {
spy = jest.spyOn(PouchDB.prototype, 'createIndex').mockReturnValue({})
await setup()
await link.ensureIndex(TODO_DOCTYPE, {
indexedFields: ['myIndex'],
partialFilter: {
$and: [
{ SOME_FIELD: { $exists: true } },
{ SOME_FIELD: { $gt: null } }
],
$or: [{ SOME_FIELD: { $eq: '1' } }, { SOME_FIELD: { $eq: '2' } }]
}
})
expect(spy).toHaveBeenCalled()
expect(spy).toHaveBeenCalledWith({
index: {
ddoc:
'by_myIndex_filter_((SOME_FIELD_$exists_true)_$and_(SOME_FIELD_$gt_null))_and_((SOME_FIELD_$eq_1)_$or_(SOME_FIELD_$eq_2))',
fields: ['myIndex'],
indexName:
'by_myIndex_filter_((SOME_FIELD_$exists_true)_$and_(SOME_FIELD_$gt_null))_and_((SOME_FIELD_$eq_1)_$or_(SOME_FIELD_$eq_2))',
partial_filter_selector: {
$and: [
{ SOME_FIELD: { $exists: true } },
{ SOME_FIELD: { $gt: null } }
],
$or: [{ SOME_FIELD: { $eq: '1' } }, { SOME_FIELD: { $eq: '2' } }]
}
}
})
})

it('uses the specified index', async () => {
let spyIndex = jest
.spyOn(CozyPouchLink.prototype, 'ensureIndex')
Expand Down
7 changes: 7 additions & 0 deletions packages/cozy-pouch-link/src/PouchManager.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,10 @@ import { formatAggregatedError, getDatabaseName } from './utils'

const DEFAULT_DELAY = 30 * 1000

// See view_update_changes_batch_size in https://pouchdb.com/api.html#create_database
// PouchDB default is 50, which badly hurt performances for large databases
const DEFAULT_VIEW_UPDATE_BATCH = 1000

/**
* @param {import('cozy-client/types/types').Query} query The query definition whose name we're getting
*
Expand Down Expand Up @@ -43,6 +47,9 @@ class PouchManager {
async init() {
const pouchPlugins = get(this.options, 'pouch.plugins', [])
const pouchOptions = get(this.options, 'pouch.options', {})
if (!pouchOptions.view_update_changes_batch_size) {
pouchOptions.view_update_changes_batch_size = DEFAULT_VIEW_UPDATE_BATCH
}

forEach(pouchPlugins, plugin => this.PouchDB.plugin(plugin))
this.pouches = fromPairs(
Expand Down
Loading