Improve single schema cache #7214

dplewis · 2021-02-21T02:17:16Z

New Pull Request Checklist

I am not disclosing a vulnerability.
I am creating this PR in reference to an issue.

Issue Description

The Server has been known to show increasing memory usage over time.
The server keeps a copy of the schema per request. In order to make a copy it must query the database _SCHEMA class. To prevent an invalid schema it may query the database multiple times.

Closes: #6405, #6193, #7036, #6543

Approach

Create a global schema cache instead of a cache per request. This global schema cache will be initialized the the server starts and will be updated internally. If you are running multiple instances of the server and they connect to a single database the global schema will also update by a database hook that listens to the _SCHEMA class.

Removes enableSchemaCache and schemaCacheTTL options
Adds horizontalScaling option to use db hooks.

This PR is a continuation of #6743 Shoutout to @SebC99 for starting the Proof of Concept. 🎉

TODOs before merging

Add test cases
Add entry to changelog
Add changes to documentation (guides, repository pages, in-code descriptions)
Add security check
Add new Parse Error codes to Parse JS SDK
...

codecov · 2021-02-21T02:23:42Z

Codecov Report

Merging #7214 (e88fc9a) into master (32fc45d) will decrease coverage by 0.07%.
The diff coverage is 91.93%.

@@            Coverage Diff             @@
##           master    #7214      +/-   ##
==========================================
- Coverage   93.97%   93.90%   -0.08%     
==========================================
  Files         179      179              
  Lines       13138    13152      +14     
==========================================
+ Hits        12347    12350       +3     
- Misses        791      802      +11

Impacted Files	Coverage Δ
src/Options/index.js	`100.00% <ø> (ø)`
src/PromiseRouter.js	`90.58% <ø> (+0.47%)`	⬆️
src/Controllers/DatabaseController.js	`93.70% <79.41%> (-1.77%)`	⬇️
...dapters/Storage/Postgres/PostgresStorageAdapter.js	`95.36% <90.00%> (-0.14%)`	⬇️
src/Adapters/Auth/instagram.js	`91.66% <100.00%> (ø)`
src/Adapters/Cache/SchemaCache.js	`100.00% <100.00%> (ø)`
src/Adapters/Storage/Mongo/MongoStorageAdapter.js	`92.59% <100.00%> (+0.33%)`	⬆️
src/Config.js	`93.30% <100.00%> (-0.06%)`	⬇️
src/Controllers/SchemaController.js	`97.35% <100.00%> (+0.02%)`	⬆️
src/Controllers/index.js	`97.77% <100.00%> (+1.11%)`	⬆️
... and 14 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 32fc45d...9cd2986. Read the comment docs.

SebC99 · 2021-02-21T09:16:30Z

Nice!!

dplewis · 2021-02-21T17:11:50Z

@vitaly-t If you have a second can you look this over? I'm basically using Notify / Listen to get notified of changes to the _SCHEMA class. I used this approach instead of triggers for permission reasons.

https://github.com/parse-community/parse-server/pull/7214/files#diff-06a6bf7bf8f7198c26642684e9c8ecaebb042d28eec43c17171d8054dc675964

Any feedback would be greatly appreciated.

dplewis · 2021-02-21T17:22:49Z

@davimacedo @SebC99 @mtrezza At this point the tests are passing (excluding flaky ones). I'm going to run a few more tests.

dplewis · 2021-02-22T14:26:21Z

Quick update, looks like the PushController.spec and Parse.Push.spec doesn’t like this PR. Everything passes locally. I’ll try to rewrite the tests or increase the timeout.

Edit: If you want to reproduce this locally setTimeout to 1000 to 0 here

dplewis · 2021-02-23T10:47:23Z

The tests are passing finally, I thought I would never see it.
I did have to xit a few tests. This feature is too good to not merge because of a few flaky tests.

mtrezza

Great PR, can't wait to try this out! Just MongoDB needs an upgrade it seems to the CI check to pass.

dplewis · 2021-03-16T21:06:06Z

I'm going to run this in production and report back

mtrezza · 2021-03-16T21:08:00Z

Oh, but did you upgrade MongoDB? The CI check didn't pass.
@dplewis never mind, I'll open a quick PR to make the main branch pass CI.

SebC99 · 2021-03-16T21:13:41Z

Congratulations @dplewis !!!!

mtrezza · 2021-03-17T22:10:17Z

@dplewis This test added here seems to be flaky, already added to #7180:

Schema Performance test schema update class
Error: Could not add field fooOne

* Initial Commit * fix flaky test * temporary set ci timeout * turn off ci check * fix postgres tests * fix tests * node flaky test * remove improvements * Update SchemaPerformance.spec.js * fix tests * revert ci * Create Singleton Object * properly clear cache testing * Cleanup * remove fit * try PushController.spec * try push test rewrite * try push enqueue time * Increase test timeout * remove pg server creation test * xit push tests * more xit * remove skipped tests * Fix conflicts * reduce ci timeout * fix push tests * Revert "fix push tests" This reverts commit 05aba62. * improve initialization * fix flaky tests * xit flaky test * Update CHANGELOG.md * enable debug logs * Update LogsRouter.spec.js * create initial indexes in series * lint * horizontal scaling documentation * Update Changelog * change horizontalScaling db option * Add enableSchemaHooks option * move enableSchemaHooks to databaseOptions

mtrezza · 2021-03-31T11:36:53Z

I think this should be looked at closely before making a release with this PR merged, because we want to make sure there isn’t a bug in the new schema mechanism.

In retrospect I think a PR of this magnitude should have been phased in instead of being a sudden breaking change, to collect feedback over some time. That speaks for the deprecation policy that we’ll apply from 5.0.0 going forward.

mtrezza · 2021-04-03T20:51:36Z

I'm going to run this in production and report back

@dplewis Do you have any feedback to share yet?

parseplatformorg · 2021-11-01T16:32:09Z

🎉 This change has been released in version 5.0.0-beta.1

parseplatformorg · 2022-03-14T13:04:11Z

🎉 This change has been released in version 5.0.0

Initial Commit

9db2063

dplewis added 10 commits February 20, 2021 20:43

fix flaky test

311ac85

temporary set ci timeout

a4a1a3a

turn off ci check

ca3884e

fix postgres tests

b7e28cd

fix tests

7233af6

node flaky test

cf65542

remove improvements

a3ab545

Update SchemaPerformance.spec.js

f91d0c5

fix tests

2c8cf56

revert ci

d49e4d4

dplewis added 2 commits February 21, 2021 10:56

Create Singleton Object

a6c692d

properly clear cache testing

ebf67d3

dplewis force-pushed the schema-improvement branch from 0f7b4c7 to ebf67d3 Compare February 21, 2021 20:59

Cleanup

5454895

dplewis requested a review from vitaly-t February 21, 2021 23:13

dplewis added 2 commits February 21, 2021 17:53

remove fit

3b91380

try PushController.spec

415df3b

dplewis added 6 commits February 22, 2021 11:34

try push test rewrite

81a7d2a

try push enqueue time

231383b

Increase test timeout

e36bf3c

remove pg server creation test

3871115

xit push tests

a83ef11

more xit

07b06b8

dplewis requested a review from mtrezza March 16, 2021 20:47

mtrezza approved these changes Mar 16, 2021

View reviewed changes

dplewis merged commit a02014f into master Mar 16, 2021

dplewis deleted the schema-improvement branch March 16, 2021 21:05

dplewis mentioned this pull request Mar 16, 2021

Lots of calls to the cache to get Schema during the same query #6193

Closed

dplewis mentioned this pull request Mar 16, 2021

Attempt to remove the schemaCache #6743

Closed

3 tasks

mtrezza mentioned this pull request Mar 16, 2021

Memory Leak On Queries (Due To Caching/Certain Parse-Server And Node Versions) #6405

Closed

This was referenced Mar 16, 2021

Parse Server performance loss #7036

Closed

Fetch large data for reports slow #6543

Open

mtrezza mentioned this pull request Mar 16, 2021

bump mongodb version in CI #7272

Merged

dplewis mentioned this pull request Mar 17, 2021

Enable direct access by default #6636

Merged

6 tasks

mtrezza mentioned this pull request Apr 1, 2021

Add deprecation policy #7199

Merged

3 tasks

parseplatformorg added the state:released-beta Released as beta version label Nov 1, 2021

mtrezza mentioned this pull request Mar 12, 2022

build: release 5.0 #7851

Merged

parseplatformorg added the state:released Released as stable version label Mar 14, 2022

mtrezza mentioned this pull request Sep 14, 2022

docs: databaseOptions missing in Parse Server Options docs #8168

Closed

mtrezza mentioned this pull request Oct 12, 2022

Re-add schema pulling #8229

Closed

4 tasks

This was referenced Mar 8, 2023

Remove unsupported db options from mongo connection SynapLearning/parse-server#2

Merged

Database options connect fix #8470

Closed

dblythy mentioned this pull request Mar 9, 2023

GridFS fails with enableSchemaHooks is not supported #8466

Closed

4 tasks

cbaker6 mentioned this pull request Jun 25, 2023

refactor: Remove duplicate user index creation #8662

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve single schema cache #7214

Improve single schema cache #7214

dplewis commented Feb 21, 2021 •

edited by TomWFox

Loading

codecov bot commented Feb 21, 2021 •

edited

Loading

SebC99 commented Feb 21, 2021

dplewis commented Feb 21, 2021 •

edited

Loading

dplewis commented Feb 21, 2021

dplewis commented Feb 22, 2021 •

edited

Loading

dplewis commented Feb 23, 2021 •

edited

Loading

mtrezza left a comment •

edited

Loading

dplewis commented Mar 16, 2021

mtrezza commented Mar 16, 2021 •

edited

Loading

SebC99 commented Mar 16, 2021

mtrezza commented Mar 17, 2021

mtrezza commented Mar 31, 2021 •

edited

Loading

mtrezza commented Apr 3, 2021

parseplatformorg commented Nov 1, 2021

parseplatformorg commented Mar 14, 2022

Improve single schema cache #7214

Improve single schema cache #7214

Conversation

dplewis commented Feb 21, 2021 • edited by TomWFox Loading

New Pull Request Checklist

Issue Description

Approach

TODOs before merging

codecov bot commented Feb 21, 2021 • edited Loading

Codecov Report

SebC99 commented Feb 21, 2021

dplewis commented Feb 21, 2021 • edited Loading

dplewis commented Feb 21, 2021

dplewis commented Feb 22, 2021 • edited Loading

dplewis commented Feb 23, 2021 • edited Loading

mtrezza left a comment • edited Loading

Choose a reason for hiding this comment

dplewis commented Mar 16, 2021

mtrezza commented Mar 16, 2021 • edited Loading

SebC99 commented Mar 16, 2021

mtrezza commented Mar 17, 2021

mtrezza commented Mar 31, 2021 • edited Loading

mtrezza commented Apr 3, 2021

parseplatformorg commented Nov 1, 2021

parseplatformorg commented Mar 14, 2022

dplewis commented Feb 21, 2021 •

edited by TomWFox

Loading

codecov bot commented Feb 21, 2021 •

edited

Loading

dplewis commented Feb 21, 2021 •

edited

Loading

dplewis commented Feb 22, 2021 •

edited

Loading

dplewis commented Feb 23, 2021 •

edited

Loading

mtrezza left a comment •

edited

Loading

mtrezza commented Mar 16, 2021 •

edited

Loading

mtrezza commented Mar 31, 2021 •

edited

Loading