Add MongoDB 5 compatibility

[mtrezza]

New Feature / Enhancement Checklist

• I am not disclosing a vulnerability.
• I am not just asking a question.
• I have searched through existing issues.

Current Limitation

PR to upgrade to mongodb node driver 4.x currently doesn't pass, needs some refactoring.
See #7586.

❗️ This is a critical upgrade for MongoDB 5 compatibility. All M0/M2/M5 Atlas clusters will automatically be upgraded to MongoDB 5.0 in early February 2022. That means after that, Parse Server will be incompatible with these and all free MongoDB Atlas plans.

A separate question is whether we want / can easily port this back to the 4.x branch. If not, then users of free or small (non-dedicated) MongoDB Atlas plans will be forced to upgrade to Parse Server 5.0. With that comes the necessity to adapt the deployment for the breaking changes in Parse Server 5.0.

Feature / Enhancement Description

Upgrade to the latest mongodb node driver.

Example Use Case

n/a

Alternatives / Workarounds

n/a

[parse-github-assistant]

Thanks for opening this issue!

• 🎉 We are excited about your ideas for improvement!

[mtrezza]

@parse-community/server Anyone wants to take on this critical upgrade?
Given its time-criticality we'll also merge this into beta.

[Moumouls]

Hi @mtrezza

All M0/M2/M5 Atlas clusters will automatically be upgraded to MongoDB 5.0 in early February 2022. That means after that, Parse Server will be incompatible with these and all free MongoDB Atlas plans.

Do you have the Mongo DB announcement link, just to track the direct source of the issue ?

A separate question is whether we want / can easily port this back to the 4.x branch. If not, then users of free or small (non-dedicated) MongoDB Atlas plans will be forced to upgrade to Parse Server 5.0. With that comes the necessity to adapt the deployment for the breaking changes in Parse Server 5.0.

Here Mongo Node JS driver v4 migration guide: https://github.com/mongodb/node-mongodb-native/blob/4.0/docs/CHANGES_4.0.0.md

From my point of view and knowledge of parse-server, here the list of BK changes that can affect Parse:

• https://github.com/mongodb/node-mongodb-native/blob/4.0/docs/CHANGES_4.0.0.md#db-no-longer-emits-events (fixed in refactor: minor changes to remove mongo deprecation warnings #7626, but here in back port use case we will need to also backport my PR about db warnings)
• Grid FS: https://github.com/mongodb/node-mongodb-native/blob/4.0/docs/CHANGES_4.0.0.md#gridstore-removed
• replace returnOriginal with returnDocument option (already fixed by refactor: minor changes to remove mongo deprecation warnings #7626)
• update find and modify interfaces for 4.0
• remove top-level write concern options

As we saw in https://github.com/parse-community/parse-server/pull/7626/files, since Parse Mongo driver is globally well written, i think it's feasible to backport the mongo v5 PR to parse-server 4 without too many effort.

Here i'm also sorry i can't tell if i could work on this asap.

[mtrezza]

Do you have the Mongo DB announcement link, just to track the direct source of the issue ?

It's standard MongoDB Atlas procedure for free and shared clusters. These are always kept on the latest version and users have no control over the MongoDB version. We are actually late on upgrading because the 4.x driver has been released back in July 2021 already.

It's interesting though that we are currently running CI tests with MongoDB 5.0.3 but according to the compatibility table, MongoDB 5 requires the 4.x driver. Maybe this will be an Atlas specific restriction, or Parse Server is currently just not using or not testing features that are incompatible.

@dariakp do you have any insight to what extend the 3.x driver works with MongoDB 5 or should we open a ticket?

[dariakp]

@mtrezza I'm not sure I understand the question. The same features that work in the 3.x driver against pre-5.0 MongoDB servers should work against 5.0 MongoDB servers. However, the 3.x driver will not necessarily support the new features available in MongoDB 5.0. To get the new features, the 4.x driver should be used.

[mtrezza]

Thanks @dariakp that was my question. The compatibility table wasn't clear to me because it says that the 3.x driver is incompatible with MongoDB 5.0 but it doesn't specify what that means. That confirms our priority on this upgrade.

[Moumouls]

@dariakp here can you confirm that parse server instances using the Mongo v3 driver will work with a Mongo DB v5.

How many time MongoDB v5 will support v3 driver ?
Following the recent mongo blog post about new version release calendar, a major MongoDB version each year, does MongoDB v6 in 2023 will drop the support of mongo driver v3 ?

If v3 driver is fully compatible with MongoDB v5, this issue is not as critical as it seems, and parse servers using mongo atlas will just work normally.

The #7586 could be developed/fixed later.

[mtrezza]

@Moumouls I think your questions have been answered already. This PR stays a priority for us, because we are already too far behind with the driver. Let's please keep this thread focused on the driver upgrade to 4.x. If you or someone else could reserve some time for it, that would be great 🙂

[Moumouls]

@Moumouls I think your questions have been answered already

Yes, but i think it could help us to start to prevent the community if the v3 driver will be not supported on MongoDB v6 (should be released in 2023 following the new mongo release rhythm). So developers on atlas will "only" have 1 year to upgrade there parse server before the old v3 driver will be dropped. @mtrezza

[mtrezza]

1 year to upgrade should be more than enough, we also don't give more time when we deprecate features. There are many other reasons why developers should upgrade to Parse Server 5.0. We will begin an upgrade campaign across the community once Parse Server 5.0 is released in Jan 2022 to facilitate that.

We cannot say that Parse Server is compatible with MongoDB 5.0 if we don't know whether it supports all of the 5.0 features that are customizable via Parse Server, think aggregation pipeline or settings that Parse Server does not validate before being sent to the DB. In that sense, the current MongoDB 5.0 compatibility indicated in the Parse Server README may be incorrect. Upgrading the driver will give us assurance.

[jonas-db]

I just received an email about this:

Your M0/M2/M5 Atlas cluster(s) will automatically be upgraded to MongoDB 5.0 in mid February 2022.

What is the status of this ticket? Despite its importance, it seems almost nothing has been done in the past 2 months.
Can someone of the head contributors make a clear checklist of what needs to be done? Also which versions of 4.x (only 4.1.1?) need to be supported?

[mtrezza]

1. Update MongoDB driver dependency to the latest version
2. Open PR and see which tests fail in the CI
3. Make changes to Parse Server to make tests pass

[Moumouls]

@jonas-db i sent a PR to upgrade mongo driver to v4. But if you run on a recent parse server version like > 4.5.X driver v3 should works with mongo 5 in all major use cases.

Here results of 4.5.X on Mongo 5

2683 specs, 6 failures, 71 pending specs

Failures

1) OAuth Should fail a GET request
  Message:
    Uncaught exception: SyntaxError: Unexpected end of JSON input
  Stack:
        at <Jasmine>
        at IncomingMessage.<anonymous> (/Users/antoinecormouls/Desktop/OpenSource/parse-server/lib/Adapters/Auth/OAuth1Client.js:31:21)
        at IncomingMessage.emit (events.js:327:22)
        at IncomingMessage.EventEmitter.emit (domain.js:486:12)
        at endReadableNT (_stream_readable.js:1327:12)
        at processTicksAndRejections (internal/process/task_queues.js:80:21)

2) Parse.Query hint query aggregate with hint string
  Message:
    Expected 'PROJECTION_SIMPLE' to be 'COLLSCAN'.
  Stack:
    Error: Expected 'PROJECTION_SIMPLE' to be 'COLLSCAN'.
        at <Jasmine>
        at UserContext.<anonymous> (/Users/antoinecormouls/Desktop/OpenSource/parse-server/spec/ParseQuery.hint.spec.js:66:44)
        at <Jasmine>
        at processTicksAndRejections (internal/process/task_queues.js:93:5)
  Message:
    Expected 'PROJECTION_SIMPLE' to be 'FETCH'.
  Stack:
    Error: Expected 'PROJECTION_SIMPLE' to be 'FETCH'.
        at <Jasmine>
        at UserContext.<anonymous> (/Users/antoinecormouls/Desktop/OpenSource/parse-server/spec/ParseQuery.hint.spec.js:73:44)
        at <Jasmine>
        at processTicksAndRejections (internal/process/task_queues.js:93:5)
  Message:
    Expected undefined to be '_id_'.
  Stack:
    Error: Expected undefined to be '_id_'.
        at <Jasmine>
        at UserContext.<anonymous> (/Users/antoinecormouls/Desktop/OpenSource/parse-server/spec/ParseQuery.hint.spec.js:74:59)
        at <Jasmine>
        at processTicksAndRejections (internal/process/task_queues.js:93:5)

3) Parse.Query hint query aggregate with hint object
  Message:
    Expected 'PROJECTION_SIMPLE' to be 'COLLSCAN'.
  Stack:
    Error: Expected 'PROJECTION_SIMPLE' to be 'COLLSCAN'.
        at <Jasmine>
        at UserContext.<anonymous> (/Users/antoinecormouls/Desktop/OpenSource/parse-server/spec/ParseQuery.hint.spec.js:86:44)
        at <Jasmine>
        at processTicksAndRejections (internal/process/task_queues.js:93:5)
  Message:
    Expected 'PROJECTION_SIMPLE' to be 'FETCH'.
  Stack:
    Error: Expected 'PROJECTION_SIMPLE' to be 'FETCH'.
        at <Jasmine>
        at UserContext.<anonymous> (/Users/antoinecormouls/Desktop/OpenSource/parse-server/spec/ParseQuery.hint.spec.js:93:44)
        at <Jasmine>
        at processTicksAndRejections (internal/process/task_queues.js:93:5)
  Message:
    Expected undefined to equal Object({ _id: 1 }).
  Stack:
    Error: Expected undefined to equal Object({ _id: 1 }).
        at <Jasmine>
        at UserContext.<anonymous> (/Users/antoinecormouls/Desktop/OpenSource/parse-server/spec/ParseQuery.hint.spec.js:94:60)
        at <Jasmine>
        at processTicksAndRejections (internal/process/task_queues.js:93:5)

4) Parse.Query hint query aggregate with hint (rest)
  Message:
    Expected 'PROJECTION_SIMPLE' to be 'COLLSCAN'.
  Stack:
    Error: Expected 'PROJECTION_SIMPLE' to be 'COLLSCAN'.
        at <Jasmine>
        at UserContext.<anonymous> (/Users/antoinecormouls/Desktop/OpenSource/parse-server/spec/ParseQuery.hint.spec.js:134:44)
        at <Jasmine>
        at processTicksAndRejections (internal/process/task_queues.js:93:5)
  Message:
    Expected undefined to equal Object({ _id: 1 }).
  Stack:
    Error: Expected undefined to equal Object({ _id: 1 }).
        at <Jasmine>
        at UserContext.<anonymous> (/Users/antoinecormouls/Desktop/OpenSource/parse-server/spec/ParseQuery.hint.spec.js:146:60)
        at <Jasmine>
        at processTicksAndRejections (internal/process/task_queues.js:93:5)

5) Parse.Query testing Use a regex that requires all modifiers
  Message:
    Failed: {"message":{"ok":0,"code":51091,"codeName":"Location51091","name":"MongoError"},"code":1}
  Stack:
    Error: Failed: {"message":{"ok":0,"code":51091,"codeName":"Location51091","name":"MongoError"},"code":1}
        at <Jasmine>
        at global.jfail (/Users/antoinecormouls/Desktop/OpenSource/parse-server/spec/helper.js:393:3)
        at /Users/antoinecormouls/Desktop/OpenSource/parse-server/spec/ParseQuery.spec.js:2062:11
        at <Jasmine>
        at processTicksAndRejections (internal/process/task_queues.js:93:5)

6) Parse.Query testing select keys query
  Message:
    Expected false to be truthy 'other error: {"message":{"ok":0,"code":40352,"codeName":"Location40352","name":"MongoError"},"code":1}'.
  Stack:
    Error: Expected false to be truthy 'other error: {"message":{"ok":0,"code":40352,"codeName":"Location40352","name":"MongoError"},"code":1}'.
        at <Jasmine>
        at ok (/Users/antoinecormouls/Desktop/OpenSource/parse-server/spec/helper.js:265:16)
        at /Users/antoinecormouls/Desktop/OpenSource/parse-server/spec/ParseQuery.spec.js:3207:11
        at <Jasmine>

@jonas-db i can suggest to test asap your (staging/pre prod) parse app with a mongo 5 (locally or atlas) then you will be able to verify if you are safe regarding your usage of parse 🙂

[mtrezza]

Closing via #7794; great work @Moumouls!