Migrate ModuleStore off of MongoDB

[e0d]

TLDR;

Mongo DB introduces excessive hosting and maintenance cost for the value that we derive from in on the platform. With improvements to serializing courses in the RDBMS, it's even less valuable. Removing it will let us reduce cost and maintenance burden, scale down better, and thereby scale platform adoption.

The full details of this are here.

Related PRs:

• doc: ADR for removing MongoDB usage. edx-platform#29098
• Store split modulestore's course indexes in Django/MySQL edx-platform#27565
• Store split modulestore's course indexes in Django/MySQL (Take 2) edx-platform#29058
• Add a data migration to copy all course index data into MySQL edx-platform#29144
• Add a data migration to copy all course index data into MySQL (Take 2) edx-platform#29293

[ormsbee]

Worth noting that this work cannot be fully completed until Old Mongo support is removed (DEPR-58).

[ormsbee]

Posted this on the forums:

To give a little more context, there are a few independent pieces of deprecation that need to happen before MongoDB can be removed:

Old Mongo Removal

Remove Old Mongo support entirely. @mikix done great work recently in cutting off access to Old Mongo courses:

• test: switch default test store to the split store edx-platform#29855
• test: more Old Mongo removal from tests edx-platform#29911
• feat: hide Old Mongo courses from catalog edx-platform#29945
• feat: mark Old Mongo courses as invitation-only edx-platform#29987
• feat!: remove all access to Old Mongo courses edx-platform#29848

What remains is a lot of code deletion and test fixing. This is an area where people can contribute with relatively little ramp-up, since it's mostly deleting test permutations. Please comment here if you're interested in that work.

Convert the Split Modulestore to use django-storages

This will require three parallelizable streams of work:

1. Converting how we store Structure documents (the skeleton outline of a course) from MongoDB to django-storages, and migration scripts. This may also require some porting of the structures.py cleanup script, depending on the projected costs.
2. Converting how we store Definition documents (the content of each block) from MongoDB to django-storages, and migration scripts. Because of access patterns and the variability of latency with object stores like S3, this would likely require an improved caching layer as well.
3. Converting how we store store course static assets (e.g. images, PDFs) to django-storages. We have to take care here to (a) not break CDN caching for certain assets; and (b) not break security restrictions for enrollment-restricted assets. The way we ship over static assets right now is implemented as middleware and is frankly kinda wacky, so there will likely be more cleanup here than appears at first glance.

Remove MongoDB usage from Forums

Once the two sections above this are finished, it's possible to have a basic install of Open edX without MongoDB. The last piece that I know of that actively uses MongoDB is the forums experience. I don't know what the current plans for deprecation of this usage is. The last I recall talking with anyone about it, the general idea was that we wanted to switch away from MongoDB and towards the Django ORM, but only after removing the Ruby code. But again, I'm not sure where that stands now.

At the very least, if the other sections are completed, MongoDB can be a dependency of only the forums, and not Open edX as a whole.

[regisb]

Can I ask what is the status of this issue? If I understand correctly, step 2 "Convert the Split Modulestore to use django-storages" is not yet completed, right?