-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[DEPR]: DraftModuleStore (Old Mongo Modulestore) #62
Comments
An interesting implementation point came up in a forum discussion related to deleting courses (and how poorly supported that is). The long term architectural goal is for CourseOverviews (or something like it) is to represent course metadata independent of the content storage represented by ModuleStore. In other words, things like enrollments, student state, certificates, etc. should continue to work, even if there is no data in the ModuleStore for that course content. Right now, that is not the case. CourseOverviews act almost as materialized caches for CourseBlocks, and need to reach back into the root CourseBlock when regenerating when there is a version bump to CourseOverview fields. There are certain other fields where CourseOverview isn't even a cache, and always delegates back to the CourseBlock it gets from ModuleStore. For a while, I assumed that transition would happen as part of this DEPR effort–because this DEPR means that we'd be removing Old Mongo as a storage mechanism. But I've come around to the idea that we could do this DEPR with less risk by lobotomizing Old Mongo, instead of completely killing it. We could leave just enough of it behind to find a course ( It would also potentially let us explore other options for how we want to separate catalog-level metadata from course content data. CourseOverview is the most readily available mechanism, but it has its own issues around regeneration, overrides, locking, data dependency, etc. Decoupling that longer term question from this DEPR may give us more space to design a robust solution. |
update: we plan to merge changes to shut off LMS access to old mongo with this PR on Apr 4, 2022 prior to the Nutmeg release branch being cut |
That PR landed - learner access to old mongo courses should be removed now. Hopefully opening the floodgates to further old mongo deprecation. |
@mikix thinking about the release notes for Nutmeg, if people do have old-mongo courses, they'll disappear when the start running Nutmeg is that right? And if they want to recover those courses, they'll have to export them pre-nutmeg and load them into new split courses IDs. The student data won't carry over so probably they should close the course and start funneling new students to a newer coures if they're in this situation? |
@feanil if they still have runs from 2015 in active use and have been ignoring the banner in several releases of Studio saying we would remove all support in March 1, 2021, then I still have good news! Studio support remains for now. They should be able to still export their course or rerun it. My change just affects learner access. The long term plan is to remove Studio access and the data, but I think that approach is in Ormsbee’s hands now. |
Next steps:
Again, the goal is to get it down to a point where we can delete most of the tests and functionality but leave just a little bit on the edges so that things wanting to read the root course block and its settings can do so. |
@feanil: After Raccoon Gang gets through their current slate of DEPR work, could we discuss adding Old Mongo in a follow-on phase? Is there a place where we're collecting potential work like that? |
@e0d this was not on the initial list, could it be added? |
Weeks worth of work. I can take a stab at the first set of tickets, but there will likely be edge cases that I don't remember, which will necessitate further tickets. How detailed should I get on specifying those? |
I assume there's no way for the team to do an accurate estimate with what's
in the ticket to date? If so let's capture the known work I tickets and
create some buffer for edge cases.
Ed
…On Thu, May 12, 2022, 6:45 PM David Ormsbee ***@***.***> wrote:
Weeks worth of work. I can take a stab at the first set of tickets, but
there will likely be edge cases that I don't remember, which will
necessitate further tickets. How detailed should I get on specifying those?
—
Reply to this email directly, view it on GitHub
<#62 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAJWEASVS62TELEEWAWMWS3VJWCXHANCNFSM5QNSNTXA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
This removes user-facing Studio edit support for Old Mongo courses (courses that have a CourseKey of the format {org}/{course}/{run}). This does not affect our normal courses, which have CourseKeys starting with "course-v1:". After this commit: * Old Mongo courses will continue to appear on the Studio course listing page, but are not clickable. * Any attempt to directly access an Old Mongo course in Studio via URL fail with a 404 error. * Course certificates will still be available for Old Mongo courses. * Old Mongo courses will continue to be returned by CourseOverviews and get_course_summaries() calls. We decided against removing Old Mongo courses from the listing entirely because that would require very expensive CourseOverviews query to filter them out. Making that query more efficient would involve a database migration to add appropriate indexing, which is something else that we are looking to avoid. CourseOverviews are used everywhere in the system, so we want to avoid changing how they work so that we can minimize risk. This is part of the Old Mongo Modulestore deprecation effort: openedx/public-engineering#62
This commit removes code that was used to copy Old Mongo courses into new Split Mongo courses. This includes both the migrate_to_split management command, as well as the backend code that would be invoked to re-run Old Mongo courses as Split courses using Studio (the UI for this was already removed in b429e55). This is a part of the Old Mongo removal effort tracked in: openedx/public-engineering#62
This is currently being handled as a funded contribution project at tCRIL. |
I am marking this work as complete with the merge of openedx/edx-platform#31134. The bulk of Old Mongo code and tests is now gone. There are three parts that will be left for future cleanup, for various reasons: Course Static AssetsReading static assets is still supported, so that we don't break links to certificate-related assets that are stored in contentstore. The long term fix for this is to migrate those files into credentials. Read access for the root CourseBlockThe root CourseBlock is used in various places to get high level metadata about a course (name, settings, etc.), and is also used when re-building CourseOverviews. See this comment for more details. MixedModuleStore/DraftModuleStoreBecause we still have some access to the old ModuleStore, we need to maintain the MixedModuleStore facade. |
@ormsbee could you write up the release notes for Quince on this ticket? https://openedx.atlassian.net/wiki/spaces/COMM/pages/3726802953 |
@dianakhuang: Done. |
CURRENT STATUS (as of the QUINCE release)
Most functionality has been removed from the long-deprecated Old Mongo Modulestore. This was the old system for storing and accessing course content, used by courses with Org/Course/Run style identifiers, e.g. MITx/6.002x/2012_Fall. All end-user access to this course content was removed with the Nutmeg release, so any course actively being run in the Nutmeg release or later should be unaffected.
To preserve compatibility with old installations, some limited Old Mongo functionality remains:
More details below:
Original Ticket
The DraftModuleStore (also sometimes referred to as "Old Mongo") is the interface used to store courseware content for courses with course run keys of the format "Org/Course/Run", e.g. "edX/Demo/2012". Newer courses of the format "course-v1:Org+Course+Run" use the DraftVersioningModuleStore (also commonly called "Split Mongo"). This has been available since the Birch release and the vast majority of sites have never used DraftModuleStore. One of the main motivations for the newer modulestore was atomic publishes, as it was possible to get partially applied updates if there was an error during the import process with DraftModuleStore.
Because DraftModuleStore has a very different data structure from DraftVersioningModuleStore, supporting both formats simultaneously has resulted in significant complexity, bugs, and performance issues going as far back as the initial implementation of cohorts. There are also thousands of extra tests run specifically to ensure compatibility across ModuleStores.
Some sites intentionally continued to use DraftModuleStore because of storage-related concerns. DraftVersioningModuleStore did not free up disk space used by old versions of the content. However, this problem has been addressed with tubular's structures.py script, which can be run on a regular basis to prune unused old versions of course content.
Proposal
Note: In previous conversations, I had discussed the possibility of having a data migration that would convert DraftModuleStore course content into DraftVersioningModuleStore courses while preserving IDs. I created a proof of concept for this approach as a hackathon project. This has the upside of letting us get rid of a chunk of the old code without giving up compatibility, but it also had a number of strong drawbacks, including:
Because usage of the old modulestore outside of edX itself is limited and we did not want to introduce any more complexity to what is already a major source of bugs in edx-platform, this DEPR is going the simpler route of removing support altogether.
Compatibility notes
Additional Info
Original Jira Issue: https://openedx.atlassian.net/browse/DEPR-58
Useful comments
From Mike Terry
Approach
In order to preserve CourseOverview metadata and further reduce risk, the end state is not a complete removal of Old Mongo, but instead a really limited version that only implements
has_course
and the ability to read from the root CourseBlock.Implementation Tickets
The text was updated successfully, but these errors were encountered: