cms-job taking too long to finish

[vunguyen-dmt]

My instance has about 10k courses, every time I run "tutor k8s launch" the process gets stuck at "job.batch/cms-job-20250207040323 created" for hours. I checked logs and meilisearch was indexing courses, eventually it went through this step. It didn't happen with elasticsearch before. While this is not a bug, I think definitely a serious problem for larger instances, I created an issue here so we can improve it.

[regisb]

Can you identify which part of the CMS initialization is taking too long? Is it ./manage.py cms reindex_studio --experimental --init or ./manage.py cms reindex_course --active? (ref: https://github.com/overhangio/tutor/blob/release/tutor/templates/jobs/init/cms.sh)

[vunguyen-dmt]

@regisb I'm not sure which one it was, but the log outputs start with [search.meilisearch] [user None] [ip None] meilisearch.py:144 - Index request: index=tutor_course_info sources=
And it ended up with Killed the last time I launch.

[regisb]

Then it is ./manage.py cms reindex_course --active. This comment has come from the past to bite us: #1174 (review)

Basically, we are asking the CMS to reindex all active courses. This change was introduced in Sumac. We should probably remove that line from cms.sh...