MySQL issue Tutor upgrade from v15.3.7 to v16.0.3 #887
Labels
bug
Bugs will be investigated and fixed as quickly as possible.
Comments
|
Thanks for the detailed information @nratineau, this is super useful.
Is it just this query that causes the container to crash, or did you observe the same behaviour with other queries on different tables? |
|
@regisb I have reconfigured a new test environment with the data. I can query one item without the mysql container crashing. Here is the same query on the database upgrade from v15.3.7 to v16 by first upgrading tutor with mysql v5.7 then upgrading to mysqlv8 afterwards: The Database upgrade from v15.3.7 to v16 by first upgrading tutor with mysql v5.7 then upgrading to mysqlv8 afterwards: Other tables I have tested don't seem to have any issue, only the |
regisb
added a commit
that referenced
this issue
Aug 15, 2023
This fix is for a rather serious issue that affects users who upgrade from Olive to Palm. The client mysql charset and collation was incorrectly set to utf8mb4, while the server stil runs utf8mb3. Only users who run the mysql container are affected. To resolve this issue, we explicitely configure the client to use the utf8mb3 charset/collation. Important note: users who have somehow managed to upgrade from olive to Palm before may find themselves in an undefined state. They might have to fix their mysql data manually. Same thing for users who launched Palm from scratch. Close #887.
regisb
added a commit
that referenced
this issue
Aug 15, 2023
This fix is for a rather serious issue that affects users who upgrade from Olive to Palm. The client mysql charset and collation was incorrectly set to utf8mb4, while the server stil runs utf8mb3. Only users who run the mysql container are affected. To resolve this issue, we explicitely configure the client to use the utf8mb3 charset/collation. Important note: users who have somehow managed to upgrade from olive to Palm before may find themselves in an undefined state. They might have to fix their mysql data manually. Same thing for users who launched Palm from scratch; although, according to my preliinary tests, they should be able to downgrade their connection from utf8mb4 to utf8mb3 without issue. Close #887.
|
Thanks for the details @nratineau. I have discovered that the issue stems from an incorrect charset/collation from the client side. Here's a more complete explanation with a proposed fix: #890 I could definitely use some help on this important issue. Would you mind having a look at the PR and comment there? |
|
@regisb I will try to have a deeper look this week but it might have to wait until next week. I did a quick run with the changes from the PR. Data is still corrupted but can query the whole table without crashing the container: mysql> describe course_overviews_courseoverview; |
regisb
added a commit
that referenced
this issue
Aug 16, 2023
This fix is for a rather serious issue that affects users who upgrade from Olive to Palm. The client mysql charset and collation was incorrectly set to utf8mb4, while the server stil runs utf8mb3. Only users who run the mysql container are affected. To resolve this issue, we explicitely configure the client to use the utf8mb3 charset/collation. Important note: users who have somehow managed to upgrade from olive to Palm before may find themselves in an undefined state. They might have to fix their mysql data manually. Same thing for users who launched Palm from scratch; although, according to my preliinary tests, they should be able to downgrade their connection from utf8mb4 to utf8mb3 without issue. In addition, we upgrade to mysql 8.1.0. Among many other fixes, this avoids a server restart after the upgrade: > An in-place upgrade from MySQL 5.7 to MySQL 8.0, without a server > restart, could result in unexpected errors when executing queries on > tables. This fix eliminates the need to restart the server between the > upgrade and queries. (Bug #35410528) https://dev.mysql.com/doc/relnotes/mysql/8.0/en/news-8-0-34.html See also the 8.1.0 release notes: https://dev.mysql.com/doc/relnotes/mysql/8.1/en/news-8-1-0.html Close #887.
regisb
added a commit
that referenced
this issue
Aug 16, 2023
This fix is for a rather serious issue that affects users who upgrade from Olive to Palm. The client mysql charset and collation was incorrectly set to utf8mb4, while the server stil runs utf8mb3. Only users who run the mysql container are affected. To resolve this issue, we explicitely configure the client to use the utf8mb3 charset/collation. Important note: users who have somehow managed to upgrade from olive to Palm before may find themselves in an undefined state. They might have to fix their mysql data manually. Same thing for users who launched Palm from scratch; although, according to my preliinary tests, they should be able to downgrade their connection from utf8mb4 to utf8mb3 without issue. In addition, we upgrade to mysql 8.1.0. Among many other fixes, this avoids a server restart after the upgrade: > An in-place upgrade from MySQL 5.7 to MySQL 8.0, without a server > restart, could result in unexpected errors when executing queries on > tables. This fix eliminates the need to restart the server between the > upgrade and queries. (Bug #35410528) https://dev.mysql.com/doc/relnotes/mysql/8.0/en/news-8-0-34.html See also the 8.1.0 release notes: https://dev.mysql.com/doc/relnotes/mysql/8.1/en/news-8-1-0.html Close #887.
regisb
added a commit
that referenced
this issue
Aug 16, 2023
This fix is for a rather serious issue that affects users who upgrade from Olive to Palm. The client mysql charset and collation was incorrectly set to utf8mb4, while the server stil runs utf8mb3. Only users who run the mysql container are affected. To resolve this issue, we explicitely configure the client to use the utf8mb3 charset/collation. Important note: users who have somehow managed to upgrade from olive to Palm before may find themselves in an undefined state. They might have to fix their mysql data manually. Same thing for users who launched Palm from scratch; although, according to my preliinary tests, they should be able to downgrade their connection from utf8mb4 to utf8mb3 without issue. In addition, we upgrade to mysql 8.1.0. Among many other fixes, this avoids a server restart after the upgrade: > An in-place upgrade from MySQL 5.7 to MySQL 8.0, without a server > restart, could result in unexpected errors when executing queries on > tables. This fix eliminates the need to restart the server between the > upgrade and queries. (Bug #35410528) https://dev.mysql.com/doc/relnotes/mysql/8.0/en/news-8-0-34.html See also the 8.1.0 release notes: https://dev.mysql.com/doc/relnotes/mysql/8.1/en/news-8-1-0.html Close #887.
github-project-automation
bot
moved this from In Progress
to Done
in Tutor project management
moonesque
pushed a commit
to edSPIRIT/tutor
that referenced
this issue
Nov 20, 2023
This fix is for a rather serious issue that affects users who upgrade from Olive to Palm. The client mysql charset and collation was incorrectly set to utf8mb4, while the server stil runs utf8mb3. Only users who run the mysql container are affected. To resolve this issue, we explicitely configure the client to use the utf8mb3 charset/collation. Important note: users who have somehow managed to upgrade from olive to Palm before may find themselves in an undefined state. They might have to fix their mysql data manually. Same thing for users who launched Palm from scratch; although, according to my preliinary tests, they should be able to downgrade their connection from utf8mb4 to utf8mb3 without issue. In addition, we upgrade to mysql 8.1.0. Among many other fixes, this avoids a server restart after the upgrade: > An in-place upgrade from MySQL 5.7 to MySQL 8.0, without a server > restart, could result in unexpected errors when executing queries on > tables. This fix eliminates the need to restart the server between the > upgrade and queries. (Bug #35410528) https://dev.mysql.com/doc/relnotes/mysql/8.0/en/news-8-0-34.html See also the 8.1.0 release notes: https://dev.mysql.com/doc/relnotes/mysql/8.1/en/news-8-1-0.html Close overhangio#887.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I was able to workaround the issue but wanted to post here if it helps someone with the same issue.
Bug description
Upgrading tutor from v15.3.7 to v16.0.3 changes the MySQL version from 5.7 to 8.0.33.
The upgrade complete successfully, however, after the upgrade when trying to browse the LMS dashboard to list all the courses an error 500 is thrown.
Navigating to the Studio
/admin/course_overviews/courseoverview/, the MySQL container crashes completely with the error:Navigating to the 'about' page of a course throw a 500 as well, the LMS logs have:
Connecting to the MySQL container and running a simple select query against the
course_overviews_courseoverviewtable crashes the container as well:Process used to upgrade (from v15.3.7):
For testing/troubleshooting, I deleted the overview (using the Django admin) for all courses prior to the upgrade which allow to access the Django course overview section afterwards. However, trying the recreate the course details/overview afterward produce the same issue.
I then tried to set MySQL version to 8.0.32 and 8.0.34 instead of 8.0.33 without success.
The only way I was able to complete the upgrade and not run into the issue was to set the MySQL version back to 5.7
Then once the upgrade was completed to v16.0.3, unset the MySQL parameter version to upgrade to v8.0.33
With this process, tutor is now running properly without issue and the MySQL container is not crashing.
It seems the tutor upgrade which runs a number of migrations and also the MySQL upgrade to v8.0.33 is corrupting the course_overviews_courseoverview table causing the issue.
Environment
Amazon EC2 t3.xlarge
OS: Ubuntu 22.04
Docker version: 24.0.5
Docker compose: 2.20.2
Tutor # of courses: 9
tutor config:
Plugin customcert:
Plugin ms365-auth:
I have kept an AWS snapshot on v15.3.7 to help troubleshoot further if needed.
The text was updated successfully, but these errors were encountered: