-
-
Notifications
You must be signed in to change notification settings - Fork 4.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Bug]: occ db:convert-type
broken since PrimaryReadReplicaConnection
support added
#45257
Comments
Logging the Parameter FromDB
ToDB
It is notable that in the second case (which should be the postgres-database configured via the CLI arguments), the host, dbname and port in the top-level array are set correctly from the CLI, Both the |
Possibly related: #45097 I don't have time to look tonight, but your analysis seems sound (and, given the added evidence from the log output, even more so). Cc: @juliushaertl |
I can verify but can't contribute logs. I used the --clear-schema option in my attempt and my original MariaDB DB got deleted. Good thing I have backups. |
I modified https://github.com/nextcloud/server/blob/master/core/Command/Db/ConvertType.php#L247
and seems all converted Also have to remove mysql.utf8mb4 parameter from config.php to perform conversion to remove client_encoding from params of DbFactory |
Got hit by this a week ago. Is that particular Nextcloud instance completely screwed? |
Yes, unfortunately. 😢 The end result is the parameters get swapped so it likely dropped the source database. Fortunately, recovery should be a simple matter of restoring from backup (just the db portion): https://docs.nextcloud.com/server/latest/admin_manual/maintenance/restore.html#restore-database (or whatever other mechanism you utilize in your environment for database backup/recovery) |
@Delagen Mind pushing that as PR? I know there may be additional changes needed, but it'll start things in motion. |
Needed until #45257 is addressed to prevent data loss Signed-off-by: Josh <josh.t.richards@gmail.com>
@joshtrichards I don't mind that my changes solve the problem correctly. It only makes possible for me successful conversion from MySQL to Postgres |
I have just experienced this problem on a new installation. |
Needed until #45257 is addressed to prevent data loss Signed-off-by: Josh <josh.t.richards@gmail.com>
Would not be it the simplest solution, if the ConnectionFactory::getConnection() method would first check if the |
Needed until #45257 is addressed to prevent data loss Signed-off-by: Josh <josh.t.richards@gmail.com>
Hi, is this #45013 fixing the problem? |
I continue to have issues w/ MariaDB/MySQL breaking and corrupting itself specifically just w/ the nextcloud database, I'd really like to move to Postgres but at this point I think I may just abandon nextcloud alltogether -- I've been waiting for 4 months for this to be corrected and it keeps being pushed back. I realized the developers of Nextcloud owe me absolutely nothing, but this has still been a frustrating experience. |
It'll be a bug fix. Any fix will be backported to whatever earlier majors need it. It can't have a v30 milestone since it's already released. Development occurs against the
I'm sorry it has been a frustrating experience. I'm hopeful someone more familiar that area of code than me (or with more time on their hands to get familiar with it) will take a closer took soon. If you are a customer of Nextcloud GmbH, you can tell your contacts there through the appropriate channels that this Issue is a priority for you as a customer and ask them to fix it on your behalf. Otherwise you're at the mercy of whenever someone gets around to it in the project. @CorneliousJD I'm curious about your corruption situation, but this isn't the place to follow-up on it. Have you posted about your corruption issues somewhere (such as on the community help forum) where I can take a look and gain some further context? |
occ db:convert-type
broken since PrimaryReadReplicaConnection
support added
I have not posted elsewhere about it in great detail, but it will randomly have corrupt tables and it only happens within the nextcloud database. I had previously run w/ postgres w/out these issues so I wanted to revert back to postgres and see how things go before abandoning Nextcloud, but setting up a whole new instance w/out doing a db convert just isn't an option for me right now. Before I dive too deep into it I am just waiting for the Postgres conversion to be back in the cards so I Can test that and see how it goes with that setup. |
For me the combined solutions of @Queuecumber, @normen and @lightonflux worked, and I was able to convert my mariadb to postgresql. So combined it would look like this: In - $connectionParams = $this->createConnectionParams('', $additionalConnectionParams);
+ $connectionParams = $this->createConnectionParams('', $additionalConnectionParams, $type); - public function createConnectionParams(string $configPrefix = '', array $additionalConnectionParams = []) {
- $type = $this->config->getValue('dbtype', 'sqlite');
+ public function createConnectionParams(string $configPrefix = '', array $additionalConnectionParams = [], $type = '') {
+ if ($type === '') {
+ $type = $this->config->getValue('dbtype', 'sqlite');
+ } + if ($type === '') {
+ $type = $this->config->getValue('dbtype', 'sqlite');
+ }
- if ($this->config->getValue('mysql.utf8mb4', false)) {
+ if ($this->config->getValue('mysql.utf8mb4', false) && $type==="mysql") { In - $connectionParams = $this->connectionFactory->createConnectionParams();
+ $connectionParams = $this->connectionFactory->createConnectionParams('', [], $type); Maybe someone else could also check this? |
@Delogon It worked for me (v30). Finally I was able to migrate to pgsql. At the first glance, all seems to be OK. Thanks! |
@Delogon this is the way, worked great sqlite -> psql. Cheers and good luck everyone. |
@Delogon Thanks for the summary! Looks good on my end. Migrated from sqlite to mysql (mariadb Ver 15.1 Distrib 10.6.18-MariaDB) with the latest docker image (Nextcloud 30.0.0) |
@Delogon I'm getting an error message:
I'm not sure if relevant, but this table is empty for me.
removing the group folder plugin solves the issue... however, now the group folders are gone, but this I can survive. |
@Delogon Since your fix seems to work, do you want to make a pull request? If not, I might be able to do it. |
So I am also trying to move from mariadb to postgres and on nextcloud v30. I did the fix that @Delogon posted and that got me further. However i end up with this during the conversion.
Any ideas? |
Found this related bug and it seems to be the same. Looks like NC has a bit more work to do on the postgres side |
@hagak what is your encoding in mysql? |
UTF8 the issue from what i gather in that other related bug that i apparently forgot to put the link to. Shows that it is due to postgres schema using a TEXT column instead of a Bytea . I believe there is a PR to resolve that as well Particular data in the oc_properties table with a valuetype=3. I only had one entry that was causing this so I just manually deleted that row and the convert finished. Add the link to the issue number #37754 |
I created the pull request #48666 but tbh I don't know if I did that correct, since creation nothing has happened. @joshtrichards or @juliusknorr maybe you could tell me what to do, to get it merged. |
I want to migrate from a standalone nginx/php-fpm/mariadb Nextcloud installation to AIO "The official Nextcloud installation method" but I'm blocked by this issue since several weeks. Is the patch provided by @Delogon here #45257 (comment) or in the PR #48666 safe to use? |
I cannot answer that but even if it is you should make a database backup anyway |
Of course. I'm just worried to use it, don't detect any problem, and start using my new postgres DB without detecting a possible data loss (or other problem) :-/ |
I just applied the PR [#48666] manually on nextcloud 29.0.9-fpm-alpine and successfully converted from mysql to postgres. So far everyhting looks good. |
I'll be, probably, trying to migrate this evening. I'll report how it goes. |
That bug is already known. A potential solution would be to disable the group folder plugin, but I do not know how suitable that is for you.
|
That crossed my mind but Nextcloud kinda doesn't want to do it. I don't use the plugin but it was disabled but installed. While writing this I managed to removed it with EDIT: God Talk is broken now... |
Ok. The database migration itself went smoothly after disabling group folders. Importing data into AIO is the part that was full of pain. It had issues with the database dump, then the data files, then 3 apps that I had installed, ... But it seems to be done now. MAKE BACKUPS and keep them! Even after you migrate and everything seems well you might find something broken. I, for example, had all my calendar entries missing and I had to redump the PostgreSQL database, change the data directory within it and the owner of the database and then re-import it again to get it back. |
Bug description
The OCC-Command
db:convert-type
seems to ignore the DB properties (type, user, host, database, port, password) provided through the CLI and instead operates on the database configured inconfig.php
I believe this may be related to the commit 79c4986 as this reworked the parameter handling in
ConnectionFactory.php
.If I read this correctly, the default parameters are now always generated based on the system config instead of the passed arguments.
Running
db:convert-type --clean-schema
drops the tables in the "old" database.Steps to reproduce
php occ db:convert-type --password=<PASSWORD> --port=5432 pgsql nextcloud_app postgres nextcloud
(note: the hostpostgres
is a Kubernetes service name)oc_accounts
, I assume because it tries to migrate the content of the old database into itself.Expected behavior
db:convert-type
Installation method
Community Docker image
Nextcloud Server version
29
Operating system
Debian/Ubuntu
PHP engine version
None
Web server
None
Database engine version
MySQL
Is this bug present after an update or on a fresh install?
None
Are you using the Nextcloud Server Encryption module?
None
What user-backends are you using?
Configuration report
List of activated Apps
Nextcloud Signing status
Nextcloud Logs
No response
Additional info
nextcloud:29.0.0
mysql:8.4.0
postgres:16.3
The text was updated successfully, but these errors were encountered: