Update OC 7.0.10 to OC 8.0.8 with encryption app very slow

[hostingnuggets]

I am currently testing the upgrade of OC 7.0.10 to 8.0.8 using occ upgrade but somehow it lasts for hours and hours, now I checked what the occ upgrade PHP process is doing and it looks like its doing stat of every possible key files as you can see here an example:

stat("/mnt/owncloud/username/files_encryption/share-keys/somefile.shareKey", {st_mode=S_IFREG|0644, st_size=512, ...}) = 0

The output of the upgrade itself shows:

Turned on maintenance mode
Checked database schema update
Checked database schema update for apps
Updated database
Disabled 3rd-party app: admin_dependencies_chk
Disabled 3rd-party app: calendar
Disabled 3rd-party app: contacts
Disabled 3rd-party app: documents
Disabled 3rd-party app: search_lucene
Updated <files_pdfviewer> to 0.7
Updated <activity> to 1.2.2

So yes I am using the encryption app and have around 66 GB of data. So my question here is that behavior normal? and why does the upgrade need to stat every files? I didn't even get to the stage of running occ encryption:migrate yet...

[karlitschek]

@schiesbn

[hostingnuggets]

FYI: @DeepDiver1975 mentions that this very same issue (#14012) will be fixed in 8.0.3 but it looks like this fix hasn't made it to 8.0.8.

[hostingnuggets]

any news? I would really like to upgrade to OC 8...

[hostingnuggets]

@karlitschek could you raise the priority of this issue? this is a critical blocker as anyone who is using encryption with OC 7 will not be able to upgrade to OC 8. I let the upgrade run for 7 hours on "only" 60 GB of data and it still wasn't finished.

[PVince81]

@hostingnuggets the fix mentioned above was for reducing the database calls which made it very slow. Strange thing that in your case the stat calls still makes it slow.

@schiesbn @icewind1991 any way to accelerate this further ?

[icewind1991]

Is the owncloud data dir on a remote filesystem? (smb/nfs/etc)

[hostingnuggets]

@icewind1991 we have the data directory on an a dedicated Linux NFS file server on the same LAN (1 Gigabit) as the owncloud server.

[chrsch]

I'm planning to move some big oc 7 instances to 8 - some have 100 - 200 gb encrypted data and a lot of files (80-100k). Now with this ticket and the other one referenced from here I have no idea how to upgrade those instances to oc 8 with reasonable down time (< 1d). So please raise the priority of this issue @karlitschek as also proposed by @hostingnuggets, so the migration is possible for bigger oc 7 instances with encrypted data. Please let me know if can contribute. Thanks a lot!

[schiessle]

The fix mentioned by @DeepDiver1975 is also in 8.0.8: https://github.com/owncloud/core/blob/v8.0.8/apps/files_encryption/lib/migration.php#L38

Basically we disable the file cache update and update it only once at the end instead of updating the cache for every key we move. This improves the performance significantly. Beside that I fear that there is not much we can improve. In order to fix a potential data lose because of non unique file key names we had to change the way we store the file keys and move all file keys to the new structure. So the performance is limited by the hard disc (+ network) performance to rename a significant number of files/folders.

[schiessle]

Do you have the files locking app enabled? You could try to disable it before you start the migration. This could improve the performance as well. Would be great if you could give me some feedback if this was useful, then we can add it to the migration script.

[chrsch]

Thanks for your suggestion, File Locking is not involved here.

[PVince81]

The slowness was fixed already, closing.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.