Problem with upgrading files_encryption from OC 7 => OC 8

[tomcur]

Steps to reproduce

1. Have a working installation of owncloud version 7.0.4.2 with files_encryption enabled.
2. Attempt to upgrade to owncloud 8.0.0.7.

Expected behaviour

The upgrade should succeed.

Actual behaviour

The upgrade hangs. After disabling maintenance mode and trying again, it still hangs.

After disabling files_encryption through the occ console, the upgrade succeeded. However, encrypted files are not decrypted (and owncloud does not prompt users to decrypt their files). After re-enabling files_encryption, owncloud shows the upgrade-screen again. Upgrading still fails.

Server configuration

Operating system: Ubuntu 12.04

Web server: Apache 2.4.9

Database: MySQL

PHP version: 5.5.12

ownCloud version: 7.0.4.2=>8.0.0.7

Updated from an older ownCloud or fresh install: During upgrading

List of activated apps:

• files
• files_encryption
• files_external
• files_pdfviewer
• files_sharing
• files_texteditor
• files_trashbin
• files_versions
• files_videoviewer
• firstrunwizard
• gallery
• templateeditor
• updater

The content of config/config.php:

<?php
$CONFIG = array (
  'instanceid' => 'oc2197ab4a5f',
  'trusted_domains' =>
  array (
    0 => 'kepow.org',
  ),
  'datadirectory' => '/www/kepow/owncloud_data',
  'dbtype' => 'mysql',
  'version' => '8.0.0.7',
  'dbname' => 'owncloud',
  'dbhost' => 'localhost',
  'dbtableprefix' => 'oc_',
  'dbuser' => 'oc_Thomas',
  'installed' => true,
  'forcessl' => true,
  'mail_smtpmode' => 'php',
  'mail_smtpname' => 'thomas',
  'maintenance' => true,
  'theme' => '',
);

Are you using external storage, if yes which one: No.

Are you using encryption: Yes.

Client configuration

Browser: Issue occurs on firefox and chrome

Operating system: Windows

Logs

Web server error log

No errors are logged during the upgrade process.

ownCloud log (data/owncloud.log)

First try:

{"reqId":"230bd67050902ddeb36c9af31787d650","remoteAddr":"84.85.215.83","app":"PHP","message":"Cannot modify header information - headers already sent at \/www\/kepow\/owncloud\/lib\/private\/user\/session.php#282","level":3,"time":"2015-02-09T20:51:06+00:00"}
{"reqId":"230bd67050902ddeb36c9af31787d650","remoteAddr":"84.85.215.83","app":"PHP","message":"Cannot modify header information - headers already sent at \/www\/kepow\/owncloud\/lib\/private\/user\/session.php#283","level":3,"time":"2015-02-09T20:51:06+00:00"}
{"reqId":"230bd67050902ddeb36c9af31787d650","remoteAddr":"84.85.215.83","app":"PHP","message":"Cannot modify header information - headers already sent at \/www\/kepow\/owncloud\/lib\/private\/user\/session.php#284","level":3,"time":"2015-02-09T20:51:06+00:00"}
{"reqId":"230bd67050902ddeb36c9af31787d650","remoteAddr":"84.85.215.83","app":"PHP","message":"Cannot modify header information - headers already sent at \/www\/kepow\/owncloud\/lib\/private\/user\/session.php#287","level":3,"time":"2015-02-09T20:51:06+00:00"}
{"reqId":"230bd67050902ddeb36c9af31787d650","remoteAddr":"84.85.215.83","app":"PHP","message":"Cannot modify header information - headers already sent at \/www\/kepow\/owncloud\/lib\/private\/user\/session.php#288","level":3,"time":"2015-02-09T20:51:06+00:00"}
{"reqId":"230bd67050902ddeb36c9af31787d650","remoteAddr":"84.85.215.83","app":"PHP","message":"Cannot modify header information - headers already sent at \/www\/kepow\/owncloud\/lib\/private\/user\/session.php#289","level":3,"time":"2015-02-09T20:51:06+00:00"}
{"reqId":"230bd67050902ddeb36c9af31787d650","remoteAddr":"84.85.215.83","app":"PHP","message":"opendir(\/www\/kepow\/owncloud_data\/files_encryption\/keyfiles): failed to open dir: No such file or directory at \/www\/kepow\/owncloud\/lib\/private\/files\/storage\/local.php#78","level":3,"time":"2015-02-09T20:51:06+00:00"}
{"reqId":"230bd67050902ddeb36c9af31787d650","remoteAddr":"84.85.215.83","app":"core","message":"unable to rename, file does not exists : files_encryption\/Jordi.private.key","level":3,"time":"2015-02-09T20:53:38+00:00"}

Retry:

{"reqId":"1637703e855ad9def0d8b34063f10a6e","remoteAddr":"84.85.215.83","app":"PHP","message":"Cannot modify header information - headers already sent at \/www\/kepow\/owncloud\/lib\/private\/user\/session.php#282","level":3,"time":"2015-02-09T20:55:39+00:00"}
{"reqId":"1637703e855ad9def0d8b34063f10a6e","remoteAddr":"84.85.215.83","app":"PHP","message":"Cannot modify header information - headers already sent at \/www\/kepow\/owncloud\/lib\/private\/user\/session.php#283","level":3,"time":"2015-02-09T20:55:39+00:00"}
{"reqId":"1637703e855ad9def0d8b34063f10a6e","remoteAddr":"84.85.215.83","app":"PHP","message":"Cannot modify header information - headers already sent at \/www\/kepow\/owncloud\/lib\/private\/user\/session.php#284","level":3,"time":"2015-02-09T20:55:39+00:00"}
{"reqId":"1637703e855ad9def0d8b34063f10a6e","remoteAddr":"84.85.215.83","app":"PHP","message":"Cannot modify header information - headers already sent at \/www\/kepow\/owncloud\/lib\/private\/user\/session.php#287","level":3,"time":"2015-02-09T20:55:39+00:00"}
{"reqId":"1637703e855ad9def0d8b34063f10a6e","remoteAddr":"84.85.215.83","app":"PHP","message":"Cannot modify header information - headers already sent at \/www\/kepow\/owncloud\/lib\/private\/user\/session.php#288","level":3,"time":"2015-02-09T20:55:39+00:00"}
{"reqId":"1637703e855ad9def0d8b34063f10a6e","remoteAddr":"84.85.215.83","app":"PHP","message":"Cannot modify header information - headers already sent at \/www\/kepow\/owncloud\/lib\/private\/user\/session.php#289","level":3,"time":"2015-02-09T20:55:39+00:00"}
{"reqId":"1637703e855ad9def0d8b34063f10a6e","remoteAddr":"84.85.215.83","app":"PHP","message":"copy(\/www\/kepow\/owncloud_data\/owncloud_private_key): failed to open stream: No such file or directory at \/www\/kepow\/owncloud\/lib\/private\/files\/storage\/local.php#211","level":3,"time":"2015-02-09T20:55:39+00:00"}
{"reqId":"1637703e855ad9def0d8b34063f10a6e","remoteAddr":"84.85.215.83","app":"PHP","message":"copy(\/www\/kepow\/owncloud_data\/public-keys): failed to open stream: No such file or directory at \/www\/kepow\/owncloud\/lib\/private\/files\/storage\/local.php#211","level":3,"time":"2015-02-09T20:55:39+00:00"}
{"reqId":"1637703e855ad9def0d8b34063f10a6e","remoteAddr":"84.85.215.83","app":"PHP","message":"opendir(\/www\/kepow\/owncloud_data\/public-keys): failed to open dir: No such file or directory at \/www\/kepow\/owncloud\/lib\/private\/files\/storage\/local.php#78","level":3,"time":"2015-02-09T20:55:39+00:00"}
{"reqId":"1637703e855ad9def0d8b34063f10a6e","remoteAddr":"84.85.215.83","app":"PHP","message":"opendir(\/www\/kepow\/owncloud_data\/files_encryption\/keyfiles): failed to open dir: No such file or directory at \/www\/kepow\/owncloud\/lib\/private\/files\/storage\/local.php#78","level":3,"time":"2015-02-09T20:55:39+00:00"}
{"reqId":"1637703e855ad9def0d8b34063f10a6e","remoteAddr":"84.85.215.83","app":"PHP","message":"opendir(\/www\/kepow\/owncloud_data\/owncloud_private_key): failed to open dir: No such file or directory at \/www\/kepow\/owncloud\/lib\/private\/files\/storage\/local.php#78","level":3,"time":"2015-02-09T20:55:39+00:00"}

Browser log

Using //@ to indicate sourceMappingURL pragmas is deprecated. Use //# instead jquery.min.js:1
Using //@ to indicate sourceMappingURL pragmas is deprecated. Use //# instead jquery-migrate.min.js:3
Content Security Policy: The page's settings blocked the loading of a resource at self ("script-src https://kepow.org 'unsafe-eval'"). index.php
Use of getPreventDefault() is deprecated.  Use defaultPrevented instead. jquery.min.js:5

[MorrisJobke]

cc @schiesbn

[doits]

I had the same "problem" with the hang but eventually the upgrade did not hang but instead was simply taking a really long time (> 30 minutes). strace showed the update process was opening each encrypted file (that's why I know it was doing something and therefore I did not abort it) and it was doing it at about 5 files per second ... so if you have a lot of number of files it can take some time. Maybe that's the cause here?

Edit: The file the upgrade opened were all located in the subfolder data/username/files_encryption/ and ended with .key or fileKey or .shareKey - so I assume it encryption had to be updated.

[tomcur]

It's certainly a possibility. I've had the upgrader running for a while (~1 hour), and I don't have a large number of files per se. I'm currently running it again and am logging the system calls using strace. It's still going strong (and quick), but I am seeing some errors, such as:

[pid 27995] 23:52:12.429875 lstat("/www/kepow/owncloud_data/Jordi/encryption_migration_backup_2015-02-09_22-50-58/share-keys/***/***/***/***.Jordi.shareKey", 0x7fff33790390) = -1 ENOENT (No such file or directory)

I also noticed that a problem with the private key associated with the user Jordi was reported in the errors logged in owncloud.log for the first upgrade attempt.

[schiessle]

The upgrade will rename all encryption keys. If you have a lot of files this can take some time. We needed to re-structure the way the keys are stored to avoid file name collisions for some rare username-filename combinations. Before the upgrade script starts to rename the encryption keys it creates a backup of all your keys. So nothing should be lost. If you know how the key where stored in OC7 you can restore the backup and try to run the migration again with the occ script.

If you are not sure how to restore the backup I can give you more details tomorrow.

[tomcur]

I am not entirely sure how to restore the backup. Because of several upgrade attempts I now have a number of folders with names such as encryption_migration_backup_2015-02-09_16-07-16. I believe the folders from the first attempt contain everything I need.

Should I just copy the contents of all these folders from the first attempt back to their parent directories (i.e., cp -a ./ ../) and then remove the backup folders?

[kkretsch]

Having a similar problem I switched back to 704. I have nearly 10GB of encrypted media which seems to upgrade for such a long time OC9 might even be released befaore it finishes.

Any hints how to upgrade in a reasonable timeframe?

[schiessle]

@Beskhue yes, take the backup from the first attempt because this should contain the state from OC7. Every user should have such a backup folder. Just that we don't lose any state I would make a manual backup from each users files_encryption folder in the current state, then delete the content of the files_encryption folder. Now copy the content of encryption_migration_backup_<date> from the first attempt for each users to files_encryption folder.

Also in data/ you should have a system wide 'files_encryption' folder for which also backups with the pattern encryption_migration_backup_<date> should exists. Like for the users first backup the existing files_encryption folder and then copy the backup to the system wide files_encryption folder.

Afterwards try to run the migration again ./occ encryption:migrate-keys. I would also suggest to stop your web server before you perform this steps, just to make sure that no update gets triggered somewhere in between or that any sync client tries to read some files etc.

[schiessle]

@kkretsch we need to rename all encryption keys, there is no way arround it. Depending on how many files you have and how often they are shared you have more or less keys we need to migrate. But the keys are quite small files, we don't touch the actual files. So the crucial factor is the number if files and how often they are shared, not the size of the files. Renaming the keys should be possible in a reasonable time, depending on you hard disc performance. I would suggest to shutdown your web server and trigger the update with the occ command line tool to avoid any timeouts.

[inotriel]

Maybe this is the "problem" I mentioned here #13992 (comment)

I have about 7 oder 8 thousand files in my oc and the harddisk might not be the fastest in the world. So waiting for about 30 minutes maybe was just to little time.

Is there any chance to see if occ is still working or stuck?

[schiessle]

You could monitor the files_encryption folder of your users the size of some of the folders should change from time to time if it still runs. Also the backup folders should appear over time for the users.

[inotriel]

Good idea, thank you. I'll give it another try tonight!

[ghost]

I'm running the occ upgrade cmd now for 3h !!! I have something about 5GB total. I didn't have such an issue before with the encryption enabled. The process runs with about 3 to 8% CPU usage and I would like to know what he's doing.... oh wait. It just finished ... lol.

I think it's definately a good idea to shut down the apache server. That's what I just did and the process finished right after... Could that be correlated?

[kkretsch]

After removing two old entries in the Database that "local::/..." stuff with old path not existing any more I reran it on a clean version 7 restore using the command line and after under an hour it went thru without any error. The machine was using it2 100% cpu for that task, but there where still some spare cores left :-)

[khlschrnk]

If have about 135k (not including the files_encrytion folder) files that use about 75GB. I started the ./occ upgrade 12 hours ago and it is still running on an idle pc.
As you can see in the attachment (output of iotop) that the process is still running.
I am not sure if it is unusual to have the highest IO in [jbd2/md2-8]. Maybe my software raid is causing issues?
My CPU load is around 1-2% for ./occ.

As a rough calculation I expect the renaming process per file to be way faster than 50ms. With respect to the number of files it should not exceed 2 hours total. So in my opinion there is something broken.

I don't find any other usefull information in my logs.

[doits]

If you want to see what it is doing (which files operating and how fast), use strace:

strace -e open -p PID

# for example (not tested if this really gets the pid but I think it should)
strace -e open -p `ps ax | grep occ | grep -v grep | sed 's/ .*//' | head -n 1`

For me it was about 3 file per second only.

[jknockaert]

I have a very similar issue. However, the key migration stopped after two or three minutes. After restoring the key backup as explained by @schiesbn, I tried to restart the migration using ./occ encryption:migrate-keys, but that gives a InvalidArgumentException: "There are no commands defined in the "encryption" namespace." Not sure how to proceed now.

[khlschrnk]

Finished. It took me 22 hours to complete my 135k files upgrade.

Sync clients are syncing again on all platforms but my webinterface stays blank. No idea why but i guess this is a seperate issue ...

[inotriel]

OMG. So if that is a point of reference I should wait some hours. We are using oc in a productive environment so ... well, it will be a little bit complicated.

[Robstaaaa]

I ran into a similar issue:

My library is about 60 GB including lots of small files, so I expect the process to take quite long. However, the update process stops with an fatal error after short time (1h).

• Apache is stopped
• Command is: sudo -u www-data php /var/www/owncloud/occ upgrade

Errors were slightly different for consecutive executions of the same command:

• 1st try: PHP Fatal error: Maximum execution time of 3600 seconds exceeded in /var/www/owncloud/lib/private/files/filesystem.php on line 516
• 2nd try: PHP Fatal error: Maximum execution time of 3600 seconds exceeded in /var/www/owncloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOStatement.php on line 104
• 3rd try: PHP Fatal error: Maximum execution time of 3600 seconds exceeded in /var/www/owncloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOStatement.php on line 91

Any advice?

[EDIT] I could not find this limitation of 3600 seconds for my php configuration. In php.ini the default value for max_execution_time is 30 seconds, in the owncloud directories I could not find anything that overwrites the default.

[jknockaert]

Running ./occ upgrade does complete successfully (after 2 hours). However, after doing so I get the message "Encryption App is enabled but your keys are not initialized, please log-out and log-in again". In the log it says "Session does not contain a private key, maybe your login password changed" and "private key exists but public key is missing". I definitely did not change any password. Could something have gone wrong because the database schema update was run again on an already updated database?

[peperunas]

+1

[tflidd]

What is your recommendation for users on shared hosting solutions? They often won't have ssh-access and no execution times of several hours. Wait for OC 8.0.1?

[karlitschek]

@schiesbn

[Robstaaaa]

My solution if you run into the 3600 second timeouts:

Go to the file: "/var/www/owncloud/lib/base.php" and change the 3600 second timeout settings to 0.

After that, the upgrade went through without any errors and all encrypted files are available again in the updated owncloud version without further issues. For me it took about 2 hours.

The real pain is that parameters for php can be configured in various places which is irritating. Potential places which I checked first were the following, where I found settings which eventually were overwritten:

• /etc/php
• /var/www/owncloud/.htaccess
• /var/www/owncloud/config/config.php

[jknockaert]

I finally managed to finish the upgrade. I misunderstood the explanation by @schiesbn: there should be no system wide files_encryption folder in /data; rather should the back up of system-wide folders (like public keys) be placed directly in /data. And ./occ encryption:migrate-keys is only available after a previous upgrade to 8 was completed, otherwise run ./occ upgrade. This was definitely one of the rougher owncloud upgrades, taking two hours on a small scale server. I doubt there's any reasonable application of this upgrade for large scale servers.

[jknockaert]

@armaccloud Before running ./occ encryption:migrate-keys again, restore the pre-upgrade folder layout as pointed out by @schiesbn; this means that the content of /data/<user>/encryption_migration_backup_<date> should go to /data/<user>/files_encryption; however, the content of /data/encryption_migration_backup_<date> should go to /data (not /data/files_encryption). If you have multiple values for <date>, use the oldest one (i.e. the first one created upon the initial upgrade to 8). Note that there is a /data/encryption_migration_backup_<date>/files_encryption, but it should be empty, as in pre-OC8 installations there was no use for /data/files_encryption.

[armaccloud]

Many thanks to you both; I have managed to run the script. However it only performed the action on some files, the majority was not executed. Only the following couple of folders have been created and they are not complete by far;

The following errors are appearing in the logs;

{"reqId":"54248f7778847","app":"OC\\Files\\Cache\\Scanner","message":"!!! Path 'files\/Photos\/2007\/Oud en Nieuw 06-07\/01012007175.mp4' is not readable !!!","level":0,"time":"2014-09-25T21:56:07+00:00","method":"PUT","url":"\/owncloud\/remote.php\/webdav\/Photos\/2007\/Oud%20en%20Nieuw%2006-07\/01012007175.mp4"} {"reqId":"54248f7e98453","app":"OC\\Files\\Cache\\Scanner","message":"!!! Path 'files\/Photos\/2007\/Oud en Nieuw 06-07\/311220063585.jpg' is not readable !!!","level":0,"time":"2014-09-25T21:56:14+00:00","method":"PUT","url":"\/owncloud\/remohte.php\/webdav\/Photos\/2007\/Oud%20en%20Nieuw%2006-07\/311220063585.jpg"}

This might have to do with the folder path, which in the above example is the following:
Photo's and Video's\2007\Oud en Nieuw\... Could it have an issue with the ' in the folder names?

Also, the folder Photo's and Video's used to be called Photo's & Video's (was renamed a long time before I did the upgrade) and the keyfiles folder seems to contain both foders (see screenshot). That might also be creating issues.

Makes it strange though, why e.g. the folders Study and Work are also not been converted.
Any ideas?

[tomcur]

I ran the upgrade process again (thank Zeus for screen) and it finished successfully. It took 10+ hours.

Thanks for your help, everyone!

[lupa18]

I have the same problem. I upgraded by running:

sudo apt-get upgrade

Env:

• OS: Ubuntu 14.04.1
• Kernel: 3.2.0-67-generic

and now?

[eminaksehirli]

Hello, I've managed to complete the upgrade. I'm using owncloud on a shared hosting and I had to mirror the installation on my local machine, do the upgrade, and redeploy. The update script ran around 12 hours and used more than 7 gigs of memory. With these requirements, I think there should be a warning on the upgrade page. I documented my process here: http://memin.tk/informatics/2015/02/22/Upgrading-to-Owncloud-8.html

[DeepDiver1975]

@icewind1991 can you please have a look at the encryption migration - https://github.com/owncloud/core/blob/master/apps/files_encryption/lib/migration.php ?

The migration is using files view which is know to be performance intensive compared to using the storage directly - might this help here as well?

The high memory consumption is a bit frightening as well 🙊

[PVince81]

I had the same "problem" with the hang but eventually the upgrade did not hang but instead was simply taking a really long time (> 30 minutes). strace showed the update process was opening each encrypted file (that's why I know it was doing something and therefore I did not abort it) and it was doing it at about 5 files per second ... so if you have a lot of number of files it can take some time. Maybe that's the cause here?

5 files a second ?! How can it be that slow ?

[PVince81]

If I remember well, some keys are cached within a single PHP call to speed up the upgrade and some file operations. If run during the upgrade, I wonder if that cache's size is increasing when there are many files. That cache should probably be adapted to have invalidation logic.

[PVince81]

I tried an upgrade with 1600 files, here is the result: https://blackfire.io/profiles/b07deb06-7a94-4e67-9479-a9e9798f2867/graph

[PVince81]

Looks like it's spending time trying to propagate/calculate the folder sizes, even for "files_encryption" ?! That should be blocked and maybe rescanned at the end.

That might be what is causing the 255K database queries...

[icewind1991]

#14573 Should greatly reduce the time the migration takes, does anyone here with a large amount of encrypted data have the opportunity to retest the migration with that patch

[PVince81]

I also noticed that the keys are copied but not moved from the old "keyfiles" to the new "keys" folder. Maybe moving them (if possible) that could provide an additional boost.

I'll retest with that PR first.

[PVince81]

The PR works well, I also improved it further.

[DeepDiver1975]

too late for 8.0.1 -> 8.0.2

[rjaeckel]

I also had the Issue that after the Timeout occured, the encryption-keys got currupted.

For anyone who needs to recove the previous keys as well, here is a script generating the actions to be made to recover the pre-update state

#!/bin/bash
### !!! needs to be run as webserver-user
### !!! EDIT this pattern to match all user directories in OC data directory
userpattern='^(.{5}|cloud-administration-user)$'
# recover user keys
for username in $(ls -1tr|egrep -o $userpattern); do
  backup=$(ls -1tr $username|egrep '^encryption_migration_backup'|head -n1)
  if [ "x" != "x$backup" ]; then
    # echo "$username --> $backup"
    # create backup of broken state, just in case
    echo "mv $username/files_encryption $username/files_encryption.broken"
    # recreate the renamed folder
    echo "mkdir -p $username/files_encryption"
    # recover the pre-update state
    echo "cp -r $username/$backup/* $username/files_encryption"
  fi
done

# recover global encryption directories
globbackup=$(ls -1tr|egrep '^encryption_migration_backup'|head -n1)
for bu in $(ls -1 $globbackup); do
  echo "mv $bu $bu.broken"
  echo "cp -r $globbackup/$bu $bu"
done

[DeepDiver1975]

the long execution time of the key migration will be fixed with 8.0.3

[rsling]

Well, it was not fixed, at least not for me! I tried multiple times to upgrade from 7 to 8.0.3 on a Debian system. The installation has 5 users and 100GB total, some folders with many small files. After 4 days, I interrupted the procedure. strace showed that the updater kept working on tens of thousands of encryption keys. This is completely useless!

[rjaeckel]

@DeepDiver1975
what about considering pcntl as an option? this would make it possible to use the whole computing power... there are several libraries here on github providing php threads this way.

[hostingnuggets]

This issue has not been fixed, see #19319