Openstack Swift Errors

[joakimohman]

We have a couple of problems when using Openstack Swift external storage and I am listing them in this Issue as I do not know if they are related. Everything works fine when using local storage and the basic functionality works in the web GUI.

Expected behaviour

a. No errors for files stored on Swift in the Owncloud log.

b. Restore a deleted file stored in Swift should work.

c. Sync files from a client to swift storage should work.

Actual behaviour

Tell us what happens instead
a. We get errors in the Owncloud log for all the files that are stored in swift.
Error files_external Problem retrieving object [http://domain/file...]

b. Delete a file works as expected and the file can be found when accessing browsing the deleted files but when you restore the file is moved back to its original location but the file size is now 0 Bytes.

c. When you try to sync files from a client you get "Bad Request" error when syncing against swift storage but works fine against local storage.

Steps to reproduce

1. Store files on Swift stroage and check the Owncloud log.
2. Delete a file stored on Swift storage, Restore the file and check the file size.
3. Try to sync a file from a client to a Owncloud Swift folder.

Server configuration

Operating system: Centos 6.4

Web server: Apache 2.2.15

Database: mySQL 5.5.34

PHP version: 5.4

ownCloud version: 6.0.1

Updated from an older ownCloud or fresh install: Fresh

List of activated app:
Activity
Deleted files
External storage support
PDF Viewer
Share Files
Text Editor
Updater
Video Viewer

The content of config/config.php: (Without the database password and passwordsalt)

'xxx', 'passwordsalt' => 'xxx', 'datadirectory' => '/var/www/webroot/ROOT/data', 'dbtype' => 'mysql', 'version' => '6.0.0.16', 'dbname' => 'owncloud', 'dbhost' => 'xxx', 'dbtableprefix' => 'oc_', 'dbuser' => 'owncloud', 'dbpassword' => 'xxx', 'installed' => true, 'loglevel' => '1', 'forcessl' => true, ); Are you using external storage, if yes which one: Openstack Swift ### Client configuration Browser: Chrome Operating system: Ubuntu / Windows 7 ### Logs #### Web server error log 88.80.172.4 - - [13/Feb/2014:15:43:05 +0000] "GET /remote.php/webdav/home/construction.gif HTTP/1.1" 200 109894 "-" "Mozilla/5.0 (Linux) csyncoC/0.91.4 neon/0.29.6" 88.80.172.4 - test [13/Feb/2014:15:43:06 +0000] "PROPFIND /remote.php/webdav/ HTTP/1.1" 207 438 "-" "Mozilla/5.0 (Linux) mirall/1.5.0" #### ownCloud log (data/owncloud.log) Error files_external Problem retrieving object [http://domain/file...] #### Browser log No browser problem

[yannickgrammont]

same problem here.
I see 11 curl call per file on swift each page reload or ajax call.
folders generate lots of error because of trying 2 URL : "test" and "test/" and test was not found...

[PVince81]

I just had a look in the code and there are definitely too many calls and some caching needs to be implemented. For many operations, it checks file_exists, is_dir and such multiple times. The first time could be cached and reused.

[PVince81]

It seems that the API that is used requires folders to be called with a trailing slash, so the code is calling is_dir very often to convert the path... so it's not possible to simply remove all these sanity checks to make it faster.

[PVince81]

Tentative fixes to reduce the curl calls: #7897 #7898

[julienfastre]

Hi,

We encounter this problem also, using owncloud 6.0.4 (we are about to switch to owncloud 7, but it does not seem it will be resolved in this new version.

Another problem we encounter: using File application, and synchronisation, is very slow !

[LordGaav]

Using ownCloud 8, I'm running into similar issues. It seems that ownCloud creates a file named dot (".") in the given Swift bucket. Retrieving this file causes issues, using the swift CLI client fails because "." can never be used as a filename.

Isn't it a better idea to use another name, something like .metadata ?

[PVince81]

I believe the entry with name "." is actually the entry for the filesystem root. It looks like there was a similar issue with S3 that was fixed by converting "." to "/": 599ab7a#diff-440d6acd86f4ca1cb35ebcdaaaed8e54R75

You could try doing this in https://github.com/owncloud/core/blob/master/apps/files_external/lib/swift.php#L81
Maybe add an additional if that checks if $path is "." then convert it to "/".

[LordGaav]

Adding a

                if ($path === ".") {
                        return "/";
                }

Does indeed prevent the creation of a dot file in the root. Uploading files still does not work, no file is created and a lot of log entries like this are created:

{"reqId":"a0f1f35b6f2405e2feac8f321d413934","remoteAddr":"2a01:518:2::41","app":"files_external","message":"Client error response\n[status code] 404\n[reason phrase] Not Found\n[url] http:\/\/XXXX\/swift\/v1\/nick-owncloud\/horizon_2014.1.3.orig.tar.gz","level":3,"time":"2015-04-20T09:52:32+00:00"}
{"reqId":"a0f1f35b6f2405e2feac8f321d413934","remoteAddr":"2a01:518:2::41","app":"files_external","message":"Client error response\n[status code] 404\n[reason phrase] Not Found\n[url] http:\/\/XXXX\/swift\/v1\/nick-owncloud\/horizon_2014.1.3.orig.tar.gz\/","level":3,"time":"2015-04-20T09:52:32+00:00"}
{"reqId":"a0f1f35b6f2405e2feac8f321d413934","remoteAddr":"2a01:518:2::41","app":"files_external","message":"Client error response\n[status code] 404\n[reason phrase] Not Found\n[url] http:\/\/XXXX\/swift\/v1\/nick-owncloud\/horizon_2014.1.3.orig.tar.gz","level":3,"time":"2015-04-20T09:52:32+00:00"}

[PVince81]

Hmmm... then maybe SWIFT doesn't work like S3 and would need another name for the root 😦
Or maybe there are other parts of the code that strip it again, like getContainerName.

It is still strange that it doesn't work because SWIFT has been working for a while. Maybe OC does work with "." but it's only the SWIFT clients themselves that don't like it ?

[markholland83]

Any update on this?

It's quite annoying that the OC implementation to sync with swift is so poor.

{"reqId":"IOvP5CnIZDBocZ6sx0fh","remoteAddr":"12.34.567.89","app":"files_external","message":"Client error response\n[status code] 404\n[reason phrase] Not Found\n[url] http://object.XX.openstack.XX/swift/v1/owncloud_2/filename.pdf","level":3,"time":"2015-08-26T11:55:11+00:00"}

Tests show that uploading files <1Mb works. Larger files give the above error.

[PVince81]

Are you guys still having issues with 8.2.2 ?

We now have integration tests running against a SWIFT docker (Ceph) and the tests run fine.

[MorrisJobke]

Since ownCloud 6 a lot has changed regarding the swift stuff. Please retest with a more recent version. Thanks

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.