Openstack SWIFT storage performance

[smadev]

cachegrind.out.zip

Hi,

We have difficulties interacting with with our external SWIFT container through ownCloud. All operations are really slow, from listing a directory content to uploading/downloading a file. The container itself is fine, interacting with it through any cli client does not raise any problem, everything is fast.

For example, listing a directory content with the following command takes ~21sec. (the folder contains 6 files, 6MB total).

curl -X PROPFIND -u "XXXX":"XXXX" "https://our.owncloud.instance.com/remote.php/webdav/container/directory" -v

Sharing is totally disabled.

We used webgrind to profile our ownCloud instance in order to track what was going on during file operations, it seems that LOTS of requests are sent by ownCloud to our Openstack instance. The total execution time of these request is almost 20sec for the previous curl command.

Attached you can find the webgrind file generated by the previous command.

Steps to reproduce

1. Register a swift container as an external storage
2. Add/remove files or navigate the container

Expected behaviour

Everything should be usable.

Actual behaviour

Everything is really slow.

Server configuration

Operating system: Debian 8.5
Web server: nginx/1.8.1
Database: PostgreSQL 9.4.8
PHP version: PHP 5.6.
ownCloud version: ownCloud 9.1
Updated from an older ownCloud or fresh install: fresh install
Where did you install ownCloud from: website

List of activated apps:

Antivirus App for files
Activity
Comments
External storage support
Federation
First run wizard  
Gallery
Mail Template Editor
Notifications
PDF Viewer
Provisioning API
Share Files
Text Editor
Update notification  
Versions
Video player

The content of config/config.php:

  'trusted_domains' =>
  array (
    0 => 'XXX.XXX.XXX.XXX',
    1 => 'our.owncloud.instance.com',
  ),
  'datadirectory' => '/var/www/owncloud/data',
  'overwrite.cli.url' => 'http://XXX.XXX.XXX.XXX',
  'dbtype' => 'pgsql',
  'version' => '9.1.0.15',
  'dbname' => 'owncloud',
  'dbhost' => '127.0.0.1',
  'dbtableprefix' => 'oc_',
  'dbuser' => 'XXXX',
  'dbpassword' => 'XXXX',
  'logtimezone' => 'UTC',
  'installed' => true,
  'quota_include_external_storage' => false,
  'enable_avatars' => false,
  'enable_previews' => false,
  'trashbin_retention_obligation' => 0,
  'loglevel' => 0,
  'memcache.local' => '\OC\Memcache\APCu',

Are you using external storage, if yes which one: SWIFT
Are you using encryption: no

[PVince81]

Hmmm, in 9.1 we added a stat cache which should have reduced the number of requests with SWIFT as external storage.

Are you using the SWIFT external storage or the home object store one ?

[smadev]

We are using APCu.
I updated the content of my previous post with the missing configuration entry .

[PVince81]

Okay, no home storage in config.php, so you're using the external storage variant.

Looks like it needs further optimization.

[smadev]

Sorry, I misunderstood your question. We were indeed using the external storage variant.
I configured our SWIFT container as primary storage, and everything works much better than before.

The problem seems only related to the external storage variant.

[PVince81]

The external storage variant works completely differently.

In the external storage mode, every file is stored using the full path as key. This means OC is simulating a tree structure inside the object store, similar to how some object store file managers do. This is bad for performance because need to filter every time a folder is accessed.

As primary storage it's much faster because all OC does is store the data of the file and the key is the file id. There is no path / tree lookup magic happening.

[PVince81]

In 9.1 perf for the SWIFT external storage should already have improved with the introduction of the stat cache: #7897.

If that's still not enough, might need to find a way to optimize this further.

[PVince81]

@smadev is it still slow with 9.1.4 or 9.1.5 RC2 ?

[hodyroff]

No further feedback.
Swift will be deprecated in favor of full S3 support. Which also works with Swift inside Openstack.

[GeorgeforChina]

hi ,have try one week to install a cluster server of swift,not cool ,since i didn't install openstack ,i just want using openstack swift ,but th website(https://docs.openstack.org/swift/latest/howto_installmultinode.html) show us the install muti swift server was need openstack ,then can install swift ,so i was so confused,it's any doc can help us to install a multi server ,not SAIO. thanks a lot if any one can reply!

[hodyroff]

I would really recommend to use Ceph/S3 even in the OpenStack context. Otherwise, as written above, for ownCloud in the future you'll need the S3 layer on top of OpenStack/Swift.

[GeorgeforChina]

Thanks ,it's just for using openstack swift cluster server i need ,did anyone can shared about it experience for deploy a cluster server of swift doc/wiki . Thanks a lot !

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.