files-app: Bad Performance on folder with many images in WebUI only

[mightycoco]

Steps to reproduce

1. Browse a folder in the WebUI containing about 5000 files
   -> takes ~5 minutes to load
2. Browse the same folder when mounted via a WebDav folder (windows)
   -> takes ~30 seconds to load
3. Browse the same folder on a mobile app (e.g. HopicExplorer on Windows Mobile)
   -> takes ~30 seconds to load

Expected behaviour

The performance shouldn't be THIS much worse in the WebUI

Actual behaviour

The performance is nearly 6 times slower in the WebUI

Investigation

In core/js/files/client.js in the method getFolderContents a WebDav request is made using following options:

[
"{DAV:}getlastmodified",
"{DAV:}getetag",
"{DAV:}getcontenttype",
"{DAV:}resourcetype",
"{http://owncloud.org/ns}fileid",
"{http://owncloud.org/ns}permissions",
"{http://owncloud.org/ns}size",
"{DAV:}getcontentlength",
"{http://nextcloud.org/ns}has-preview",
"{http://owncloud.org/ns}tags",
"{http://owncloud.org/ns}favorite",
"{http://owncloud.org/ns}comments-unread",
"{http://owncloud.org/ns}owner-display-name",
"{http://owncloud.org/ns}share-types"
]

that happens in the following block:

		this._client.propFind(
			this._buildUrl(path),
			properties,
			1
		)

This request takes 4m 51s for ~5800 files.

Now some tests by omitting certain properties:

without "{http://owncloud.org/ns}share-types" 1m 43s
without "{http://owncloud.org/ns}share-types", "{http://owncloud.org/ns}comments-unread" 0m 52s
without "{http://owncloud.org/ns}share-types", "{http://owncloud.org/ns}comments-unread", "{http://owncloud.org/ns}tags" 0m 52s
without "{http://owncloud.org/ns}*" (only leaving the core {DAV:} properties) 0m 31s

Alone taking away the "share-types" request improves the performance 4 times (~4.9 minutes to ~1.8 minutes)
Taking all custom http://owncloud.org/ns away and only taking the core WebDav DAV: properties reduces the request for ~5800 files to 30s. Something (esp. in the share-types propfind) takes a serious amount of time. Maybe this should be improved.

Server configuration

Operating system:
Debian Jessy
Linux myhost.mydomain 4.8.4-sunxi #6 SMP Sun Oct 23 15:55:47 CEST 2016 armv7l GNU/Linux
2GB ram
1Ghz DualCore ARM CPU (CubieTruck)

Web server:
lighttpd/1.4.35 (ssl) - a light and fast webserver
Build-Date: Aug 2 2016 18:01:49
ACPu
Redis
Compression

Database:
mysqladmin Ver 8.42 Distrib 5.5.53, for debian-linux-gnu on armv7l
Copyright (c) 2000, 2016, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Server version 5.5.53-0+deb8u1
Protocol version 10
Connection Localhost via UNIX socket
UNIX socket /var/run/mysqld/mysqld.sock
Uptime: 2 hours 37 min 45 sec

Threads: 1 Questions: 263578 Slow queries: 0 Opens: 178 Flush tables: 1 Open tables: 171 Queries per second avg: 27.847

PHP version:
PHP 5.6.27-0+deb8u1 (cli) (built: Oct 23 2016 13:49:28)
Copyright (c) 1997-2016 The PHP Group
Zend Engine v2.6.0, Copyright (c) 1998-2016 Zend Technologies
with Zend OPcache v7.0.6-dev, Copyright (c) 1999-2016, by Zend Technologies

Nextcloud version: (see Nextcloud admin page)
10.0.1

[LukasReschke]

cc @icewind1991

[icewind1991]

Alone taking away the "share-types" request improves the performance 4 times (~4.9 minutes to ~1.8 minutes)

Is already fixed for nc11

[rullzer]

@mightycoco if you are a brave soul you could try the beta!

[mightycoco]

@rullzer on my list for tomorrow

[icewind1991]

I'm aware that loading the comments information will still give a pretty heavy slowdown, hoping to work on that once 11 is out of the door (requires non-trivial changes).

Disabling the comments app is a workaround for people not relying on the functionallity

[mightycoco]

Performance in the beta is better, disabling the comments app does really bring a huge performance boost on these large folders.

Not sure, if there should be paging when listing so many files. Currently the entire folder contents are downloaded which generates a ~5MB xml response for the folder. But that's a different story.

Thanks guys and feel free to close this issue as I doesn't seem to have permissions to do so.

[LukasReschke]

I'm aware that loading the comments information will still give a pretty heavy slowdown, hoping to work on that once 11 is out of the door (requires non-trivial changes).
Disabling the comments app is a workaround for people not relying on the functionallity

@icewind1991 Can you file a new issue about this one or should we keep that one here?

@mightycoco Thanks a lot for testing 🚀

[goddib]

Hi everyone,

I'm not sure if I should create a new issue but I think mine relates to this: I have a folder with ~30k photos. At the moment I'm neither able to scroll through them all (30k gives a long list with lazyload) nor search through the folder (page crashes). When I search for files that I know are in that folder from the main view they get displayed correctly but I am unable to open them since the link is back to the folder (with the scrollto attribute). I am therefore unable to open these pictures in the web interface. WebDav works without any issue.

I hope this is the right spot to mention these issues, otherwise please direct me.

Thanks and regards,
goddib

12GB Ram
Intel E5-2620 v4 @ 2.10GHz

nextcloud 11.0.1
Debian 8.7‬
apache2 2.4.10 (+ACPu)
PHP 7.1.1

[icewind1991]

Fix for bad performance caused by having comments enabled is here: #4146

[MorrisJobke]

Should be fixed by #4146 - we will release this in 12.0.0. Please report back if this is still a problem with this version. Thanks

[elpedriyo]

The android app continues loading endless when trying to display photos

[MorrisJobke]

Could you report this in the android repository