After deleting image, the preview files remain forever

[ho1ger]

This issue is obviously related to #5558, which was closed for Nextcloud 12. I run Nextcloud 13.0.6 and the issue exists for me. Can anyone confirm?

Steps to reproduce

1. Upload images (in my case I had about 5gb of JPEGs)
2. Let previews be generated (e.g. using the preview generator app and a cron job)
3. Delete those images
4. Purge trashbin
   (5. Wait for trashbin_retention_obligation is exceeded)

Expected behaviour

The cached previews should be deleted at some point in time...

Actual behaviour

The cached previews are NOT deleted, in my case I have 13GB of preview images of different sizes

Server configuration

Operating system:
Debian

Web server:
Nginx

Database:
Maria

PHP version:
7.2
Nextcloud version: (see Nextcloud admin page)
13.0.6

Updated from an older Nextcloud/ownCloud or fresh install:
Updated; but previews were generated on NC 13

List of activated apps:

``` - activity: 2.6.1 - admin_audit: 1.3.0 - admin_notifications: 1.0.1 - apporder: 0.5.0 - bookmarks: 0.12.2 - bookmarks_fulltextsearch: 1.0.0 - bruteforcesettings: 1.1.0 - calendar: 1.6.1 - caniupdate: 0.1.3 - contacts: 2.1.5 - dav: 1.4.7 - federatedfilesharing: 1.3.1 - files: 1.8.0 - files_fulltextsearch: 1.0.0 - files_fulltextsearch_tesseract: 1.0.0 - files_pdfviewer: 1.2.1 - files_rightclick: 0.8.4 - files_sharing: 1.5.0 - files_texteditor: 2.5.1 - files_trashbin: 1.3.0 - files_versions: 1.6.0 - files_videoplayer: 1.2.0 - fulltextsearch: 1.0.0 - fulltextsearch_elasticsearch: 1.0.0 - gallery: 18.0.0 - impersonate: 1.0.4 - logreader: 2.0.0 - lookup_server_connector: 1.1.0 - metadata: 0.7.0 - nextcloud_announcements: 1.2.0 - notifications: 2.1.2 - oauth2: 1.1.1 - password_policy: 1.3.0 - previewgenerator: 1.1.0 - provisioning_api: 1.3.0 - serverinfo: 1.3.0 - sharerenamer: 1.5.0 - twofactor_backupcodes: 1.2.3 - twofactor_totp: 1.4.1 - updatenotification: 1.3.0 - workflowengine: 1.3.0 ```

Nextcloud configuration:

``` { "system": { "instanceid": "***REMOVED SENSITIVE VALUE***", "passwordsalt": "***REMOVED SENSITIVE VALUE***", "secret": "***REMOVED SENSITIVE VALUE***", "trusted_domains": [ "xxxxxx", "xxxxxx" ], "datadirectory": "***REMOVED SENSITIVE VALUE***", "overwrite.cli.url": "xxxxxxxxx", "dbtype": "mysql", "version": "13.0.6.1", "dbname": "***REMOVED SENSITIVE VALUE***", "dbhost": "***REMOVED SENSITIVE VALUE***", "dbport": "", "dbtableprefix": "oc_", "dbuser": "***REMOVED SENSITIVE VALUE***", "memcache.local": "\\OC\\Memcache\\APCu", "dbpassword": "***REMOVED SENSITIVE VALUE***", "installed": true, "mail_from_address": "***REMOVED SENSITIVE VALUE***", "mail_smtpmode": "smtp", "mail_smtpauthtype": "LOGIN", "mail_domain": "***REMOVED SENSITIVE VALUE***", "maintenance": false, "logtimezone": "Europe\/Berlin", "loglevel": 0, "theme": "", "trashbin_retention_obligation": "auto, 7", "updater.release.channel": "production", "mail_smtpauth": 1, "mail_smtphost": "***REMOVED SENSITIVE VALUE***", "mail_smtpport": "587", "mail_smtpsecure": "tls", "mail_smtpname": "***REMOVED SENSITIVE VALUE***", "mail_smtppassword": "***REMOVED SENSITIVE VALUE***" } ```

[nextcloud-bot]

GitMate.io thinks possibly related issues are #5558 (After deleting an (image) file, the preview files remain forever), #9523 (A file completely disappears after deleting from a group folder), #8241 (image), #2413 (Previews for empty files broken), and #3041 (Deleting a preview folder stops writing previews to disk).

[rullzer]

Can you check if you have an entry in your background jobs: OC\Preview\BackgroundCleanupJob?

Also note that if you use ajax cron it will only delete 10 at a time (to not run into timeouts).

[ho1ger]

I am using a Linux cron job, not Ajax cron.

Sorry; how do I access OC\Preview\BackgroundCleanupJob?

[ho1ger]

In the logs I found the following which is related to background Cleanup Jobs:

{"reqId":"0br9L4RsCRwi9HJhnVr9","level":1,"time":"2018-06-08T10:32:51+02:00","remoteAddr":"xxx","user":"--","app":"updater","method":"GET","url":"\/core\/ajax\/update.php?requesttoken=SvT6GXpQphnGbl9Mg9WdNOUBmJ35bJJ5GLpgNhsC%2FQk%3D%3AJ6yiexIZkHu8I24G6L6sYLBY0tyQAuU6XdE5cllluD0%3D","message":"\\OC\\Repair::step: Repair step: Add preview cleanup background jobs","userAgent":"Mozilla\/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit\/605.1.15 (KHTML, like Gecko) Version\/11.1.1 Safari\/605.1.15","version":"13.0.2.1"}
{"reqId":"JiMmzSdZG0ya4yWgnY5p","level":1,"time":"2018-06-12T07:45:32+02:00","remoteAddr":"","user":"--","app":"updater","method":"--","url":"--","message":"\\OC\\Repair::step: Repair step: Add preview cleanup background jobs","userAgent":"--","version":"13.0.3.2"}
{"reqId":"Vuip1twG4c8HRKSb8r1M","level":1,"time":"2018-08-30T18:40:17+02:00","remoteAddr":"","user":"--","app":"updater","method":"--","url":"--","message":"\\OC\\Repair::step: Repair step: Add preview cleanup background jobs","userAgent":"--","version":"13.0.4.0"}
{"reqId":"3N5R2TG63uE5RtzH9Hh1","level":1,"time":"2018-08-30T18:40:48+02:00","remoteAddr":"xxxxxx","user":"--","app":"updater","method":"GET","url":"\/core\/ajax\/update.php?requesttoken=Y9%2FT7HxGSiDjwrAVSLSFWrJQPYx4Lr9Dd44Qy4F6hmc%3D%3AULXl3R8pPGKRuoRsC4G1PcZ7CcVNVsUkJtdyr9VL1gU%3D","message":"\\OC\\Repair::step: Repair step: Add preview cleanup background jobs","userAgent":"Mozilla\/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit\/605.1.15 (KHTML, like Gecko) Version\/11.1.2 Safari\/605.1.15","version":"13.0.4.0"}

[ho1ger]

Q: What would actually happen if I just delete the entire preview folder? Would the preview generator understand that some needed previews are missing and recreate them?

[rullzer]

AH sorry. Check the oc_jobs table.

No it won'tyou need to rescan your appdata folder with the occ command

[ho1ger]

Hope I got that right: select * from oc_jobs where class="OC\Preview\BackgroundCleanupJob"; does not return anything.

[nachoparker]

I can confirm this is still an issue in NC15.0.8

[Iolaum]

I have experienced the same issue in my installation of NextCloud on a nextcloudbox running nextcloudpi. I believe the problem lied with the preview generator app. When I realized the previews were eating up a huge amound of space I ended up deleting the app and manually removed the offending data.

[nachoparker]

@lolaum since you are using NCP it is configured now to not make unnecessary previews so it should be faster and take up less space.

Previews are generated one way or another (either by the pregen app or through browsing). Whatever the case may be when you delete the files the previews remain both in the previews folder and in the database and they are never pruned.

[skjnldsv]

@rullzer didn't we fixed this a few releases ago?
I cannot reproduce the issue. Shall we add a background job to cleanup old invalid files id?

[ghost]

This issue has been automatically marked as stale because it has not had recent activity and seems to be missing some essential information. It will be closed if no further activity occurs. Thank you for your contributions.

[James-Oakley]

This issue was the one that came up in searches, so someone please link to a more active issue if there is one. I had found that updating previews was taking a very long time, so I moved the appdata/preview folder to one side, rescanned my files, and then reran the preview generator. It took a while, but got there soon enough. The newly generated preview folder contains 61 MB of data. The one I moved contained 1.1 GB of data. So it seems that the preview folder bloats rather than keeping itself clear of previews of files that no longer exist.