S3 - Previews not deleted

[SimplyCorbett]

This is related to #20333

When removing images previews are not deleted from the S3 bucket if using primary storage.

Before upload:

"num_objects": 1421

After uploading 10 images and generating previews:

"num_objects": 1451

With previews that brings it up to 30 total images/files.

After removing them, deleting the trashbin and running the S3 garbage collection process this is the amount of files leftover:

"num_objects": 1441

Previews are not removed and I didn't realize this because my primary use case is not images.

Configuration:
18.0.3
Primary S3 multibucket storage | path style false
Debian 10
PHP 7.4-FPM
HAProxy + Nginx
Postgresql

[MorrisJobke]

Mind to run this query: (it's might be a bit costly, but it should give us some hints if there is already a mismatch in the DB or just on the object store)

SELECT fileids FROM (
	SELECT DISTINCT(SUBSTRING_INDEX(SUBSTRING_INDEX(f1.path, '/', -2), '/', 1)) AS fileids
	FROM oc_filecache f1
	WHERE f1.storage = (SELECT numeric_id FROM oc_storages WHERE id LIKE "local::%" LIMIT 1) AND f1.path LIKE 'appdata_%/preview/%.png'
) as r
WHERE r.fileids NOT IN (SELECT f2.fileid FROM oc_filecache f2);

If you have multiple local storages (when using the external storage app with local storage for example) then maybe replace the (SELECT numeric_id FROM oc_storages WHERE id LIKE "local::%" LIMIT 1) with the specific numeric ID of local::/YOUR/DATA/DIR/PATH.

This query basically extracts all the file IDs from the preview paths and then checks them against the entries in the file cache. That then should give us a hint on which previews exist for files that don't exist anymore.

[solracsf]

@MorrisJobke as i'm having these issues (as pointed in the 1st post on #20333 ) your query returns an empty result:

MariaDB [nextcloud]> SELECT fileids FROM (
    -> SELECT DISTINCT(SUBSTRING_INDEX(SUBSTRING_INDEX(f1.path, '/', -2), '/', 1)) AS fileids
    -> FROM oc_filecache f1
    -> WHERE f1.storage = (SELECT numeric_id FROM oc_storages WHERE id LIKE "local::%" LIMIT 1) AND f1.path LIKE 'appdata_%/preview/%.png'
    -> ) as r
    -> WHERE r.fileids NOT IN (SELECT f2.fileid FROM oc_filecache f2);
Empty set (0.004 sec)

I don't think this are (only) previews, as i pointed at #20333 (comment) because i can see objects in the bucket >10Mb so clearly these are not previews.

[SimplyCorbett]

I don't think this are (only) preview

Then your issue is different from mine.

I'm no longer on S3 though so I'm going to go ahead and close this ticket.

[fnkr]

@SimplyCorbett Can you please keep this open? It's still an issue. You can unsubscribe if you want. :)

[SimplyCorbett]

@SimplyCorbett Can you please keep this open? It's still an issue. You can unsubscribe if you want. :)

Is it? Nobody else has actually posted anything that replicates my issue.

Is the issue what @acsfer posted in #20333 or is the issue preview-only as mentioned in my OP?

[solracsf]

I can replicate your exact issue. I'm just saying this is bigger than simply the previews with my test case.

Please keep in mind that Git issues are not your issues or my issues but issues that are lacking triage.

You're not using S3 anymore so you can't contribute anymore to debug it. No problem. Just unsubscribe from it and thanks for having report it 👍

[SimplyCorbett]

I can replicate your exact issue. I'm just saying this is bigger than simply the previews with my test case.

Please keep in mind that Git issues are not your issues or my issues but issues that are lacking triage.

You're not using S3 anymore so you can't contribute anymore to debug it. No problem. Just unsubscribe from it and thanks for having report it +1

Well, I stopped using S3 because of this issue so I'm still invested.

But I did not have the issues you mention in #20333 so perhaps you are seeing a placebo effect with this one?

Or one of the nextcloud updates introduced a new problem..

I'll go ahead and keep this open then.

[szaimen]

Related #26014

[charleypaulus]

I have the same issue with Swift primary object storage.

[charleypaulus]

I wrote this script. It:

• gets from database all previews that no longer have their full original image
• checks that these full images are actually absent from object storage
• deletes these previews both from object storage and from database

[szaimen]

Hi, please update to 24.0.8 or better 25.0.2 and report back if it fixes the issue. Thank you!