[Bug]: enabling external storage on 28.0.2 generates error during occ files:scan
#43408
Comments
nfguide
added
0. Needs triage
|
Unable to reproduce. Tried both without and with files in the bucket. Can you please:
For the record, what S3 back-end (or provider) are you using? |
joshtrichards
changed the title
occ files:scan
Here is a snippet of the scan:
Thanks for taking the time to review. There is another config with 27.1.6 connecting to the same object storage, and the scan works correctly, with no errors. |
|
Can confirm this is a bug with S3 like buckets, I have the same issue. I am using Backblaze B2 as the bucket |
|
I am also confirming this issue with S3. Using Dreamhost's DreamObjects. "Identical" error list as above except for the line(s) with "Scanner.php(400)". Those are not included. This occurred after updating Nextcloud from 25.0.2.3 to 28.0.2.5 (mutiple upgrade steps) and php from 8.0 to 8.2. The log seems to indicate that the issue started after the core upgrade from 27.1.6.2 to 28.0.2.5. |
I concur the started "after the core upgrade from 27.1.6.2 to 28.0.2.5", as the error is not present on a test config running 27.1.6. |
|
hello |
|
Not sure if it helps to add another voice, but am tracking this. Exact same issue as OP so not posting any new details unless it helps. |
This comment was marked as off-topic.
This comment was marked as off-topic.
|
Upgraded from 27 to 28.0.2 and got a wall of red when the scanner ran as the final step. Identical to what has already been reported. External storage mounted to Wasabi's S3. Everything is accessible in Nextcloud as far as I can tell though, but would be nice if the scan worked. This is also filling my log with errors after an otherwise successful upgrade, once I solved all the nginx related javascript changes. This is a bit annoying as it might hide actual error errors that I'd care about.
|
|
Hi I face the same issue with a AWS S3 external storage. Upgraded to nextcloud 28.0.2 and the issue persist. Note: nextcloud server works fine, just huge amount of logs produced. |
|
Reproduced! Finally. :-) Requires a sub-folder to be created within the S3 mount. |
joshtrichards
added
1. to develop
Fixes #43408 Signed-off-by: Josh <josh.t.richards@gmail.com>
joshtrichards
added
3. to review
|
I have this exact same issue, running the latest NC with SMB shares and an iDrive S3 bucket attached. But I also noticed something else, I don't know if it's important or if it makes any difference, but for me the error only occurs when my pc is on, which is running the NC desktop app. When I shut down the desktop app, the spamming of the log stops immediately. I can even trigger it manually by clicking 'Sync now'. |
|
If you feel so inclined, a possible fix is in #43794 (you'll have to manually patch to test though). |
|
I do feel inclined, but wouldn't know where to begin, running the linuxserver NC container. So I'll wait this out, good to know that a solution is in the works. Thanks! |
|
Hi I feel inclined too 😆 I can test the patch manually, so far after running some investigation on my side I noticed I could trigger it using:
If I repeat the above actions and I see the error in the logs, then I patch the code, then I repeat the actions and I don't see the error. Would this be enough as an end2end test ? 🤔 |
I went ahead and applied Josh's proposed fix to a 28.0.3 installation because this appears to be scheduled in 29 and I didn't want to wait. The level 3 errors in the log re. array to string conversion are gone. We have three large stores on external S3 so I will let you know if there are any performance issues re. serializing/unserializing but so far things look good. Thank you! EDIT: I did a quick test on one store and I really don't see a performance issues on a files scan: |
|
@lavigne958 Correct. @stratacorps Thanks for sharing. And glad to hear it! |
Fixes #43408 Signed-off-by: Josh <josh.t.richards@gmail.com>
|
I too have just applied the fix from #43794 and things look to working again, running a user scan now to see if that sorts my slow down issue from the v28 upgrade |
|
The error still exists in 29-Beta. |
Yes, it is listed as a milestone for 29, so I assume if they decide to include this it will be in the final release, not necessarily in all of the beta(s) leading up to a release maybe It is still listed as a PR in Beta 3 from 7 hours ago, so I'm sure it will be there. Some comments on overhead concerns, I'm not sure how to read it, but #44186 |
|
28.0.4 and two months later... still millions of errors |
Unfortunately its not even slated until 29. You will have to manually patch if you want it today, and patch every time you update 😕 |
|
yes, if you check the associated PR that solves the issue, you'll notice it's targeted for nextcloud-30, so not gonna be in production until quite some time. the manual patch works like a charm ! |
For the record, bug fixes typically get backported[1]. The milestone on the main PR is the first release that PR is a candidate for making it into. It'll likely then be backported and accepted into maintenance releases for all then currently still supported major releases. But it does have to be implemented in a final enough way to be committed into the latest dev release, correct. Unfortunately I've been busy in other areas so haven't had the opportunity to revisit the PR to optimize it for a final merge. If someone is so inclined, they're welcome to make an attempt at finishing it up. Otherwise you'll have to patch manually for now or wait for me or someone else to finish it up more officially. [1] https://docs.nextcloud.com/server/latest/admin_manual/release_schedule.html#maintenance-releases |
|
This is exactly the kind of thing that makes Nextcloud just give up a highly unprofessional hobby tinkerer vibe. If Nextcloud wants to convince corporations and governments to invest into using this, you can't have errors that fill up the logs with errors due to a bald-faced bug remain unfixed for months. Oh well, good thing I only run it for myself. |
This isn't the place, but (a) those types of institutions - of any significant size - rarely run the latest of anything so I'd argue this bug isn't yet relevant in those environments; (b) it's easily patched if important regardless of deployment size; (c) the existing patch works fine but is being held so that it can be optimized for performance which is kind of important to precisely the environments you just mentioned; (d) this issue only has four upvotes and no duplicate reports so indications are it's not impacting a lot of people at present so it's not necessarily the highest priority bug. |
|
If I were a PHP developer I'd be happy to volunteer, but I'm a .Net developer. While I appreciate you guys fixing these bugs for the community basically for free, this one could be one of those that could escalate into a bigger problem if left unchecked for too long. If you guys have a Patreon or equivalent, please us know so that if The Important Professional Large Corporations are so worried about it then instead of complaining on here they could talk with their wallets over there and get it prioritized. |
|
If you think running occ:files scan making user's data disappear to them is not highest priority, you have never run a service with a large user base. |
What data loss or data disappearing is occurring with this bug? |
|
The users that are affected will not see any files. Removing the external storage and using occ:files scan again will make them see their files again. Now external storage is not available to the users. |
And with #43794 in place? The fix has been available for over a month. Others have stated it addresses this matter. I just haven't had time to adjust it (so it doesn't rely on serialization), which is an optimization preferred before merging formally.
Commercial clients of Nextcloud GmbH already have a dedicated channel for bug reporting, escalation, etc. It's outside of GitHub via https://portal.nextcloud.com. For the record, I'm a member of the open source Nextcloud project, but I'm not employed by Nextcloud GmbH. |
and thank you for taking the time to fix this issue. I tried it myself and could not figure it out (I'm too new to Nextcloud development) The fix works in term of error rate, I have some external storage connected to my nextcloud but I'm too afraid to remove it and potentially lose it all 😬 or need to upload everything again 😞 , too risky for me to test it. If that works I can connect a new external storage and check the behavior (run a scan, remove it, run a scan, connect it again, run a scan etc), if this has no effect on other external storages already connected 🙄 |
|
The sledgehammer approach I used was to rollback to 27.x and will stay on 27 until 30.x is released which properly addresses the issue. I did taker a look at the patch mentioned above, and as I am NOT a coder and NOT completely knowledgeable enough to sort through the .php file to slap some code into file on the hopes it works, 27.x was the best option available. Additionally, it's not real clear what the advantages are moving from 27.x when the primary use is online file storage. Looking forward to 30.x. |
Now I cant speak for larger deployments, but I was able to apply the patch to the latest 28.x release. I've not seen any issues on my instance (knock on wood)..... |
|
Was hoping the issue would be solved in version 29.0.0 as I read in other related threads... guess not: I NEVER change anything with my nextcloud setup except for version updates which seems to break something different every time. Quite disappointing when the system as a whole has so much potential yet I had the same buggy experience 7 years ago when trying nextcloud for the first time and giving it up shortly thereafter. |
Fixes #43408 Signed-off-by: Josh <josh.t.richards@gmail.com>
Fixes #43408 Signed-off-by: Josh <josh.t.richards@gmail.com>
Fixes #43408 Signed-off-by: Josh <josh.t.richards@gmail.com>
|
Thanks for the patch. I've been dealing with random sync errors for a few months, and after applying the patch to address the log errors, I think the sync errors have gone away, too. I only just applied the patch about an hour ago, so there still might be something else, but so far, it's looking pretty good. |
Fixes #43408 Signed-off-by: Josh <josh.t.richards@gmail.com>
Fixes #43408 Signed-off-by: Josh <josh.t.richards@gmail.com>
Bug description
New / fresh installation of Nextcloud 28.0.2
Steps to reproduce
alias nocc-cldtst01="sudo -u www-data php /var/www/cldtst01.site1.com/html/occ"
Expected behavior
Output
root@X.X.X.X:
$ nocc-cldtst01-site1 files:scan --all$Starting scan for user 1 out of 1 (admin)
Error during scan: Array to string conversion
Error during scan: Array to string conversion
....................
Error during scan: Array to string conversion
Error during scan: Array to string conversion
+---------+-------+-----+---------+---------+--------+--------------+
| Folders | Files | New | Updated | Removed | Errors | Elapsed time |
+---------+-------+-----+---------+---------+--------+--------------+
| 49 | 218 | 0 | 1 | 0 | 84 | 00:00:01 |
+---------+-------+-----+---------+---------+--------+--------------+
root@X.X.X.X:
Installation method
Community Manual installation with Archive
Nextcloud Server version
28
Operating system
Debian/Ubuntu
PHP engine version
PHP 8.2
Web server
Nginx
Database engine version
MariaDB
Is this bug present after an update or on a fresh install?
Fresh Nextcloud Server install
Are you using the Nextcloud Server Encryption module?
Encryption is Disabled
What user-backends are you using?
Configuration report
List of activated Apps
Nextcloud Signing status
Nextcloud Logs
root@X.X.X.X:/var/log/nextcloud$ tail -f cldtst01-site1.log {"reqId":"BsKXnLgxxgk9OrTCg4dN","level":2,"time":"2024-02-06T13:25:37-08:00","remoteAddr":"Z.Z.Z.Z","user":"--","app":"no app in context","method":"POST","url":"/login","message":"Login failed: timv (Remote IP: Z.Z.Z.Z3)","userAgent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36","version":"28.0.2.5","data":[]}Additional info
No response
The text was updated successfully, but these errors were encountered: