-
-
Notifications
You must be signed in to change notification settings - Fork 4.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Files search takes a long time with unified search #23835
Comments
Searching cancel other searches. |
It's mainly slow because the files search implementation doesn't paginate. So you'd always build the full results list on the back-end before it's chopped into the individual pages. This doesn't scale, as you observed. Our instance just has a lot of files and shares, hence the slowness. |
@ChristophWurst non-pagination is NOT the problem: Pasting Executing a query directly on the PostgreSQL 12.0 server (2.6e6 rows in oc_filecache):
This makes NC file search at least 4 times slower than the database full table search. If the search term is not entered by copy/paste, but typed in, things get much worse, since each character will trigger a new query on the server, consuming more resources (when typing fast, the final search request will take 16s). @skjnldsv A new search does not cancel the running queries in the database backend. |
Right, it cancel the frontend request of course. |
I enabled log_min_duration_statement=800 in the PostgreSQL Server, and found that each search will execute the same query four times, which explains the x4 observed above. |
If there was pagination the query would include more But yeah, |
Not really, if the result set is small, or not at all when using window functions, since the time consuming part is the sequential scan, not the result set transfer. For PostgreSQL, speed could be improved by not using But there are fruits hanging much lower:
More advanced:
|
I found the reason why the database was hammered with the same query 4x for a single search. They correspond with four shares from the same user. These result in four MountPoints, having different different roots on the same physical storage (same numericId). Unfortunately, the filtering code is buried very deep in CacheWrapper, and each wrapper retrieves the full data from its Cache instance before filtering. |
Suggest re-titling this issue in line with #24029 , since this is a regression in behavior from NC19. At best, we have lost the "fast search" capability. At worst, we've lost "search" alltogether. My personal use case: updated to NC20 last night. Today, any search query will peg the mysql server and make nextcloud unresponsive. Having just performed a edit: it works... just very very slowly. > 10 seconds to search
|
#25136 should help with search performance, how much of a difference it makes will depend greatly on setup specifics. |
A quick search just in the current folder, like the old search this - basically adding a search provider that is named 'current folder' would do the trick to at least give some results quickly, covering many cases, right? what do you think @jancborchardt |
I think there's something more; if people get frustrated by getting no result they change the search, but the old search continues to put load on the DB. We saw a load of 60 on the DB server just from a few users trying to use search and getting frustrated with it. Why would the search continue if the user moves on (to another search or manually looking for the files)? |
Good point. That then even applies to a search on the users and apps pages: Search for a user or an app, find quickly what you've been looking for (since those pages instantly narrow down), but have the unrelated unified search load remaining in the background. |
We're having increased frustration levels with the search issue (big instance as in many files); is there anything we can do to patch this, either to improve performance back to NC19 levels or limit search to specific groups - even disable, unless that prevents user lookups for sharing. |
NB is search much faster in NC21? |
There were optimizations as far as I know, but most of the technical bottlenecks are yet to be improved. |
Some improvements fresh out of the oven: #26049 |
Well that improvement is merely a drop in the ocean. It limits the amount of data being transferred (if not cancelled anyway by the next longer search term), but will not tackle the multitude of huge database queries. |
Well, I tried to help and keep you updated. |
@ChristophWurst updates are appreciated. |
I can't tell because I don't work on those internals. |
Subscribing here. I can confirm that search in 19 was almost instant, and now it takes dozens of seconds at times, which is suboptimal. I did think increasing MySQL memory and cache size would make a difference, but alas it didn't. |
@gennaios if you can provide the SQL it will help a lot. I or another php-head can figure out how to work that into the existing PHP... |
@ohthehugemanatee I might very possibly be able to sometime though I may not be able to get to it for a little while. If you or someone could provide the current SQL for record insertion and selection, I'd have an idea of the relevant needed columns and table structure. SQLite likely could use an external content table. In such a case, one needs manaually manage: an additional insert into the FTS table, and with any change, first grab old value before change (such as rename), provide such values for a delete statement, then insert new. Or for delete, merely grab value from file table and delete from FTS before deleteing from file. MySQL and PostgreSQL I believe are easier. Concerning other areas where FTS could be used, I recall some issue for contacts where one ended up patching and implementing himself. One maybe could track that down and possibly merge. |
21.0.9 |
Tnxs! Im gonna check later. |
In my schema |
This is the diff with my running version. $ diff my_filecache git_filecache
|
Hi I'm wondering if my Issue is related. I'm running Nextcloud 25rc2 and I have this querry:
And 5 other like this running for 19 hours now. Somewhere yesterday I searched for "icons.png". |
Hi, please update to at least 23.0.12 and report back if it fixes the issue. Thank you! |
I think this is good on 25 already. |
Thanks for testing! |
For what it's worth, I think it was also good on 24, as I was running it for a few weeks before 25 was stable. Thanks for checking. |
For those that have thousands of files and potentially many search results with each search, displaying five at a time is not an ideal solution. Does such merely mask the issue? |
I agree with the comment about the UI and only showing five results at a time, but it is unrelated to the performance issues or fixes. The performance issues were there when the UI was already only showing 5 results at a time, unless I'm mistaken. Nothing changed there. It's just the performance that improved. Edit: still, I'd be happy with a way to open search results in a proper, long list format. Like it used to be before it got reduced to showing results in the search panel alone. |
I second this. The current small search panel is not very user friendly. It's small and doesn't show the full names/file paths of files when you search, and is clunky to scroll through. Furthermore, if you type a search then hit enter ↩, it'll automatically open the first result, which is counter-intuitive. It'd be great to have a full search page available. Gmail gives a good example of good search usability. Showing a popup window initially, and then another window if you hit enter. |
Can I second this too ? second seconder? or a thirder? The UI has some room for improvement. |
Depending on the amount of files, the file search component of unified search takes a long time to get results back. It’s the slowest of all – Talk, Mail and everything loads before. On our own instance, it takes roughly 40 seconds.
Also, there is no x in the input field during loading so it’s impossible to cancel the search – you have to wait for it to finish to start a different one. To fix this, we could replace the spinner with an x on hover/focus.
Gif for illustration, showing how it takes 40 seconds:
(This is very related to the direct filtering of the current folder, which also doesn’t work: #23432 )
cc @nextcloud/search
The text was updated successfully, but these errors were encountered: