-
Notifications
You must be signed in to change notification settings - Fork 18
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Recommendations app causes slow queries saturating mysql #544
Comments
If you open |
@ChristophWurst thanks! That might make it easier to trigger this and test fixes. Disabled recommendations app for now and server is responsive again. I guess when everyone logs in at the same time in the morning and the dashboards trigger some long running queries we're running out of CPUs. Visiting most recent files is not as common i guess. |
Oh... and also... there were quite a few group folders mentioned in the sloow queries. Might be just because they contain a lot of files or so - but maybe there's some additional complexity there. |
Explain with the given query says it uses the |
Looping in @CarlSchwan since the keyword "recent files" triggered a hit for your work during the performance hackweek or at least I link it to you :) |
The issue is that it's using the search API that is really heavy on the DB level. @icewind1991 improved it quite a bit during the last hackweek so that the number of queries generated decreased by a lot. Unfortunately, the big DB query that does the actual search across all the mountpoints of an user (share, groupfolders, ...) and generating this slow query is still here :( I discussed it a bit with Robin during the hack week and we don't see a way to make it lighter unless we make it less powerful e.g. not searching for mountpoints but only the user storage. This is the same issue with the photo app that is currently timing out in perf testing due to this search. |
@ChristophWurst besides the emojis what is your view on a "lighter" search query with respect to the recommendation app's result-quality? Would this be a viable option (your emoji suggests otherwise, so just asking explicitly)
@CarlSchwan this implies we will have to come up with a fix anyways because even without the recommendations app we would then kill an instance with the new photos which is to be expected to be used by magenta at some point. So who would we need to gather to tackle this? |
It looks like the sort is taking most of the time. I don't know if the database knows to limit the sort to the entries that can actually make it into the top 7. update: looks like we can find this out by setting some parameters for slow query logs: https://mariadb.com/kb/en/filesort-with-small-limit-optimization/ |
Implemented @max-nextcloud's idea in nextcloud/server#33788 Unit tests are failing though, probably because the mtime in the test cases are not correctly configured. I will need to look into |
AFAIK recommend files aren't as useful as we originally thought, so if we do a lighter way and exclude some data it would probably not harm user experience but rather enhance it loading the files becomes faster. As in, rather have 5 not so relevant files in 5s than slightly more relevant files in a timeout. PR: Just to make the situation clear: this app is a consumer of the server API and has no direct influence on how the server generates the list of recent files. This problem is also found in other places, like the Recent page in the Files app. |
From what i heard the +0 was added to prevent the mtime index from being used. Tests on a productions instance look like the + 0 actually does more harm then good here. (>60s instead of 2.5 sec) We are limiting the results in the recommendation app to the 7 most recent. So i think the two approaches compare like this:
If a user has access to a lot of files sorting all of them might well outweight the cost of going through the most recent files and only picking the ones with access. I don't quite understand why we cannot have the best of both worlds. Can't we do away with the edit: (Note that the |
Actually looking at the index used my comparison is wrong... Looks like right now we use the index to exclude non empty folders... Those typically less than 50% of the nodes. |
We're seeing a lot of sloooow queries due to the recommendations app. So many it actually brought the server to a halt.
They all have a similar pattern but differ in lenght.
Here's one example from
mysqldumpslow
. As you can see the query is taking ~ 25 min. on average.query from mysqldump slow
The actual values for the order and limit query are:
This looks very much like the call to
getRecent
inrecommendations/lib/Service/RecentlyEditedFilesSource.php
Lines 47 to 71 in 0342f4c
Given that
$max
is 7 by default.SHOW PROCESSLIST
shows them asCreating sort index
most of the time.So maybe all we are missing is an index on
mtime + '0'
. Or maybe we should do away with the+ '0'
.The text was updated successfully, but these errors were encountered: