Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve Search performance by introducing per-tag index hashes #555

Closed
Difegue opened this issue Nov 26, 2021 · 2 comments
Closed

Improve Search performance by introducing per-tag index hashes #555

Difegue opened this issue Nov 26, 2021 · 2 comments
Assignees
Labels

Comments

@Difegue
Copy link
Owner

Difegue commented Nov 26, 2021

I don't think search can get faster than it currently is if we keep looking at every file each time.
The most simple solution imo is to build index tables for each namespace: The structure for this already exists in the stats minion job, it's mostly a matter of extending it so that Search can rely on those stat tables as well.

In case the stat tables aren't built (first searches), we'll probably have to fallback on the existing look-in-every-redis-key algorithm.

@Difegue Difegue added this to the 2021 User Survey Requests milestone Nov 26, 2021
@Difegue Difegue self-assigned this Nov 26, 2021
@Difegue Difegue pinned this issue Nov 26, 2021
@Difegue Difegue unpinned this issue Dec 9, 2021
@Difegue Difegue pinned this issue Jan 9, 2022
Difegue added a commit that referenced this issue Dec 16, 2022
title sorting is optimized thanks to the title sorted set, other namespaces less so.
@Difegue
Copy link
Owner Author

Difegue commented Dec 16, 2022

Some notes on the search rewrite that's currently landing (wow, it's been more than a year...):

  • Spaces are no longer a delimiter for different tags in a search prompt, commas should now be used instead. This matches what the autocompletion box does and how metadata as a whole is entered into the app, so I don't think this will cause too much pain.
  • Tags will for now only be accepted as exact matches without any wildcards, aka category:doujinshi (or doujinshi if no namespaces). Spaces before/after a tag will be accepted. You might have to rewrite some of your dynamic categories.
  • Title searches will work the same as before (quotes or $ for an exact match, ? * _ % as wildcards

The new system makes heavy use of Redis sets and sorted sets, which are all lodged in a separate, third database.
See the DB Architecture document. (the config keys moving to a fourth database is a pending change.)

@Difegue
Copy link
Owner Author

Difegue commented Dec 16, 2022

Additional comment: Search now requires the index tables to be built (does so on app start, and then updates them as tags/titles/etc are modified), so large instances might take a few mins to get a usable search again after updating.

I haven't added a visual warning for this, but it'd certainly be a possibility.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant