Refactor and enhance issue indexer to support both searching, filtering and paging #26012
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
wxiaoguang
added
the
status/blocked
techknowlogick
added
reviewed/wait-merge
GiteaBot
removed
the
reviewed/wait-merge
zjjhot
added a commit
to zjjhot/gitea
that referenced
this pull request
Jul 31, 2023
* upstream/main: (26 commits) Add 'Show on a map' button to Location in profile, fix layout (go-gitea#26214) Use shared template for webhook icons (go-gitea#26242) Reduce margins on user settings page, introduce `flex-container` (go-gitea#26046) Refactor and enhance issue indexer to support both searching, filtering and paging (go-gitea#26012) Show image size on view page (go-gitea#25884) Fix pull request check list is limited (go-gitea#26179) Fix API leaking Usermail if not logged in (go-gitea#25097) [skip ci] Updated licenses and gitignores Fix typo in metadata (go-gitea#26207) Update js and py dependencies (go-gitea#26243) De-emphasize issue sidebar buttons (go-gitea#26171) Don't autosize textarea in diff view (go-gitea#26233) Add `/public/assets` to `.ignore` (go-gitea#26232) Fix attachment clipboard copy on insecure origin (go-gitea#26224) Fix commit compare style (go-gitea#26209) Fix unable to display individual-level project (go-gitea#26198) Fix access check for org-level project (go-gitea#26182) Fixed incorrect locale references (go-gitea#26218) Use calendar icon for `Joined on...` in profiles (go-gitea#26215) Add changelog for 1.20.2 (go-gitea#26208) ...
silverwind
pushed a commit
that referenced
this pull request
Aug 15, 2023
related: #26012 ### Bugs 1. missing project filter on the issue page. https://github.com/go-gitea/gitea/blob/1e76a824bcd71acd59cdfb2c4547806bc34b3d86/modules/indexer/issues/dboptions.go#L11-L15 3. incorrect SQL condition: some issue does not belong to a project but exists on the project_issue table. https://github.com/go-gitea/gitea/blob/f5dbac9d36f1678b928bee04e85fbd045c725698/models/issues/issue_search.go#L233 ### Before:  ### After:  --------- Co-authored-by: Giteabot <teabot@gitea.io>
lunny
pushed a commit
that referenced
this pull request
Aug 16, 2023
Fix #26536 Follow #26012 Whatever the comment type is, always update the issue indexer. So the issue indexer will be updated when there is a change in Status, Assignee, Label, and so on. I added the logic for `NotifyUpdateComment`, but missed it for `NotifyCreateIssueComment` and `NotifyDeleteComment`.
This was referenced Aug 16, 2023
lunny
pushed a commit
that referenced
this pull request
Aug 16, 2023
CaiCandong
added a commit
to CaiCandong/gitea
that referenced
this pull request
Aug 17, 2023
related: go-gitea#26012 1. missing project filter on the issue page. https://github.com/go-gitea/gitea/blob/1e76a824bcd71acd59cdfb2c4547806bc34b3d86/modules/indexer/issues/dboptions.go#L11-L15 3. incorrect SQL condition: some issue does not belong to a project but exists on the project_issue table. https://github.com/go-gitea/gitea/blob/f5dbac9d36f1678b928bee04e85fbd045c725698/models/issues/issue_search.go#L233   --------- Co-authored-by: Giteabot <teabot@gitea.io>
KN4CK3R
pushed a commit
that referenced
this pull request
Aug 19, 2023
Backport #26490 related: #26012 1. missing project filter on the issue page. https://github.com/go-gitea/gitea/blob/1e76a824bcd71acd59cdfb2c4547806bc34b3d86/modules/indexer/issues/dboptions.go#L11-L15 2. incorrect SQL condition: some issue does not belong to a project but exists on the project_issue table. https://github.com/go-gitea/gitea/blob/f5dbac9d36f1678b928bee04e85fbd045c725698/models/issues/issue_search.go#L233  
techknowlogick
pushed a commit
that referenced
this pull request
Aug 23, 2023
This PR has multiple parts, and I didn't split them because it's not easy to test them separately since they are all about the dashboard page for issues. 1. Support counting issues via indexer to fix #26361 2. Fix repo selection so it also fixes #26653 3. Keep keywords in filter links. The first two are regressions of #26012. After: https://github.com/go-gitea/gitea/assets/9418365/71dfea7e-d9e2-42b6-851a-cc081435c946 Thanks to @CaiCandong for helping with some tests.
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fix #24662.
Replace #24822 and #25708 (although it has been merged)
Background
In the past, Gitea supported issue searching with a keyword and conditions in a less efficient way. It worked by searching for issues with the keyword and obtaining limited IDs (as it is heavy to get all) on the indexer (bleve/elasticsearch/meilisearch), and then querying with conditions on the database to find a subset of the found IDs. This is why the results could be incomplete.
To solve this issue, we need to store all fields that could be used as conditions in the indexer and support both keyword and additional conditions when searching with the indexer.
Major changes
IndexerDatato include all fields that could be used as filter conditions.Search(ctx context.Context, kw string, repoIDs []int64, limit, start int, state string)toSearch(ctx context.Context, options *SearchOptions), so it supports more conditions now.issueIndexerQueue. UseIndexerMetadatainstead ofIndexerDatain case the data has been updated while it is in the queue. This also reduces the storage size of the queue.SearchOptions. Also, update the data versions.issues.SearchIssueIDsByKeywordinmodelsto avoid confusion where is the entry point to search issues.