#287 - Elastic search connection #307

michael-82 · 2024-06-28T12:00:59Z

This adds searching in an external elastic search instance to the backend. Elastic Search connection can be configured via env parameters (see readme.md).

Currently, this requires manual work in order to get it started. (Setting up elastic search instance, creating and populating indices). There are already issues that address these inconveniences (#289, #288)

In order to get it up and running:

Start an elastic search container (Do NOT create the index from there, but otherwise follow https://github.com/medizininformatik-initiative/feasibility-ontology-elasticsearch )
create the "Elastic Search index for ontology" from the base post at Improved Ontology Search fdpg-plus#11
Download es_content.zip and upload the contents to your elastic search instance. Assuming you are using some kind of rest client (postman, insomnia...), make a POST to /ontology/_bulk (e.g. http://localhost:9200/ontology/_bulk), as body, include the files (one by one). If successful, you should get a json response that should start with "errors:" false. Please contact me if anything fails here...
Start up the backend, setting the appropriate params

Please squash before merging!

codecov · 2024-06-28T12:06:26Z

Codecov Report

Attention: Patch coverage is 96.29630% with 6 lines in your changes missing coverage. Please review.

Project coverage is 80.96%. Comparing base (47b6897) to head (5af6203).

Files	Patch %	Lines
...lity_gui_backend/terminology/es/config/Config.java	0.00%	4 Missing ⚠️
...i_backend/terminology/es/TerminologyEsService.java	97.80%	0 Missing and 2 partials ⚠️

Additional details and impacted files

@@              Coverage Diff              @@
##             develop     #307      +/-   ##
=============================================
+ Coverage      79.72%   80.96%   +1.24%     
- Complexity       527      570      +43     
=============================================
  Files            112      125      +13     
  Lines           2224     2385     +161     
  Branches         133      144      +11     
=============================================
+ Hits            1773     1931     +158     
- Misses           393      394       +1     
- Partials          58       60       +2

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

src/main/java/de/numcodex/feasibility_gui_backend/terminology/es/TerminologyEsService.java

src/test/java/de/numcodex/feasibility_gui_backend/terminology/es/TerminologyEsServiceTest.java

- add a first implementation of a search using elastic search (requires external service) WIP

- add a switch to turn elastic search on or off

- add elastic host parameter

@crossorigin

- add missing @crossorigin annotation on TerminologyEsController - fix return value in swagger config for search (not an array)

- add ontology search result "wrapper" around search results, adding total hits - add requestparams to search endpoint (not doing anything yet)

- move search-logic from TerminologyEsController to TerminologyEsService - read available filter values from es - read available filter fields from config - wip: aggregation handling - wip: which search variant to use - todo: pagination

- use repositories for searching

- use multimatch query for our default query - add criteria-profile-data endpoint - move TerminologyEsService.java to another package - add dtos for elastic search results since mapping is needed from context.code to context (WIP) - still WIP

- add selectable attribute - adapt to context being an object instead of a string - convert entries between the model in elastic search and the api format - add (clumsy) queries considering filters (this is not solved in a sophisticated manor) - add endpoint to query for relatives of entries - add endpoint to retrieve criteria profile data (uiprofile, context and termcodes)

- change valueDefinition entry from String to AttributeDefinition.class in UiProfile

- change TermCode to List<Termcode> in CriteriaProfileData

- add referenceCriteriaSet to AttributeDefinition

- add "reference" to ValueDefinitonType - change datatype for referenceCriteriaSet in AttributeDefinition from CriteriaSet to plain String - remove unused CriteriaSet.java

- add check for availability (currently > 0 = available) - this is not very sophisticated...but it is working - remove unused methods

- Rename TerminologyEsController to TerminologyEsRestController - Add TerminologyEsRestControllerIT

- Add more tests - remove unused OntologySearchResult.java

- Change classes to records - fix parameter list in search test

- Start working on TerminologyEsServiceIT

- Increase Test Coverage

- Update swagger file - rename params on api/v3/terminology/entry/search - add elastic search parameters to readme

- Replace the strange queries for each amount of filters in OntologyListItemEsRepository with queries built with the different query builders from elastic search query dsl

- Add NoArgsConstructor and AllArgsConstructor to UiProfile classes

- Add Debug output when loading a ui profile fails

juliangruendner · 2024-08-12T15:27:56Z

src/main/java/de/numcodex/feasibility_gui_backend/terminology/api/AttributeDefinition.java

@@ -26,11 +31,13 @@ public class AttributeDefinition {
    @JsonProperty("optional")
    private Boolean optional;
    @JsonProperty("allowedUnits")
-    private List<Unit> allowedUnits = new ArrayList<>();
+    private List<TermCode> allowedUnits = new ArrayList<>();


is this really supposed to be TermCode and not Unit - please check with @geloro94

I would move that question to a new issue and merge this one here.

michael-82 linked an issue Jun 28, 2024 that may be closed by this pull request

Elastic search connection #287

Closed

github-advanced-security bot found potential problems Jun 28, 2024

View reviewed changes

michael-82 force-pushed the feature/287-elastic-search-connection branch 2 times, most recently from 18e89fa to 875bca7 Compare July 5, 2024 11:41

github-advanced-security bot found potential problems Jul 5, 2024

View reviewed changes

src/test/java/de/numcodex/feasibility_gui_backend/terminology/es/TerminologyEsServiceTest.java Fixed Show fixed Hide fixed

michael-82 marked this pull request as ready for review July 17, 2024 09:35

github-advanced-security bot found potential problems Jul 18, 2024

View reviewed changes

src/test/java/de/numcodex/feasibility_gui_backend/terminology/es/TerminologyEsServiceTest.java Dismissed Show dismissed Hide dismissed

src/test/java/de/numcodex/feasibility_gui_backend/terminology/es/TerminologyEsServiceTest.java Dismissed Show dismissed Hide dismissed

michael-82 force-pushed the feature/287-elastic-search-connection branch from 54ba3b5 to 3bd1224 Compare August 6, 2024 11:37

michael-82 added 21 commits August 6, 2024 15:07

#287 - Elastic search connection

7ecb6c3

- add a first implementation of a search using elastic search (requires external service) WIP

#287 - Elastic search connection

83fe9f4

- add a switch to turn elastic search on or off

#287 - Elastic search connection

48f748f

- add elastic host parameter

#287 - Elastic search connection

362483e

- add missing @crossorigin annotation on TerminologyEsController - fix return value in swagger config for search (not an array)

#287 - Elastic search connection

911ad1e

- add ontology search result "wrapper" around search results, adding total hits - add requestparams to search endpoint (not doing anything yet)

#287 - Elastic search connection

1836ccd

- move search-logic from TerminologyEsController to TerminologyEsService - read available filter values from es - read available filter fields from config - wip: aggregation handling - wip: which search variant to use - todo: pagination

#287 - Elastic search connection

540b4d6

- use repositories for searching

#287 - Elastic search connection

ad77024

- use multimatch query for our default query - add criteria-profile-data endpoint - move TerminologyEsService.java to another package - add dtos for elastic search results since mapping is needed from context.code to context (WIP) - still WIP

#287 - Elastic search connection

be96bd7

- change valueDefinition entry from String to AttributeDefinition.class in UiProfile

#287 - Elastic search connection

d0280c6

- change TermCode to List<Termcode> in CriteriaProfileData

#287 - Elastic search connection

1a487eb

- add referenceCriteriaSet to AttributeDefinition

#287 - Elastic search connection

fd7722d

- add "reference" to ValueDefinitonType - change datatype for referenceCriteriaSet in AttributeDefinition from CriteriaSet to plain String - remove unused CriteriaSet.java

#287 - Elastic search connection

54f297e

- add check for availability (currently > 0 = available) - this is not very sophisticated...but it is working - remove unused methods

#287 - Elastic search connection

879af53

- Rename TerminologyEsController to TerminologyEsRestController - Add TerminologyEsRestControllerIT

#287 - Elastic search connection

f0906bf

- Add more tests - remove unused OntologySearchResult.java

#287 - Elastic search connection

cab4644

- Change classes to records - fix parameter list in search test

#287 - Elastic search connection

ef25061

- Start working on TerminologyEsServiceIT

#287 - Elastic search connection

f28e446

- Increase Test Coverage

#287 - Elastic search connection

07a4ddb

- Update swagger file - rename params on api/v3/terminology/entry/search - add elastic search parameters to readme

#287 - Elastic search connection

1273150

- Replace the strange queries for each amount of filters in OntologyListItemEsRepository with queries built with the different query builders from elastic search query dsl

michael-82 force-pushed the feature/287-elastic-search-connection branch from 3bd1224 to 1273150 Compare August 6, 2024 13:07

michael-82 requested a review from juliangruendner August 6, 2024 13:36

#287 - Elastic search connection

f93befc

- Add NoArgsConstructor and AllArgsConstructor to UiProfile classes

michael-82 force-pushed the feature/287-elastic-search-connection branch from 0c9a8bb to f93befc Compare August 9, 2024 06:48

#287 - Elastic search connection

5af6203

- Add Debug output when loading a ui profile fails

juliangruendner reviewed Aug 12, 2024

View reviewed changes

juliangruendner approved these changes Aug 12, 2024

View reviewed changes

michael-82 merged commit b8a44d7 into develop Aug 13, 2024
8 checks passed

michael-82 mentioned this pull request Aug 13, 2024

AllowedUnits in AttributeDefinition - Unit or TermCode #322

Closed

michael-82 deleted the feature/287-elastic-search-connection branch August 14, 2024 07:43

michael-82 added this to the Next Release milestone Aug 21, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

#287 - Elastic search connection #307

#287 - Elastic search connection #307

michael-82 commented Jun 28, 2024 •

edited

Loading

codecov bot commented Jun 28, 2024 •

edited

Loading

juliangruendner Aug 12, 2024

michael-82 Aug 13, 2024

#287 - Elastic search connection #307

#287 - Elastic search connection #307

Conversation

michael-82 commented Jun 28, 2024 • edited Loading

codecov bot commented Jun 28, 2024 • edited Loading

Codecov Report

juliangruendner Aug 12, 2024

Choose a reason for hiding this comment

michael-82 Aug 13, 2024

Choose a reason for hiding this comment

michael-82 commented Jun 28, 2024 •

edited

Loading

codecov bot commented Jun 28, 2024 •

edited

Loading