Add support for transformation of elasticsearch data document mappings #409
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem:
A problem that's been on our backlog for a long time is sorting results from elasticsearch. Keyword fields are the only fields that can be sorted on natively, and the standard analysis of the fields makes the distinction between "jacob" and "Jacob". Normal string sorting will sort capital letters first, then lowercase. Which means "jacob" will end up at the bottom of the list, miles away from "Jacob".
In order to fix this, the mapping on the index needs to be updated so that the field you sort on is normalized without case. With a keyword normalizer the value of the field remains as it was inputted, but when doing aggregations and sorts, the field uses its normalized version. This is a post that explains how to set up a normalizer to treat keyword values as lowercase:
https://discuss.elastic.co/t/case-insensitive-sort-doesnt-work/143192/7
Solution(s):
I considered a few ways to add this mapping into the moqui elastic facade, from most complicated, to most broad:
I decided that to make this most re-usable and quick to implement, the third option would handle any possible case. If we decided to model field mappings later, we could also add that on top.
The service takes the normally generated mapping for the index and returns it with the same name and an optional settings map. The settings map gets added to the createIndex request. Unfortunately, in order to update analyzers and normalizers in the index settings, the index needs to be closed. We could code that, but it would case a little interruption in the index.
Let me know if you have any questions, suggestions, or want to see an example of a use case for this.