feat(strucutred-properties): structured properties delete and schema change support

[david-leifker]

Structured Properties Delete & Breaking Change Support

• Structured property indexing change to include a version and type field

  • Backwards compatible with legacy non-versioned properties
  • Version must be monotonically increasing by lexicographic case insensitive sort
  • Backwards incompatible changes allowed as long as version is increased
  • Old values are not removed, on breaking change target ES index field changes
  • Only the single latest version is use-able in the ES index

• Structured property hard delete implemented via MCP SideEffect post transaction

  • Relies on entity search to identify and generate Patch MCPs for existing values applied to entities
  • Hard delete on the structured property removes all versions and types
  • Index mapping clean-up. Requires reindex after removal of all non-active values.

• Bug Fixes

  • Filtering on string structured property values

Old Index Field Path
structuredProperties.<qualified name>
New Index Field Path
structuredProperties._versioned.<qualified name>.<version>.<type>

<qualified name> - Structured property qualified name with . replaced with _ to be compatible with ES index field naming restrictions.

Checklist

• The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
• Links to related issues (if applicable)
• Tests for the changes have been added/updated (if applicable)
• Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
• For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

[Nbagga14]

@david-leifker
after this code change we are facing the particular issue regarding case insensitive searches , can you please help me with the fix?
We are encountering an issue with the /entities?action=search endpoint where search results are returned only if there is an exact case match. For instance, if we search for data indexed with the value "Test", results are returned only when the search query is "Test". Queries such as "TEST" or "test" do not return any results. We have attempted to modify the ELASTICSEARCH_QUERY_EXACT_MATCH_CASE_FACTOR environment variable to both 0.0 and 1.0, but this has not resolved the issue. Could you please advise on how to configure the search to be case-insensitive, allowing us to fetch results regardless of the case of the indexed data? Thank you.