Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrades Elasticsearch integration to support v8+ #59

Merged
merged 3 commits into from
Sep 17, 2024
Merged

Upgrades Elasticsearch integration to support v8+ #59

merged 3 commits into from
Sep 17, 2024

Conversation

jerluc
Copy link
Member

@jerluc jerluc commented Sep 13, 2024

Overview

The main change included here is around dropping support for Elasticsearch v6 and adding new support for Elasticsearch v8 (there is no plan at the moment to support multiple simultaneous versions). This is meant to both support major infrastructure changes @ Station A while also addressing some historical feedback we've gotten here (e.g. #51 and #56).

What's changed?

In order to support Elasticsearch v8, quite a few things were needed:

  • Per Upgrade Elasticsearch client libraries #56, the olivere/elastic library we were originally using is no longer maintained; instead, this implementation uses the official Elasticsearch Go client
  • The new implementation now makes use of Elasticsearch's native vector tile search functionality; this has some notable implications compared to the original scroll-based implementation (though I think these are reasonable trade-offs for now):
    • Tiles are now capped at a maximum of 10,000 features
    • Feature properties are flattened by Elasticsearch before tilenol remaps them
    • Some property values (e.g. arrays) are not supported well
    • Vector tiles are actually re-encoded in order to remap fields and rename layers appropriately--this may become a source of performance issues in the future
  • Breaking changes to the YAML configuration for Elasticsearch layer sources:
    • host + port are now combined into the hosts array
    • New username and password options for clusters secured via HTTP basic auth
    • Removed the sniff option

Other notables

  • A couple of changes to the examples/index.html demo map page
    • Feature popup window now automatically overflows when properties run long
    • Mapbox access token must be set via the mapbox_access_token query parameter
    • All other query parameters get forwarded to tilenol so you can try out additional tilenol parameters such as q=<QUERY-STRING-QUERY> or s=<EXTRA-FIELD-MAPPING>
  • Updated the Docker images to Go v1.22 and Alpine v3.20

@jerluc jerluc added source:elasticsearch Relating to the Elasticsearch backend source dependencies Pull requests that update a dependency file labels Sep 13, 2024
@jerluc jerluc linked an issue Sep 13, 2024 that may be closed by this pull request
Copy link

@schimmy schimmy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just one nit

Dockerfile Show resolved Hide resolved
@jerluc jerluc merged commit 2accfbc into main Sep 17, 2024
2 checks passed
@jerluc jerluc deleted the es-8 branch September 17, 2024 23:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dependencies Pull requests that update a dependency file source:elasticsearch Relating to the Elasticsearch backend source
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Upgrade Elasticsearch client libraries
2 participants