Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature] GeoHashGrid Aggregation on Geo Shape field #4071

Closed
5 tasks done
navneet1v opened this issue Aug 1, 2022 · 15 comments
Closed
5 tasks done

[Feature] GeoHashGrid Aggregation on Geo Shape field #4071

navneet1v opened this issue Aug 1, 2022 · 15 comments
Assignees
@navneet1v
Labels
enhancement Enhancement or improvement to existing feature or request feature New feature or request Geospatial Search:Aggregations v2.9.0 'Issues and PRs related to version v2.9.0'

Comments

@navneet1v
Copy link
Contributor

navneet1v commented Aug 1, 2022 •
edited
Loading

Description

This issue tracks the implementation of GeoHashGrid Aggregation on Geo Shape field.

Tasks:

  • Refactor code of GeoHashGrid Aggregation on GeoPoint from server to modules/geo folder.
  • Implement the GeoHashGrid on GeoShape field
    • Implement the code to find the intersection with the Hashes and return the results in correct bucket.
    • Integration Tests
    • Unit Tests

Dependency

#4049

Useful Links

  1. RFC for All Aggregations on GeoShape: [RFC] - Implement Aggregations on Geo Shape Field geospatial#84
  2. Low Level Implementation Proposal: [Proposal]: Low Level Implementation Details for Geo Shape Aggregation and future Geo related Features geospatial#92
  3. Enabling Doc values for GeoShape: Enable Doc values for GeoShape Field #4049
  4. All Aggregation Implementation issue: Implement Aggregations on Geo Shape geospatial#104
  5. RFC: [RFC] - Geo Hash Grid Aggregation on Geo Shape Field geospatial#193
@navneet1v navneet1v added enhancement Enhancement or improvement to existing feature or request feature New feature or request Search:Aggregations Geospatial labels Aug 1, 2022
@navneet1v navneet1v self-assigned this Aug 1, 2022
@navneet1v navneet1v mentioned this issue Aug 1, 2022
Open
7 tasks
navneet1v added a commit to navneet1v/OpenSearch that referenced this issue Aug 4, 2022
@navneet1v
Refactored the src and test of GeoHashGrid and GeoTileGrid Aggregatio…
7d02df1 
…ns on GeoPoint from server folder to geo module.(opensearch-project#4071) (opensearch-project#4072)

The changes also include:
   * Updated Search plugin to provide the interface so that plugins can also register the compositie aggregations
   * Added YAML test for the geo_grid, geo_tile and composite aggregation

Signed-off-by: Navneet Verma <navneev@amazon.com>
@navneet1v navneet1v mentioned this issue Aug 4, 2022
Closed
5 tasks
@navneet1v navneet1v mentioned this issue Aug 9, 2022
Merged
5 tasks
navneet1v added a commit to navneet1v/OpenSearch that referenced this issue Aug 21, 2022
@navneet1v
Refactored the src and test of GeoHashGrid and GeoTileGrid Aggregatio…
da7dbea 
…ns on GeoPoint from server folder to geo module.(opensearch-project#4071) (opensearch-project#4072)

The changes also include:
   * Updated Search plugin to provide the interface so that plugins can also register the compositie aggregations
   * Added YAML test for the geo_grid, geo_tile and composite aggregation

Signed-off-by: Navneet Verma <navneev@amazon.com>
nknize pushed a commit that referenced this issue Aug 22, 2022
@navneet1v
Refactored the src and test of GeoHashGrid and GeoTileGrid Aggregatio…
5433056 
…ns on GeoPoint from server folder to geo module.(#4071) (#4072) (#4180)

The changes also includes:
   * Updated Search plugin to provide the interface so that plugins can also register the composite aggregations
   * Added YAML test for the geo_grid, geo_tile and composite aggregation

Signed-off-by: Navneet Verma <navneev@amazon.com>
navneet1v added a commit to navneet1v/OpenSearch that referenced this issue Aug 22, 2022
@navneet1v
Refactored the src and test of GeoHashGrid and GeoTileGrid Aggregatio…
2a1fa61 
…ns on GeoPoint from server folder to geo module.(opensearch-project#4071) (opensearch-project#4072) (opensearch-project#4180)

The changes also includes:
   * Updated Search plugin to provide the interface so that plugins can also register the composite aggregations
   * Added YAML test for the geo_grid, geo_tile and composite aggregation

Signed-off-by: Navneet Verma <navneev@amazon.com>
@navneet1v navneet1v mentioned this issue Aug 22, 2022
Merged
5 tasks
nknize pushed a commit that referenced this issue Aug 30, 2022
@navneet1v
Refactored the src and test of GeoHashGrid and GeoTileGrid Aggregatio…
da76f2a 
…ns on GeoPoint from server folder to geo module.(#4071) (#4072) (#4180) (#4281)

The changes also includes:
   * Updated Search plugin to provide the interface so that plugins can also register the composite aggregations
   * Added YAML test for the geo_grid, geo_tile and composite aggregation

Signed-off-by: Navneet Verma <navneev@amazon.com>
@navneet1v navneet1v added the v2.5.0 'Issues and PRs related to version v2.5.0' label Sep 7, 2022
@navneet1v navneet1v changed the title Implementation of GeoHashGrid Aggregation on Geo Shape field GeoHashGrid Aggregation on Geo Shape field Dec 2, 2022
@navneet1v navneet1v changed the title GeoHashGrid Aggregation on Geo Shape field [Feature] GeoHashGrid Aggregation on Geo Shape field Dec 2, 2022
@navneet1v navneet1v mentioned this issue Dec 5, 2022
Open
4 tasks
@navneet1v navneet1v mentioned this issue Dec 16, 2022
Merged
6 tasks
@saratvemulapalli
Copy link
Member

saratvemulapalli commented Jan 5, 2023 •
edited
Loading

@navneet1v This issue is tagged for 2.5, can you make it as the code freeze 1/10. I see few PRs out. Do you need help getting them moving?

@navneet1v
Copy link
Contributor Author

@saratvemulapalli removing the tag and marking as 2.6

@navneet1v navneet1v added v2.6.0 'Issues and PRs related to version v2.6.0' and removed v2.5.0 'Issues and PRs related to version v2.5.0' labels Jan 5, 2023
@navneet1v navneet1v added v2.7.0 and removed v2.6.0 'Issues and PRs related to version v2.6.0' labels Jan 24, 2023
@anasalkouz
Copy link
Member

Hey @navneet1v, is this on track for 2.7. The code freeze date is April 17, 2023.

@navneet1v navneet1v added v2.8.0 'Issues and PRs related to version v2.8.0' and removed v2.7.0 labels Apr 11, 2023
@navneet1v
Copy link
Contributor Author

Hey @navneet1v, is this on track for 2.7. The code freeze date is April 17, 2023.

No. This is moved to 2.8

@vamshin vamshin added v3.0.0 Issues and PRs related to version 3.0.0 and removed v2.8.0 'Issues and PRs related to version v2.8.0' labels May 15, 2023
@janhoy
Copy link

janhoy commented Jun 2, 2023

In roadmap this is listed under 2.9.0, but this has label 3.0.0. This has been pushed 2.5 -> 2.6 -> 2.7 -> 2.8 -> 2.9 now, a bit hard to plan. We need this to migrate from ES.

@navneet1v
Copy link
Contributor Author

@janhoy I completely understand the concern. The current geoshape aggregation launch is blocked because of the Visualization work required to support them on Dashboard Maps(Ref: opensearch-project/dashboards-maps#59). Hence this feature is being pushed for so many releases.

To understand your usecase more, I want to know do you need this aggregation on OpenSearch Dashboards also? or your usecase is only about getting the backend aggregation results from OpenSearch cluster?

Adding the PM for this feature @nandi-github and @vamshin to help prioritize the visualization work.

@janhoy
Copy link

janhoy commented Jun 2, 2023

Thanks for answering. We only needs the aggregation in backend and Java HighLevel Rest client. Can you consider releasing the api first, then dashboard visuals later?

@navneet1v
Copy link
Contributor Author

@janhoy thanks this is very helpful. cc: @nandi-github @vamshin

@vamshin vamshin added v2.9.0 'Issues and PRs related to version v2.9.0' and removed v3.0.0 Issues and PRs related to version 3.0.0 labels Jun 13, 2023
This was referenced Jun 27, 2023
Merged
Closed
@dreamer-89
Copy link
Member

@navneet1v @vamshin : Is this in track for 2.9 release. If not, can you please remove (or add v2.10.0) label.

@navneet1v
Copy link
Contributor Author

@dreamer-89 this has been merged and will be released with 2.9.

Closing this issue, the changes are merged to 2.9 branch of OpenSearch

@lbenedetto
Copy link

Does this issue only cover supporting it on the server side? It does not seem to be supported from the client side.
I'm expecting to find some equivalent to AggregationBuilders.geohashGrid that I have in Elasticsearch.

I see GeoHashGridAggregationBuilder in this repo, but it is in the modules directory and only stuff in the libs directory seems to be accessible from the client.

Maybe I'm doing something wrong? I don't really understand how this repo is structured.

@janhoy
Copy link

janhoy commented Sep 22, 2023

I find the same, the feature is perhaps in backend, but not in high-level-rest client.

@navneet1v
Copy link
Contributor Author

navneet1v commented Sep 22, 2023 •
edited
Loading

@janhoy , @lbenedetto

Does this issue only cover supporting it on the server side?

GeoHashgrid on geoshapes is available in OpenSearch but not in OpenSearch dashboards.

From the high level client standpoint, the high level client of OpenSearch is on deprecation path. Hence you don't see this aggregation enabled in High Level client. See note on top of every high level client: https://opensearch.org/docs/latest/clients/python-high-level/

You can use low level rest client to hit the _search request with the geohash aggregation on geoshape.

@janhoy
Copy link

janhoy commented Sep 23, 2023

Ah, bummer - that prevents us from a stepwise migration from ES with High-level client. Our plan was to first move to OpenSearch, basically changing the Java imports, and then migrating to a new Java client. This agg is part of bigger Builder patterns, so our only chance now is to make our own custom GeoHashGridAggregationBuilder :( But thanks anyway for the backend work, I think this will be within reach.

@lbenedetto
Copy link

Unfortunate, since spring-data-opensearch currently only supports the deprecated high-level client

opensearch-project/spring-data-opensearch#19

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@navneet1v navneet1v

Labels
enhancement Enhancement or improvement to existing feature or request feature New feature or request Geospatial Search:Aggregations v2.9.0 'Issues and PRs related to version v2.9.0'
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@janhoy @lbenedetto @navneet1v @saratvemulapalli @anasalkouz @vamshin @dreamer-89