Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Meta] Refactor Kibana querying infrastructure ("courier") #34022

Closed
22 tasks
lukasolson opened this issue Mar 27, 2019 · 1 comment
Closed
22 tasks

[Meta] Refactor Kibana querying infrastructure ("courier") #34022

lukasolson opened this issue Mar 27, 2019 · 1 comment
Assignees
Labels
Feature:Search Querying infrastructure in Kibana Meta

Comments

@lukasolson
Copy link
Member

lukasolson commented Mar 27, 2019

Related: #29215, #20364

Summary

Currently, our querying infrastructure (a.k.a. "courier") has been around since the early days of 4.x. Over time, lots of functionality has been stuffed/mashed in, not always considering best practices. As a result, courier has lots of complicated functionality and is a cause of a lot of technical debt. Also, it doesn't have a simple or stable API, so usage by core and 3rd party plugins is limited, resulting in a disjointed system for data access across plugins. (Things like Timelion and TSVB don't use courier.)

There are several steps we will take in refactoring courier:

Create a server API (7.4)

Add support for request cancellation (7.4, pending elastic/elasticsearch#43332)

(In progress: #39963) Kibana users & admins would like queries to be cancellable in many situations:

  • On long queries (perhaps after prompting user)
  • On window unload/route change
  • On resubmit/refresh/update/time range change/agg config change/panel config change/panel resize

Elasticsearch has this ability, yet courier has not supported it. We'd like to add these capabilities to courier and allow apps to take advantage of query cancellation.

Refactor search source to use the new server APIs and remove Elasticsearch proxy (7.5)

  • We will refactor the existing search source to use the new server-side APIs introduced above, which will remove the need for our Elasticsearch proxy, at which point we will remove it. In this step we will still be using our existing filter/agg config models.
    • Switch management of index patterns over to the new data service.
    • Switch vega over to using the new data service.
    • Remove usage of es service to keep session alive in security plugin.
    • Remove 'es' angular service.
    • Remove elasticsearch/{indexpattern}/search endpoint
    • Switch usage of elasticsearch/_msearch endpoint in tutorials to data.search service
    • Switch usage of elasticsearch/_msearch endpoint in field_editor to data.search service
    • Remove elasticsearch/_msearch endpoint

Associate search requests with a user/saved object (7.5)

Add support for query progress (7.6, pending Elasticsearch ability to do so)

  • As soon as Elasticsearch supports it, we would like to expose the ability to provide consumers with the current progress (as a percentage of complete) of the query execution.

Create a high-level API with abstractions (7.7)

  • Refactor & replace the agg config model with something better (TBD)
  • Create Typescript interfaces/models for the following:
    • Specifying which fields to query (scripted fields, stored fields, _source fields, _score, aggregation results)
    • Filtering (on fields, scripted fields, aggregation results)
    • Sorting (on fields, scripted fields, aggregation results)
    • Paging (on fields, scripted fields, aggregation results)
    • Highlighting

Related issues:

Refactoring

Performance/batching

Query cancellation

Elasticsearch query options

@lukasolson
Copy link
Member Author

Closing in favor of https://github.com/elastic/kibana/projects/43.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Feature:Search Querying infrastructure in Kibana Meta
Projects
None yet
Development

No branches or pull requests

1 participant