Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE]Move PPL Language Spec Outside SQL As an independed Repository #2821

Open
YANG-DB opened this issue Jul 11, 2024 · 1 comment
Open
Labels
enhancement New feature or request PPL Piped processing language

Comments

@YANG-DB
Copy link
Member

YANG-DB commented Jul 11, 2024

Is your feature request related to a problem?

Description

Piped Processing Language (PPL), enables users with exploration and discovery of their data, and finding search patterns in data stored in multiple locations (S3, OpenSearch, Prometheus) using a set of commands delimited by pipes (|).

During the past year the SQL/PPL team was focusing on the following tasks:

  • Transforming PPL to become OpenSearch default query language (specifically for logs/traces/metrics signals)
  • Promoting PPL as a viable candidate for the proposed CNCF Observability universal query language.
  • Seamlessly Interact with different datasources such as S3 / Prometheus / data-lake leveraging spark execution.
  • Using spark's federative capabilities as a general purpose query engine to facilitate complex queries including joins
  • Improve and promote PPL to become extensible and general purpose query language to be adopted by the community

For historical reasons, PPL language specifications is currently located in the OpenSearch SQL repository.
In addition the PPL specifications code & documents are not present as an independed (jar) artifact but is bundled with the SQL plugin as an OpenSearch Zip file.

History of PPL

PPL has become a general purpose pipeline language that finds attraction and usage in many places in the log analytics echo-system.
Its originated as a language that has a dedicated OpenSearch driver that was the only execution engine that could run the language inside opensearch.

Since PPL has evolved and is now able to run on top of Spark as a fully qualified query language.

What solution would you like?

ppl logical architecture

This PR has the goal of decoupling the PPL language specifications and documentation away from the OpenSearch SQL Plugin and move it (back) into the PPL dedicated repository.

This repository should contain the following:

  • ANTLR specifications
  • Documentations
  • Planned changes and general language issues

The repository should release a jar artifact which is not coupled with the OpenSearch release cadence and should have its own versioning.

A major advantage for this approach would be to allow different execution engines (drivers) such as OpenSearch, Spark, Prometheus and more to be decoupled from the SQL repository and maintain a independed trail of support for PPL commands and features.


Do you have any additional context?

@YANG-DB YANG-DB added enhancement New feature or request untriaged PPL Piped processing language labels Jul 11, 2024
@dblock dblock removed the untriaged label Jul 29, 2024
@dblock
Copy link
Member

dblock commented Jul 29, 2024

[Catch All Triage - 1, 2, 3, 4]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request PPL Piped processing language
Projects
None yet
Development

No branches or pull requests

2 participants