[Auto Suggest] PPL autocomplete and interface changes #7932

paulstn · 2024-08-30T02:51:22Z

Description

Adds PPL Autocomplete using ANTLR and C3 Code Completion.
Also makes interface changes to include a 'details' aspect to the suggestions which will include a relevant label for each one. Another interface change allows a suggestion to specify the range of characters that it should replace.

Note: Every file worth reviewing is under src/plugins/data/public

Issues Resolved

Screenshot

Testing the changes

Changelog

feat: PPL Autocomplete
fix: Auto Suggest interface updates
fix: Single Line Editor Suggestion Window covered

Check List

All tests pass
- yarn test:jest
- yarn test:jest_integration
New functionality includes testing.
New functionality has been documented.
Update CHANGELOG.md
Commits are signed per the DCO using --signoff

github-actions · 2024-08-30T02:51:53Z

❌ Empty Changelog Section

The Changelog section in your PR description is empty. Please add a valid changelog entry or entries. If you did add a changelog entry, check to make sure that it was not accidentally included inside the comment block in the Changelog section.

github-actions · 2024-08-30T02:56:21Z

❌ Empty Changelog Section

The Changelog section in your PR description is empty. Please add a valid changelog entry or entries. If you did add a changelog entry, check to make sure that it was not accidentally included inside the comment block in the Changelog section.

codecov · 2024-08-30T03:06:02Z

Codecov Report

Attention: Patch coverage is 35.65217% with 74 lines in your changes missing coverage. Please review.

Project coverage is 60.97%. Comparing base (e5c125d) to head (087a7f0).
Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
src/plugins/data/public/antlr/shared/utils.ts	30.00%	21 Missing ⚠️
...ntlr/opensearch_ppl/opensearch_ppl_autocomplete.ts	45.71%	19 Missing ⚠️
...ata/public/antlr/opensearch_ppl/code_completion.ts	11.76%	15 Missing ⚠️
...ata/public/antlr/opensearch_sql/code_completion.ts	0.00%	9 Missing ⚠️
...ata/public/antlr/shared/general_error_listerner.ts	0.00%	6 Missing ⚠️
src/plugins/data/public/plugin.ts	0.00%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #7932      +/-   ##
==========================================
- Coverage   64.33%   60.97%   -3.36%     
==========================================
  Files        3677     3684       +7     
  Lines       81200    87038    +5838     
  Branches    12944    13376     +432     
==========================================
+ Hits        52240    53074     +834     
- Misses      25749    30750    +5001     
- Partials     3211     3214       +3

Flag	Coverage Δ
Linux_1	`29.26% <28.69%> (-1.48%)`	⬇️
Linux_2	`56.19% <ø> (ø)`
Linux_3	`37.82% <35.65%> (-2.80%)`	⬇️
Linux_4	`29.57% <28.69%> (-1.84%)`	⬇️
Windows_1	`29.28% <28.69%> (-1.48%)`	⬇️
Windows_2	`56.14% <ø> (ø)`
Windows_3	`37.82% <35.65%> (-2.79%)`	⬇️
Windows_4	`29.57% <28.69%> (-1.84%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

docs/_sidebar.md

ashwin-pc · 2024-08-30T04:11:01Z

package.json

+    "generate:opensearchsqlantlr": "./node_modules/antlr4ng-cli/index.js -Dlanguage=TypeScript -o ./src/plugins/data/public/antlr/opensearch_sql/.generated -visitor -no-listener -Xexact-output-dir ./src/plugins/data/public/antlr/opensearch_sql/grammar/OpenSearchSQLLexer.g4 ./src/plugins/data/public/antlr/opensearch_sql/grammar/OpenSearchSQLParser.g4",
+    "generate:opensearchpplantlr": "./node_modules/antlr4ng-cli/index.js -Dlanguage=TypeScript -o ./src/plugins/data/public/antlr/opensearch_ppl/.generated -visitor -no-listener -Xexact-output-dir ./src/plugins/data/public/antlr/opensearch_ppl/grammar/OpenSearchPPLLexer.g4 ./src/plugins/data/public/antlr/opensearch_ppl/grammar/OpenSearchPPLParser.g4"


Cant we have a single command to generate all the antlr grammer? Alternatively, not for this PR by we can also use the osd script to add thsi as a command there to not bloat up the package.json commands. e.g. yarn osd antlr-grammar

We could, but I'd assume more granularity would be better when needing to regenerate the antlr files. Do you think we should make this change now or handle it when moving it to the osd script?

+1 i think in @mengweieric PR made a similiar comment. Like post 2.17, we should have something in the scripts folder that lets us run yarn antlr:generateGrammar --sql or something

ashwin-pc · 2024-08-30T04:11:59Z

src/plugins/data/public/ui/query_editor/query_editor.tsx

ashwin-pc · 2024-08-30T04:14:44Z

src/plugins/data/public/plugin.ts

+    autoComplete.addQuerySuggestionProvider('SQL', getSQLSuggestions);
+    autoComplete.addQuerySuggestionProvider('kuery', getDQLSuggestions);
+    autoComplete.addQuerySuggestionProvider('PPL', getPPLSuggestions);


nit: we should move adding the suggestion provider for PPL and SQL to the query enhancements plugin since registering the language happens there as well. Making the suggestion provider also be a part of the language service's register function would bring all these language related features into one place too.

ashwin-pc · 2024-08-30T04:15:16Z

src/plugins/data/public/antlr/shared/utils.ts

Can we add some tests for this?

…d move suggestion provider registration location (opensearch-project#7758) * add table/source as prefix to suggested fields Signed-off-by: Eric <menwe@amazon.com> * add type to column Signed-off-by: Eric <menwe@amazon.com> * move registeration to osd/monaco Signed-off-by: Eric <menwe@amazon.com> * add detail Signed-off-by: Eric <menwe@amazon.com> * Changeset file for PR opensearch-project#7758 created/updated --------- Signed-off-by: Eric <menwe@amazon.com> Co-authored-by: opensearch-changeset-bot[bot] <154024398+opensearch-changeset-bot[bot]@users.noreply.github.com>

* add initial ppl autocomplete Signed-off-by: Eric <menwe@amazon.com> * untrack .antlr files Signed-off-by: Eric <menwe@amazon.com> * ignore intermediate grammar auto-gen files Signed-off-by: Eric <menwe@amazon.com> * add rules and related functionalities Signed-off-by: Eric <menwe@amazon.com> * Changeset file for PR opensearch-project#7810 created/updated * Changeset file for PR opensearch-project#7810 created/updated * minor comment cleanning Signed-off-by: Eric <menwe@amazon.com> * add ppl generation command Signed-off-by: Eric <menwe@amazon.com> * add rules Signed-off-by: Eric <menwe@amazon.com> * correct typo Signed-off-by: Eric <menwe@amazon.com> * fix inserting text issue Signed-off-by: Eric <menwe@amazon.com> * remove colon for PPL field Signed-off-by: Eric <menwe@amazon.com> --------- Signed-off-by: Eric <menwe@amazon.com> Co-authored-by: opensearch-changeset-bot[bot] <154024398+opensearch-changeset-bot[bot]@users.noreply.github.com>

Signed-off-by: Paul Sebastian <paulstn@amazon.com>

…implemented dql value WS suggs Signed-off-by: Paul Sebastian <paulstn@amazon.com>

Signed-off-by: Paul Sebastian <paulstn@amazon.com>

joshuali925

lgtm, just questions

joshuali925 · 2024-08-30T20:22:04Z

changelogs/fragments/7810.yml

@@ -0,0 +1,2 @@
+feat:
+- Add OpenSearch PPL autocomplete to discover 2.0 with query enhancements ([#7810](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/7810))


np. there are two changelogs but non of them is for #7932?

Just updated it, added more details :)

joshuali925 · 2024-08-30T20:25:11Z

packages/osd-monaco/src/xjson/lexer_rules/opensearchppl.ts

+      [/\/.*$/, 'comment'],
+
+      [/".*?"/, 'string'],
+      [/'.*?'/, 'constant'],


curious how are these defined? i don't think PPL supports comments, and the string definition is a bit different from sql

good catch, i'm not sure why that's in there. best guess is it was part of the boilerplate. i'll look it over and make the needed changes

is the boilerplate generated by something or copied from some pre-existing code? or is this written from scratch by this PR? asking because looks like the rules are from some standard but might not completely apply to PPL syntax. probably won't break but just curious

i just mean copied from the other monarch lexer rule files, i agree on it not completely applying to ppl

joshuali925 · 2024-08-30T20:27:17Z

src/plugins/data/public/ui/query_editor/query_editor.tsx

-              range,
-            }))
+          ? suggestions
+              .filter((s) => 'detail' in s) // remove suggestion not of type MonacoCompatible


what is detail?

detail is a field in a possible suggestion that is a label on what that suggestion is. for example, if we're suggesting a field of type geo_point, it will read Field: geo_point next to the suggestion. the reason it's being used here is as a type guard since detail is a monaco specific suggestion field

Signed-off-by: Paul Sebastian <paulstn@amazon.com>

* [Autocomplete - SQL] Minor interface change to add suggestion type and move suggestion provider registration location (#7758) * add table/source as prefix to suggested fields Signed-off-by: Eric <menwe@amazon.com> * add type to column Signed-off-by: Eric <menwe@amazon.com> * move registeration to osd/monaco Signed-off-by: Eric <menwe@amazon.com> * add detail Signed-off-by: Eric <menwe@amazon.com> * Changeset file for PR #7758 created/updated --------- Signed-off-by: Eric <menwe@amazon.com> Co-authored-by: opensearch-changeset-bot[bot] <154024398+opensearch-changeset-bot[bot]@users.noreply.github.com> * [Autocomplete] PPL Autocomplete (#7810) * add initial ppl autocomplete Signed-off-by: Eric <menwe@amazon.com> * untrack .antlr files Signed-off-by: Eric <menwe@amazon.com> * ignore intermediate grammar auto-gen files Signed-off-by: Eric <menwe@amazon.com> * add rules and related functionalities Signed-off-by: Eric <menwe@amazon.com> * Changeset file for PR #7810 created/updated * Changeset file for PR #7810 created/updated * minor comment cleanning Signed-off-by: Eric <menwe@amazon.com> * add ppl generation command Signed-off-by: Eric <menwe@amazon.com> * add rules Signed-off-by: Eric <menwe@amazon.com> * correct typo Signed-off-by: Eric <menwe@amazon.com> * fix inserting text issue Signed-off-by: Eric <menwe@amazon.com> * remove colon for PPL field Signed-off-by: Eric <menwe@amazon.com> --------- Signed-off-by: Eric <menwe@amazon.com> Co-authored-by: opensearch-changeset-bot[bot] <154024398+opensearch-changeset-bot[bot]@users.noreply.github.com> * small interface updates Signed-off-by: Paul Sebastian <paulstn@amazon.com> * small type update Signed-off-by: Paul Sebastian <paulstn@amazon.com> * make inclusion for pipe, comma, and equal tokens Signed-off-by: Paul Sebastian <paulstn@amazon.com> * refactor and generalize field fetcher for dql and ppl, including details Signed-off-by: Paul Sebastian <paulstn@amazon.com> * use field fetching util for sql and update sugg detail for dql Signed-off-by: Paul Sebastian <paulstn@amazon.com> * detail for ppl Signed-off-by: Paul Sebastian <paulstn@amazon.com> * create range parameter to help identify suggestions with whitespace, implemented dql value WS suggs Signed-off-by: Paul Sebastian <paulstn@amazon.com> * single line editor overflow initial override for sugg window Signed-off-by: Paul Sebastian <paulstn@amazon.com> * update dql tests to account for details and value ranges Signed-off-by: Paul Sebastian <paulstn@amazon.com> --------- Signed-off-by: Eric <menwe@amazon.com> Signed-off-by: Paul Sebastian <paulstn@amazon.com> Co-authored-by: Eric Wei <menwe@amazon.com> Co-authored-by: opensearch-changeset-bot[bot] <154024398+opensearch-changeset-bot[bot]@users.noreply.github.com> (cherry picked from commit 0245540) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* [Autocomplete - SQL] Minor interface change to add suggestion type and move suggestion provider registration location (#7758) * add table/source as prefix to suggested fields * add type to column * move registeration to osd/monaco * add detail * Changeset file for PR #7758 created/updated --------- * [Autocomplete] PPL Autocomplete (#7810) * add initial ppl autocomplete * untrack .antlr files * ignore intermediate grammar auto-gen files * add rules and related functionalities * Changeset file for PR #7810 created/updated * Changeset file for PR #7810 created/updated * minor comment cleanning * add ppl generation command * add rules * correct typo * fix inserting text issue * remove colon for PPL field --------- * small interface updates * small type update * make inclusion for pipe, comma, and equal tokens * refactor and generalize field fetcher for dql and ppl, including details * use field fetching util for sql and update sugg detail for dql * detail for ppl * create range parameter to help identify suggestions with whitespace, implemented dql value WS suggs * single line editor overflow initial override for sugg window * update dql tests to account for details and value ranges --------- (cherry picked from commit 0245540) Signed-off-by: Eric <menwe@amazon.com> Signed-off-by: Paul Sebastian <paulstn@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Eric Wei <menwe@amazon.com> Co-authored-by: opensearch-changeset-bot[bot] <154024398+opensearch-changeset-bot[bot]@users.noreply.github.com>

github-actions bot added the valued-contributor label Aug 30, 2024

github-actions bot added the failed changeset label Aug 30, 2024

ashwin-pc reviewed Aug 30, 2024

View reviewed changes

mengweieric and others added 9 commits August 30, 2024 11:00

small interface updates

a09b5f9

Signed-off-by: Paul Sebastian <paulstn@amazon.com>

small type update

6117d5b

Signed-off-by: Paul Sebastian <paulstn@amazon.com>

make inclusion for pipe, comma, and equal tokens

bdb33e7

Signed-off-by: Paul Sebastian <paulstn@amazon.com>

refactor and generalize field fetcher for dql and ppl, including details

7aaf02c

Signed-off-by: Paul Sebastian <paulstn@amazon.com>

use field fetching util for sql and update sugg detail for dql

127fb76

Signed-off-by: Paul Sebastian <paulstn@amazon.com>

detail for ppl

516212f

Signed-off-by: Paul Sebastian <paulstn@amazon.com>

create range parameter to help identify suggestions with whitespace, …

949e046

…implemented dql value WS suggs Signed-off-by: Paul Sebastian <paulstn@amazon.com>

paulstn force-pushed the ppl-autocomplete branch from 72a0e63 to 949e046 Compare August 30, 2024 18:01

single line editor overflow initial override for sugg window

18427b6

Signed-off-by: Paul Sebastian <paulstn@amazon.com>

paulstn marked this pull request as ready for review August 30, 2024 20:06

paulstn requested review from ananzh, kavilla, AMoo-Miki, joshuarrrr, abbyhu2000, zengyan-amazon, zhongnansu, manasvinibs, ZilongX, Flyingliuhub, curq and bandinib-amzn as code owners August 30, 2024 20:06

paulstn requested review from SuZhou-Joe, ruanyl, BionIT, xinruiba, zhyuanqi, mengweieric, LDrago27, virajsanghvi, sejli and joshuali925 as code owners August 30, 2024 20:06

paulstn force-pushed the ppl-autocomplete branch from 18427b6 to 98058b3 Compare August 30, 2024 20:06

kavilla previously approved these changes Aug 30, 2024

View reviewed changes

opensearch-changeset-bot bot added a commit to paulstn/OpenSearch-Dashboards that referenced this pull request Aug 30, 2024

Changeset file for PR opensearch-project#7932 created/updated

0381469

github-actions bot removed the failed changeset label Aug 30, 2024

opensearch-changeset-bot bot dismissed kavilla’s stale review via 0381469 August 30, 2024 20:23

opensearch-changeset-bot bot added a commit to paulstn/OpenSearch-Dashboards that referenced this pull request Aug 30, 2024

Changeset file for PR opensearch-project#7932 created/updated

2de8689

joshuali925 previously approved these changes Aug 30, 2024

View reviewed changes

update dql tests to account for details and value ranges

6754d6c

Signed-off-by: Paul Sebastian <paulstn@amazon.com>

paulstn dismissed joshuali925’s stale review via 6754d6c August 30, 2024 22:01

paulstn force-pushed the ppl-autocomplete branch from 2de8689 to 6754d6c Compare August 30, 2024 22:01

Merge branch 'main' into ppl-autocomplete

087a7f0

joshuali925 approved these changes Aug 30, 2024

View reviewed changes

ashwin-pc added backport 2.x discover for discover reinvent v2.17.0 labels Aug 30, 2024

ashwin-pc approved these changes Aug 30, 2024

View reviewed changes

ashwin-pc merged commit 0245540 into opensearch-project:main Aug 30, 2024
70 of 73 checks passed

opensearch-trigger-bot bot mentioned this pull request Aug 30, 2024

[Backport 2.x] [Auto Suggest] PPL autocomplete and interface changes #7950

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Auto Suggest] PPL autocomplete and interface changes #7932

[Auto Suggest] PPL autocomplete and interface changes #7932

paulstn commented Aug 30, 2024 •

edited

Loading

github-actions bot commented Aug 30, 2024

github-actions bot commented Aug 30, 2024

codecov bot commented Aug 30, 2024 •

edited

Loading

ashwin-pc Aug 30, 2024

paulstn Aug 30, 2024

kavilla Aug 30, 2024

ashwin-pc Aug 30, 2024

ashwin-pc Aug 30, 2024

ashwin-pc Aug 30, 2024

joshuali925 left a comment

joshuali925 Aug 30, 2024

paulstn Aug 30, 2024

joshuali925 Aug 30, 2024

paulstn Aug 30, 2024

joshuali925 Aug 30, 2024

paulstn Aug 30, 2024

joshuali925 Aug 30, 2024

paulstn Aug 30, 2024

		"generate:opensearchsqlantlr": "./node_modules/antlr4ng-cli/index.js -Dlanguage=TypeScript -o ./src/plugins/data/public/antlr/opensearch_sql/.generated -visitor -no-listener -Xexact-output-dir ./src/plugins/data/public/antlr/opensearch_sql/grammar/OpenSearchSQLLexer.g4 ./src/plugins/data/public/antlr/opensearch_sql/grammar/OpenSearchSQLParser.g4",
		"generate:opensearchpplantlr": "./node_modules/antlr4ng-cli/index.js -Dlanguage=TypeScript -o ./src/plugins/data/public/antlr/opensearch_ppl/.generated -visitor -no-listener -Xexact-output-dir ./src/plugins/data/public/antlr/opensearch_ppl/grammar/OpenSearchPPLLexer.g4 ./src/plugins/data/public/antlr/opensearch_ppl/grammar/OpenSearchPPLParser.g4"

		@@ -0,0 +1,2 @@
		feat:
		- Add OpenSearch PPL autocomplete to discover 2.0 with query enhancements ([#7810](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/7810))

[Auto Suggest] PPL autocomplete and interface changes #7932

[Auto Suggest] PPL autocomplete and interface changes #7932

Conversation

paulstn commented Aug 30, 2024 • edited Loading

Description

Issues Resolved

Screenshot

Testing the changes

Changelog

Check List

github-actions bot commented Aug 30, 2024

❌ Empty Changelog Section

github-actions bot commented Aug 30, 2024

❌ Empty Changelog Section

codecov bot commented Aug 30, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joshuali925 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

paulstn commented Aug 30, 2024 •

edited

Loading

codecov bot commented Aug 30, 2024 •

edited

Loading