[Autocomplete] PPL Autocomplete #7810

mengweieric · 2024-08-22T17:50:38Z

Description

PPL autocomplete support stage 1 to go into feature branch
- Antlr parsing and suggesting for PPL query
- Custom suggesting rules
- Utilities for sharing among different languages

Issues Resolved

Screenshot

Testing the changes

Changelog

feat: Add OpenSearch PPL autocomplete to discover 2.0 with query enhancements

Check List

All tests pass
- yarn test:jest
- yarn test:jest_integration
New functionality includes testing.
New functionality has been documented.
Update CHANGELOG.md
Commits are signed per the DCO using --signoff

Signed-off-by: Eric <menwe@amazon.com>

github-actions · 2024-08-22T17:51:11Z

❌ Empty Changelog Section

The Changelog section in your PR description is empty. Please add a valid changelog entry or entries. If you did add a changelog entry, check to make sure that it was not accidentally included inside the comment block in the Changelog section.

Signed-off-by: Eric <menwe@amazon.com>

codecov · 2024-08-22T18:27:42Z

Codecov Report

Attention: Patch coverage is 29.41176% with 72 lines in your changes missing coverage. Please review.

Project coverage is 60.49%. Comparing base (23798a0) to head (61d5a5d).
Report is 1 commits behind head on feature/autocomplete-2.0.

Files	Patch %	Lines
...ata/public/antlr/opensearch_ppl/code_completion.ts	12.00%	22 Missing ⚠️
src/plugins/data/public/antlr/shared/utils.ts	4.54%	21 Missing ⚠️
...ntlr/opensearch_ppl/opensearch_ppl_autocomplete.ts	44.11%	19 Missing ⚠️
...ata/public/antlr/shared/general_error_listerner.ts	0.00%	6 Missing ⚠️
src/plugins/data/public/plugin.ts	0.00%	4 Missing ⚠️

Additional details and impacted files

@@                     Coverage Diff                      @@
##           feature/autocomplete-2.0    #7810      +/-   ##
============================================================
- Coverage                     63.83%   60.49%   -3.35%     
============================================================
  Files                          3661     3657       -4     
  Lines                         81345    86844    +5499     
  Branches                      12978    13324     +346     
============================================================
+ Hits                          51926    52532     +606     
- Misses                        26234    31143    +4909     
+ Partials                       3185     3169      -16

Flag	Coverage Δ
Linux_1	`28.65% <29.41%> (-1.50%)`	⬇️
Linux_2	`55.89% <ø> (+0.02%)`	⬆️
Linux_3	`37.66% <29.41%> (-2.74%)`	⬇️
Linux_4	`29.49% <29.41%> (-1.86%)`	⬇️
Windows_1	`28.66% <29.41%> (-1.50%)`	⬇️
Windows_2	`55.85% <ø> (+0.02%)`	⬆️
Windows_3	`37.66% <29.41%> (-2.74%)`	⬇️
Windows_4	`29.49% <29.41%> (-1.86%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: Eric <menwe@amazon.com>

kavilla · 2024-08-23T17:49:56Z

package.json

@@ -85,7 +85,8 @@
    "cypress:run-with-security": "env TZ=America/Los_Angeles NO_COLOR=1 cypress run --headless --env SECURITY_ENABLED=true,openSearchUrl=https://localhost:9200,WAIT_FOR_LOADER_BUFFER_MS=500",
    "osd:ciGroup10": "echo \"dashboard_sanity_test_spec.js\"",
    "osd:ciGroup11": "echo \"apps/vis_builder/*.js\"",
-    "generate:opensearchsqlantlr": "./node_modules/antlr4ng-cli/index.js -Dlanguage=TypeScript -o ./src/plugins/data/public/antlr/opensearch_sql/.generated -visitor -no-listener -Xexact-output-dir ./src/plugins/data/public/antlr/opensearch_sql/grammar/OpenSearchSQLLexer.g4 ./src/plugins/data/public/antlr/opensearch_sql/grammar/OpenSearchSQLParser.g4"
+    "generate:opensearchsqlantlr": "./node_modules/antlr4ng-cli/index.js -Dlanguage=TypeScript -o ./src/plugins/data/public/antlr/opensearch_sql/.generated -visitor -no-listener -Xexact-output-dir ./src/plugins/data/public/antlr/opensearch_sql/grammar/OpenSearchSQLLexer.g4 ./src/plugins/data/public/antlr/opensearch_sql/grammar/OpenSearchSQLParser.g4",


nice. i see what you did. another backlog item could be moving the scripts to a file to the scripts folder and then call the script like yarn generate:antler --sql

kavilla · 2024-08-23T17:55:43Z

src/plugins/data/public/antlr/opensearch_ppl/code_completion.ts

+import { QuerySuggestion } from '../../autocomplete';
+import { IndexPattern, IndexPatternField } from '../../index_patterns';
+
+const fetchFieldSuggestions = (


i think there can be a space in the language manager we where we register code completion stuff so that these files can be put in the query enhancements plugin.

cc: @ashwin-pc, @abbyhu2000 what do you think?

sejli · 2024-08-26T23:58:55Z

Merging into feature branch.

* add initial ppl autocomplete Signed-off-by: Eric <menwe@amazon.com> * untrack .antlr files Signed-off-by: Eric <menwe@amazon.com> * ignore intermediate grammar auto-gen files Signed-off-by: Eric <menwe@amazon.com> * add rules and related functionalities Signed-off-by: Eric <menwe@amazon.com> * Changeset file for PR opensearch-project#7810 created/updated * Changeset file for PR opensearch-project#7810 created/updated * minor comment cleanning Signed-off-by: Eric <menwe@amazon.com> * add ppl generation command Signed-off-by: Eric <menwe@amazon.com> * add rules Signed-off-by: Eric <menwe@amazon.com> * correct typo Signed-off-by: Eric <menwe@amazon.com> * fix inserting text issue Signed-off-by: Eric <menwe@amazon.com> * remove colon for PPL field Signed-off-by: Eric <menwe@amazon.com> --------- Signed-off-by: Eric <menwe@amazon.com> Co-authored-by: opensearch-changeset-bot[bot] <154024398+opensearch-changeset-bot[bot]@users.noreply.github.com>

* [Autocomplete - SQL] Minor interface change to add suggestion type and move suggestion provider registration location (#7758) * add table/source as prefix to suggested fields Signed-off-by: Eric <menwe@amazon.com> * add type to column Signed-off-by: Eric <menwe@amazon.com> * move registeration to osd/monaco Signed-off-by: Eric <menwe@amazon.com> * add detail Signed-off-by: Eric <menwe@amazon.com> * Changeset file for PR #7758 created/updated --------- Signed-off-by: Eric <menwe@amazon.com> Co-authored-by: opensearch-changeset-bot[bot] <154024398+opensearch-changeset-bot[bot]@users.noreply.github.com> * [Autocomplete] PPL Autocomplete (#7810) * add initial ppl autocomplete Signed-off-by: Eric <menwe@amazon.com> * untrack .antlr files Signed-off-by: Eric <menwe@amazon.com> * ignore intermediate grammar auto-gen files Signed-off-by: Eric <menwe@amazon.com> * add rules and related functionalities Signed-off-by: Eric <menwe@amazon.com> * Changeset file for PR #7810 created/updated * Changeset file for PR #7810 created/updated * minor comment cleanning Signed-off-by: Eric <menwe@amazon.com> * add ppl generation command Signed-off-by: Eric <menwe@amazon.com> * add rules Signed-off-by: Eric <menwe@amazon.com> * correct typo Signed-off-by: Eric <menwe@amazon.com> * fix inserting text issue Signed-off-by: Eric <menwe@amazon.com> * remove colon for PPL field Signed-off-by: Eric <menwe@amazon.com> --------- Signed-off-by: Eric <menwe@amazon.com> Co-authored-by: opensearch-changeset-bot[bot] <154024398+opensearch-changeset-bot[bot]@users.noreply.github.com> * small interface updates Signed-off-by: Paul Sebastian <paulstn@amazon.com> * small type update Signed-off-by: Paul Sebastian <paulstn@amazon.com> * make inclusion for pipe, comma, and equal tokens Signed-off-by: Paul Sebastian <paulstn@amazon.com> * refactor and generalize field fetcher for dql and ppl, including details Signed-off-by: Paul Sebastian <paulstn@amazon.com> * use field fetching util for sql and update sugg detail for dql Signed-off-by: Paul Sebastian <paulstn@amazon.com> * detail for ppl Signed-off-by: Paul Sebastian <paulstn@amazon.com> * create range parameter to help identify suggestions with whitespace, implemented dql value WS suggs Signed-off-by: Paul Sebastian <paulstn@amazon.com> * single line editor overflow initial override for sugg window Signed-off-by: Paul Sebastian <paulstn@amazon.com> * update dql tests to account for details and value ranges Signed-off-by: Paul Sebastian <paulstn@amazon.com> --------- Signed-off-by: Eric <menwe@amazon.com> Signed-off-by: Paul Sebastian <paulstn@amazon.com> Co-authored-by: Eric Wei <menwe@amazon.com> Co-authored-by: opensearch-changeset-bot[bot] <154024398+opensearch-changeset-bot[bot]@users.noreply.github.com>

* [Autocomplete - SQL] Minor interface change to add suggestion type and move suggestion provider registration location (#7758) * add table/source as prefix to suggested fields Signed-off-by: Eric <menwe@amazon.com> * add type to column Signed-off-by: Eric <menwe@amazon.com> * move registeration to osd/monaco Signed-off-by: Eric <menwe@amazon.com> * add detail Signed-off-by: Eric <menwe@amazon.com> * Changeset file for PR #7758 created/updated --------- Signed-off-by: Eric <menwe@amazon.com> Co-authored-by: opensearch-changeset-bot[bot] <154024398+opensearch-changeset-bot[bot]@users.noreply.github.com> * [Autocomplete] PPL Autocomplete (#7810) * add initial ppl autocomplete Signed-off-by: Eric <menwe@amazon.com> * untrack .antlr files Signed-off-by: Eric <menwe@amazon.com> * ignore intermediate grammar auto-gen files Signed-off-by: Eric <menwe@amazon.com> * add rules and related functionalities Signed-off-by: Eric <menwe@amazon.com> * Changeset file for PR #7810 created/updated * Changeset file for PR #7810 created/updated * minor comment cleanning Signed-off-by: Eric <menwe@amazon.com> * add ppl generation command Signed-off-by: Eric <menwe@amazon.com> * add rules Signed-off-by: Eric <menwe@amazon.com> * correct typo Signed-off-by: Eric <menwe@amazon.com> * fix inserting text issue Signed-off-by: Eric <menwe@amazon.com> * remove colon for PPL field Signed-off-by: Eric <menwe@amazon.com> --------- Signed-off-by: Eric <menwe@amazon.com> Co-authored-by: opensearch-changeset-bot[bot] <154024398+opensearch-changeset-bot[bot]@users.noreply.github.com> * small interface updates Signed-off-by: Paul Sebastian <paulstn@amazon.com> * small type update Signed-off-by: Paul Sebastian <paulstn@amazon.com> * make inclusion for pipe, comma, and equal tokens Signed-off-by: Paul Sebastian <paulstn@amazon.com> * refactor and generalize field fetcher for dql and ppl, including details Signed-off-by: Paul Sebastian <paulstn@amazon.com> * use field fetching util for sql and update sugg detail for dql Signed-off-by: Paul Sebastian <paulstn@amazon.com> * detail for ppl Signed-off-by: Paul Sebastian <paulstn@amazon.com> * create range parameter to help identify suggestions with whitespace, implemented dql value WS suggs Signed-off-by: Paul Sebastian <paulstn@amazon.com> * single line editor overflow initial override for sugg window Signed-off-by: Paul Sebastian <paulstn@amazon.com> * update dql tests to account for details and value ranges Signed-off-by: Paul Sebastian <paulstn@amazon.com> --------- Signed-off-by: Eric <menwe@amazon.com> Signed-off-by: Paul Sebastian <paulstn@amazon.com> Co-authored-by: Eric Wei <menwe@amazon.com> Co-authored-by: opensearch-changeset-bot[bot] <154024398+opensearch-changeset-bot[bot]@users.noreply.github.com> (cherry picked from commit 0245540) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* [Autocomplete - SQL] Minor interface change to add suggestion type and move suggestion provider registration location (#7758) * add table/source as prefix to suggested fields * add type to column * move registeration to osd/monaco * add detail * Changeset file for PR #7758 created/updated --------- * [Autocomplete] PPL Autocomplete (#7810) * add initial ppl autocomplete * untrack .antlr files * ignore intermediate grammar auto-gen files * add rules and related functionalities * Changeset file for PR #7810 created/updated * Changeset file for PR #7810 created/updated * minor comment cleanning * add ppl generation command * add rules * correct typo * fix inserting text issue * remove colon for PPL field --------- * small interface updates * small type update * make inclusion for pipe, comma, and equal tokens * refactor and generalize field fetcher for dql and ppl, including details * use field fetching util for sql and update sugg detail for dql * detail for ppl * create range parameter to help identify suggestions with whitespace, implemented dql value WS suggs * single line editor overflow initial override for sugg window * update dql tests to account for details and value ranges --------- (cherry picked from commit 0245540) Signed-off-by: Eric <menwe@amazon.com> Signed-off-by: Paul Sebastian <paulstn@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Eric Wei <menwe@amazon.com> Co-authored-by: opensearch-changeset-bot[bot] <154024398+opensearch-changeset-bot[bot]@users.noreply.github.com>

mengweieric added 2 commits August 21, 2024 20:22

add initial ppl autocomplete

a8d3a31

Signed-off-by: Eric <menwe@amazon.com>

untrack .antlr files

c8579c5

Signed-off-by: Eric <menwe@amazon.com>

github-actions bot added the valued-contributor label Aug 22, 2024

github-actions bot added the failed changeset label Aug 22, 2024

mengweieric and others added 3 commits August 22, 2024 17:54

ignore intermediate grammar auto-gen files

87ad760

Signed-off-by: Eric <menwe@amazon.com>

add rules and related functionalities

d740241

Signed-off-by: Eric <menwe@amazon.com>

Changeset file for PR opensearch-project#7810 created/updated

e5e3e44

github-actions bot removed the failed changeset label Aug 22, 2024

opensearch-changeset-bot bot and others added 3 commits August 22, 2024 18:03

Changeset file for PR opensearch-project#7810 created/updated

3106547

minor comment cleanning

cf0e311

Signed-off-by: Eric <menwe@amazon.com>

add ppl generation command

3ad22a9

Signed-off-by: Eric <menwe@amazon.com>

add rules

504bfe1

Signed-off-by: Eric <menwe@amazon.com>

mengweieric marked this pull request as ready for review August 22, 2024 21:56

mengweieric requested review from ananzh, kavilla, AMoo-Miki, ashwin-pc, joshuarrrr, abbyhu2000, zengyan-amazon, zhongnansu, manasvinibs, ZilongX, Flyingliuhub, curq, bandinib-amzn, SuZhou-Joe and ruanyl as code owners August 22, 2024 21:56

mengweieric requested review from BionIT, xinruiba, zhyuanqi, LDrago27, virajsanghvi, sejli and joshuali925 as code owners August 22, 2024 21:56

correct typo

57ff363

Signed-off-by: Eric <menwe@amazon.com>

mengweieric changed the title ~~[Draft] PPL Autocomplete~~ [Autocomplete] PPL Autocomplete Aug 22, 2024

mengweieric added 2 commits August 22, 2024 22:40

fix inserting text issue

5ea8018

Signed-off-by: Eric <menwe@amazon.com>

remove colon for PPL field

61d5a5d

Signed-off-by: Eric <menwe@amazon.com>

kavilla reviewed Aug 23, 2024

View reviewed changes

sejli approved these changes Aug 26, 2024

View reviewed changes

sejli merged commit 3ae20ef into opensearch-project:feature/autocomplete-2.0 Aug 26, 2024
56 of 66 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Autocomplete] PPL Autocomplete #7810

[Autocomplete] PPL Autocomplete #7810

mengweieric commented Aug 22, 2024 •

edited

Loading

github-actions bot commented Aug 22, 2024

codecov bot commented Aug 22, 2024 •

edited

Loading

kavilla Aug 23, 2024

kavilla Aug 23, 2024

sejli commented Aug 26, 2024

[Autocomplete] PPL Autocomplete #7810

[Autocomplete] PPL Autocomplete #7810

Conversation

mengweieric commented Aug 22, 2024 • edited Loading

Description

Issues Resolved

Screenshot

Testing the changes

Changelog

Check List

github-actions bot commented Aug 22, 2024

❌ Empty Changelog Section

codecov bot commented Aug 22, 2024 • edited Loading

Codecov Report

kavilla Aug 23, 2024

Choose a reason for hiding this comment

kavilla Aug 23, 2024

Choose a reason for hiding this comment

sejli commented Aug 26, 2024

mengweieric commented Aug 22, 2024 •

edited

Loading

codecov bot commented Aug 22, 2024 •

edited

Loading