[ES|QL] Autocomplete suggestions improvements

[stratoula]

Top priorities (in progress, recently done, or up-next)

Get rid of buggy experiences (@drewdaemon )

Preview Give feedback

1. Improve support for Invoke completion trigger kind (shows suggestions again when user resumes typing after the menu has closed) — [ES|QL] improve support for "invoke" autocomplete triggers #188677
2. Disable suggestions when typing values ([ES|QL] remove inaccurate values suggestions #189228) - Today, when typing a field value between double quotes, the suggest list appears with recommendations filtered, by the string, for functions and variables. This is an interim 'fix' as we await the availability of field values.
3. Fix syntax highlighting for several ESQL words — [ES|QL] improve tokenizer and theme #190170
4. ES|QL Autocomplete duplicates string in some cases #187184
   Feature:ES|QL Team:ESQL bug impact:low +4
5. Show command suggestions when blank editor is refocused (relates to [ES|QL] show suggestions on editor refocus #180839) — [ES|QL] Open the suggestion menu when the editor is focused and empty #191142
6. [ES|QL] - only suggest active integration data streams #187247
   Team:ESQL impact:medium +2
   
7. [ES|QL] menu should open again automatically when users accept suggestions #189662
   Feature:ES|QL Team:ESQL enhancement impact:medium +4
8. Respect DROP/STATS when suggesting variables — Drop here suggests the avg(bytes) which is wrong as with the previous keep I am only keeping one column
9. [ES|QL] revisit field and variable representations #191111
   Feature:ES|QL Team:ESQL enhancement impact:medium +4
   
10. [ES|QL] suggestions are shown in comments #191866
    Team:ESQL bug impact:low +3
11. A second comma loosens suggestion constraints - https://github.com/elastic/kibana/assets/315764/0e27577e-ea73-4a41-9b12-fe043fb4fd59
12. [ES|QL] incorrect suggestions in between function arguments #191876
    Feature:ES|QL Team:ESQL bug impact:low +4

Improve key functions (@qn895 )

Preview Give feedback

1. Build argument suggestions based on all available call signatures (ref) — [ES|QL] Improve suggestions based on previous function arguments and date suggestions for bucket #190828
2. BUCKET — In general, we want to guide users toward building date histograms attached to the time picker. [ES|QL] Improve suggestions based on previous function arguments and date suggestions for bucket #190828
3. CASE — Decide on an algorithm for suggesting fields and values. Improve suggestion for the first parameter (see [ES|QL] Autocomplete should help build expressions as first arg in case rather than simple boolean #177258). Technical note: possibly depart from the current function definition structure for this one. It is a really strange function and we don’t really have a way to describe it to the validator and the suggestion engine. - [ES|QL] Improve ES|QL autocomplete suggestions for case() expressions #192135
4. COUNT — suggest * - [ES|QL] Add all (*) item in list of suggestions for COUNT #192205
5. [ES|QL] Add hints upon hover for function argument types and time system types #191881
   Feature:ES|QL Team:ESQL backport:skip ci:cloud-deploy release_note:enhancement v8.16.0 +6
   

Improve field metadata display (@qn895 )

Preview Give feedback

1. Indicate field type - When writing a query and doing calculations or bucketing with STATS and BUCKET, it would be helpful to know the field type (date, number, boolean, etc.) — [ES|QL] Add ECS information to the editor hint & prioritize fields based on ECS information on the editor #187922
2. Add ECS field descriptions — [ES|QL] Add ECS information to the editor hint & prioritize fields based on ECS information on the editor #187922
3. [ES|QL] Prioritize fields based on ecs information on the editor #181036
   Feature:ES|QL Team:ESQL enhancement impact:high +4
   

Improve core metadata (@qn895 )

Preview Give feedback

1. Use Elasticsearch data types for suggestions and validation — For historical reasons we were using simplified types (e.g. number instead of double, long, int, etc). This led to occasionally-incorrect suggestions and a lack of precision in our client-side validation. [ES|QL] Leverage sub-types in AST for both validation and autocomplete #174710

Improve command suggestions

Preview Give feedback

1. Improve BY suggestion to include pipe - If you add a BY then the +,- is suggested. We should at least suggest the pipe here — [ES|QL] Improve stats BY suggestion to include pipe and comma operators #189458
2. Improve Metadata suggestions with comma - Metadata autocomplete should use comma when suggesting the rest of the available meta fields, also should suggest the pipe ([ES|QL] Improve metadata autocomplete suggestions with comma and pipe #188338)
3. SORT command should suggest uppercase options and apply highlighting — [ES|QL] improve SORT command suggestions #188579
4. SORT command should suggest ASC/DESC after subsequent field — [ES|QL] Implement OrderExpression for SORT command arguments #189959 and [ES|QL] new pattern for SORT autocomplete #193595
5. INLINESTATS support — [ES|QL] Support inline stats in autocomplete and client side validation #189356 (@qn895 )
6. RENAME: suggest comma as well as pipe — after RENAME message AS var0... or maybe don’t suggest var0 after AS? (@qn895)

Other ideas (blocked, not currently planned, or just ideas for the future)

Function helpers

Preview Give feedback

1. [ES|QL] respect existing parenthesis when renaming a function via autocomplete #180953
   Team:ESQL enhancement impact:medium usability +4
2. [ES|QL] suggest a missing command based on a function #180737
   Team:ESQL enhancement impact:low +3
3. Constant-only parameter constraints should be inherited by nested functions - For example, if func1's first parameter is constant-only, any nested functions should inherit that constraint: func1(func2(shouldBeConstantOnly))
4. Improve 'Read more' for functions - This is contextually relevant and contains useful documentation; however, the order and readability of content could be improved At left: ES|QL ... right: analog example)
5. Excel-like experience for functions (see [ES|QL] show function signature while typing #180528) - Leverage the Parameter Info capabilities provided by Intellisense for a more Excel-like experience when adding functions to a query

Smarter suggest

Preview Give feedback

1. [ES|QL] Suggest field values #184679 - This is a known ES limitation at the moment (need issue link if one exists). It's worth capturing as it is a key, missing capability of the current editing experience. As seen below, the suggest lists simply contains a redundant field list.
2. [ES|QL] show suggestions on editor refocus #180839
   Team:ESQL enhancement impact:low +3
3. Redisplay autocomplete when editing/typing a data source - After adding a data source, backspace/delete some text, start typing again, autocomplete should redisplay (filtered). In the gif below, when re-typing auditbeat, I expect the suggestion menu to redisplay, filtered to what I have typed thus far. (done in [ES|QL] improve support for "invoke" autocomplete triggers #188677)
4. Indicate when a value needs to be entered - In addition to suggesting fields, commands, and functions, it can be equally logical to enter a value such as a number. In the example below, I am desiring to enter a numerical value (bonus: a hint on the range or sampling of values would be extra delightful)
5. Suggest constant values for functions that accept constants as arguments
6. Ignore casing when sorting (?) - When one data source uses camel case, those fields appear first. In the situation where sources use the same casing, the list appears to be combined and sorted as a single list. I sense lowercase is the best practice, but other casing does not seem to be prohibited. Upon further investigation, I don't see indices in obs or esql demo environments with camel case field names.
7. Revisit how we handle remote indices - Support for remote indices was recently added. Now that it is in place, we may want to consider some enhancements with regards to scalability and organizational hierarchy. For example, clusters appear alphabetically with local indices.

Environmental / Authoring helpers

Preview Give feedback

1. Indicate data source for a field - Most relevant upon adding more than one data source, the suggest list combines/mixes fields making it difficult to know which source(s) the field is coming from. Fields may exist in both - appear as one entry - or be similar in name on more than one data source. Note that the 'Ability to view field names' recommendation would help here, too.
2. Remove not-very-useful 'Read more' on fields/variables - Currently, the same 'Read more' text is shown on every field/variable. It is both redundant and lacking useful information; therefore, we should consider disabling it.

Syntactical papercuts

Preview Give feedback

1. [ES|QL] make autocomplete respect existing closing parenthesis #180975
   Team:ESQL enhancement impact:low +3
2. [ES|QL] correct double-commas when a suggestion is accepted #180840
   Team:ESQL enhancement impact:low +3
3. Fix suggestions after + - In the above, after clicking the + the aggregation is suggested which is wrong. If the user runs it, ES will error out
4. Autocomplete shouldn't suggest on = when it is part of <= or >= - https://github.com/elastic/kibana/assets/315764/4635d08f-5618-4091-b8fd-925baff95b77

Related, previously issues

• Support data_part as proposed in docs
  These date functions can be very tricky so we would like to help our users a bit with autocomplete. It would be great if we could suggest the date_part as proposed on our official documentation

[elasticmachine]

Pinging @elastic/kibana-visualizations (Team:Visualizations)

[dej611]

These date functions can be very tricky so we would like to help our users a bit with autocomplete. It would be great if we could suggest the date_part as proposed on our official documentation

This is done already in 8.13

[elasticmachine]

Pinging @elastic/kibana-esql (Team:ESQL)

[dej611]

Drop here suggests the avg(bytes) which is wrong as with the previous keep I am only keeping one column

this is by design right now.
All location information about variables are stored but the engine is not pruning variables after stats, keep, rename or drop. All column utility functions should take those into account (same apply for validation feature as well).

[dej611]

Meta command is being suggested after the source command 🐛

[stratoula]

@drewdaemon when we start working on it can we prioritze the bucket autocomplete? I expect this to be a very popular function (date histogram) and the syntax is quite complicated so I would love to offer a better experience to our users

[dej611]

Remove not-very-useful 'Read more' on fields/variables - Currently, the same 'Read more' text is shown on every field/variable. It is both redundant and lacking useful information; therefore, we should consider disabling it.

Why not showing the type of the field? Unless the user is super familiar with the index it is unlickely to remember all field types.

[stratoula]

Why not showing the type of the field? Unless the user is super familiar with the index it is unlickely to remember all field types.

Let's sync first @drewdaemon before we do this, I have another idea for this. (ECS schema information).

[dej611]

It's a markdown string, I do not see how the two can conflicts.
Something like:

Type: ...
[ECS: ...] // if available

Would apply to all scenarios, also non ECS ones.