UX proposal for an improved data source picker (v1)

[shanilpa]

Background and context

This issue tackles three primary challenges we currently face in OpenSearch Dashboards and will continue to develop as we integrate Multiple Data Source connections to a single instance of Dashboards. The goal of this issue is to iterate on the current data source selection experience (ungrouped list) that sets us up for a much better data selection experience in the fullness of time. This initial UX only proposal is intended for use in Data Explorer and Discover to support Multi Data Source and Spark selection use cases within these experiences. Other experiences are welcome to use this experience once implemented but not required until a formal campaign is rolled out by the OUI team.

A related engineering proposal will follow for more context on the technical approach to this UX proposal and will be linked to this issue.

Current state challenges

The three broad challenges we face today:

1. Users have no visibility into where the data is coming from when selecting a data source.
   1. With the support for Multiple Data Sources in 2.7 a single instance of OpenSearch Dashboards is able to connect to multiple OpenSearch Clusters. With new capabilities like connecting to external data sources (Apache Spark) in the works users will be able to select data from multiple locations to explore in Dashboards. As a result users will need a way to select from multiple sources and be able to trace back to where their data is coming from. Currently, we show all data in a flat list behind an index pattern abstraction. This posts several issues discussed in the second issue but also lacks scalability and ease of use for users.
   2. The data source is also a driving factor for other experiences and sets the stage for exploration. For example, users selecting an OpenSearch data source can query using the DQL and PPL query languages, however, users exploring a Spark data source will likely use SQL. Currently, we do not support these selections and setting of context but will need to so users can truly leverage Dashboards for their data exploration use cases. See Data Explorer for more information.
2. Users are required to create index patterns to explore data in core Dashboards, this does not scale mental model wise and also lacks support for some plugin use cases.
   1. Our current data source selection experience forces users to select an Index pattern. Index patterns are a Dashboards construct that allows users to use a wildecard (*) to group similar OpenSearch indices into a single data source for querying and use in Dashboards. Though useful this abstraction does not always align to users needs or mental models. For example, Spark users mental models revolve around tables not indices and plugin experiences like alerting and Observability require users to select the data source directly (index) vs the index pattern.
3. Lack of a cohesive data selection experience.
   1. Data selection is crucial in almost every aspect of OpenSearch Dashboards and we have several experiences that follow different patterns making it difficult for users to understand how to perform actions or what to expect (screenshots in the additional context section). By providing a generalized component and supporting interaction patterns we can allow users to select data intuitively and through a familiar experience across Dashboards. Additionally, this will help builders implement data selection in a more efficient and streamlined fashion within their experiences while leveraging the built in benefits of OUI.

Proposed UX Solution

The following UX proposal is a first pass in addressing the above issues and seeks to leverage stock OUI components to provide an improved data selection experience in Data Explorer specifically, however the UX team is looking at standardizing across other data selection experiences in Dashboards so we can drive towards cohesion and providing a familiar and intuitive user experience for our users.

OUI component usage:
OuiComboBox - Single Select as the primary component. For the list view we can POC whether OuiComboBox-Groups or OuiComboBox-Virtualized makes the most sense from a UX and performance standpoint.
Additionally if we can POC adding OuiBadge that reflects the query language options per data source without breaking the ComboBox component that would be awesome.

Alternate explorations

Early exploration and POC’s into alternate variations can be found here: opensearch-project/dashboards-observability#545

Additional context

Current data selection in Discover:

Current data selection in Observability Logs:

Current data selection in Alerting monitor create flow:

[joshuarrrr]

and plugin experiences like alerting and Observability require users to select the data source directly (index) vs the index pattern.

@shanilpa As part of this proposal, have we reconsidered this choice? I'm in agreement with almost every other points listed, but I think the direct index selection as it exists today in these plugins is already a bit of an antipattern.

[shanilpa]

@joshuarrrr not sure I follow. Could you elaborate a little more on what the anti-pattern might be?

Users and plugins should have the option to use either an index pattern, an index, or both depending on their use cases. This gives both the flexibility to use data the way they need for their unique use cases without having to conform to a construct if they don't want too. Definitely open to discussing more and gathering different opinions.

[joshuarrrr]

Sure - it's very common for relevant docs to span multiple indices, or to have date-based indices that all have similar documents in them, and that's exactly why index patterns exist, to be able to treat a collection of indices as a singular data source. So it's not clear why an alerting or observability user wouldn't want or need that similar ability.

Additionally, there are other capabilities (such as display formatting) that plugins would need to re-implement if they don't use index patterns.