[Canvas] [Meta] UX improvements for changing your data source

[ryankeairns]

Problem statement

Canvas looks fun and people are excited by the prospect of creating a highly customizable presentation of their data! So they dive into a new workpad, add an element, and then…

What’s all this data source stuff?

• This doesn’t look familiar.
• I’ve heard of Timelion, but it’s going away, right?
• I haven’t SQL’d in a long time.
• I know some SQL, but this is just a text box.

The moment can become lost.

Live data is a key differentiator between Canvas and other presentation software, which is troublesome if it also happens to be an early hurdle for our users.

What if users could more easily see their data in real elements?
Suppose you open add your first element and within seconds you can view your own data?

It’s a much more impactful experience that leaves the user feeling as though everything is possible, they need only add imagination. There will still be challenges, sure, but knowing that you have a palette of elements and your data readily available then places the onus on the user to continue building.

If you can’t get your data into onto the workpad page, then you’ll presume the tool to is too difficult to use or, worse yet, flawed. At which point, you quickly bail.

Findings

While this initially surfaced as anecdotal feedback, it became quickly apparent while conducting user testing sessions that something is amiss. The initial pattern that a new user follows is fairly predictable:

1. Create a new workpad (or start from a sample data workpad)
2. Add a new element (or select an existing sample element)
3. Change its appearance (e.g. color, size, etc.)
4. 🙅 Change the data source

Users move quickly through steps 1-3 only to have their momentum slow or stop altogether at step 4. One frustrated (technical) user even suggested they’d simply give up at this point.

Possible solutions [ease][impact]

Initial side conversations have surfaced a number of potential improvements.

SMALL

Quick wins

• [1][2] Provide contextual help (link to ES SQL data) - [Canvas] Add link to ES SQL docs #35435
• [1][2] SQL datasource defaults to a valid index/query - [Canvas] SQL datasource should default to a valid index pattern #35967
• [?][3] Provide SQL validation - Add form validation to ESSQL #23153
• [1][2] Preserve form state in data source panel (Data tab) - Preserve state in datasource editor #23105
• [2][2] Show visual cues (validation?) steering users back to the Display tab - [Canvas][Data source] Guide users to select values after changing data source #40904
• [1][2] Use a code editor instead of a plain textarea - [Canvas][Data source] Use a text editor input for the ES SQL data source #40907
• [1][1] Re-organize the data source UI - [Canvas][Data source] Re-organize the data source UI #40910
  • Change the order of data sources (more emphasis on ES SQL and ES raw documents)
  • Play down Timelion
  • Re-style demo data (once you switch away, you likely don’t switch back)
• [2][2] Create a ‘Getting started’ workpad that walks a user through Canvas features, including the data piece - [Canvas] Create a 'Getting started' workpad that demonstrates Canvas features #40911
• [2][2] Use a guided tour that prompts a user to add an element and change a data source (select index and field). Might also prompt to open the Expression editor (and make a change?) Controlling dataset when adding Elements #23048
• [2][2] Add intermediate step after choosing an element to select a datasource? (I think Visualize does this)

MEDIUM

More effort; more potential impact

• [1][2] Improve and/or make the data preview feature better
• [2][3] Exposing/listing available indices in the UI
• [2][3] Exposing/listing available fields in the UI
• [3][3] New data source: Saved search [Canvas] Use saved searches (Discover) as an element data source #56539
• [3][3] New data source: Saved queries (Saved queries are a Saved Object) Implement saved queries and filters #39140
• [3][3] New data source: Data Frames
• [3][3] New data source: SIEM?
• [?][?] New data source: Other saved objects?
• [3][3] Do we keep using demo data? Can we make it better? (it has benefits)
• [3][3] Data wizard that connects a workpad to a datasource, and helps users get started with elements

LARGE

Most effort; most potential impact

• [3+][3+] New UI layout that improves the data source UX - [Canvas] New layout for working with data sources #44144
• [3+][3+] Autocomplete for SQL?
• [3+][3+] Visual query builder solution (inspired by Maps)
• [3+][3+] Data snippets (global Kibana solution; Heap example)

[elasticmachine]

Pinging @elastic/kibana-canvas

[ryankeairns]

I've been working on a couple of prototypes that address several of the above items in addition to some improvements to the expression editor (resizing, etc.). Working with data and features is critical to improving the Canvas experience for both new and experienced users. The prototypes below introduce some layout changes that improve the UX for both of these critical features.

Option A: Simplify data source UX in the sidebar

🖼 https://marvelapp.com/43e9fci/screen/59547020

While this simplifies the UX to a degree, and makes the Data capabilities more visible (no more tabs), this approach still presents limitations with regards to available real estate when compared to Option B.

Option B: Move data source UX to a resizable bottom panel alongside the Expression editor

🖼 https://marvelapp.com/c7fe5ie/screen/59630879
Note: the hotspot links aren't working correctly (Marvel issue); use your arrow keys to navigate
Updated screenshot to include page manager button in top menu

We've discussed a resizable panel for the Expression editor (due to the size of expressions we've been seeing and hearing that users copy/paste from their editor) and it seems highly likely that users will/are facing the same hurdles when writing SQL queries (which seems to be the most used data source by a healthy margin).

Keeping the Expression and Data features in close proximity to one another has several benefits: 1) both can leverage the resizable panel (i.e. extra real estate) 2) these experiences are similar and should be treated as such 3) this brings visibility to both features ("always on") and 4) makes a stronger visual relationship between element and code/query.

Option C: A more traditional data wizard

I've intentionally not mocked up this idea for a couple of reasons.

First, Option B provides a simpler UX and also introduces the concept of an 'Elasticsearch basic' data source which could simplify things for new users and handle many basic data needs (i.e. I just need a field or two from an index).

Second, I'm reluctant to design away the instant gratification of adding an element to a workpad. By attaching a step to select data (potentially having to write SQL) before/after the 'add element' flow, we'd would likely be degrading that 'A-ha' moment. Introducing such a hurdle early on in the experience seems unlikely to bode well for adoption.

@elastic/kibana-canvas

[clintandrewhall]

I love Option B -- do these datasources all have icons? Might be worth playing with icon-based buttons instead of a dropdown... not sure, though. It was always nice on the data source tab to see the list of available data sources... perhaps it would be enough to default the expression panel to a set of large icons + text...?

[crob611]

In Option B, where do the Page Navigation controls end up?

[ryankeairns]

Good catch @crob611 !
I removed one too many icons from the top bar when exploring this latest work. It would appear on the left akin to Google Slides and other presentation software:

It's an area that could be looked into further, but this is my current recommendation.

[ryankeairns]

@clintandrewhall good call on using the icons/logos. Not only a nice visual touch, but it draws your eye to an important feature of the application and, I presume, encourages users to interact. Additionally, since you can only ever select one data source at a time, the checkmarks are unnecessary.

[shaunmcgough]

@ryankeairns I also ++ on option B here. Really like where it's headed.

[ThomThomson]

Closing this for the time being as we are currently not prioritizing Canvas data source UX.