Index pattern creation UI/UX improvements

[cjcenizal]

Overview

Broadly speaking, index pattern creation will change in a few fundamental ways:

1. Simplified creation. Users will be able to create index patterns by typing in a pattern and (optionally) selecting a time field. There won't be any other configuration options associated with an index pattern.
2. Better, more helpful copy. The copy will be refined to clearly explain what the user is doing and why they're doing it, with links out to the documentation.
3. More feedback. The user will have more feedback about existing indices, whether or not the pattern matches any indices, and whether data is still being fetched or has successfully been fetched.
4. Standard CRUD UX. We'll have a landing page that lists all of your index patterns in a table. From this page, you can create a new index pattern or click on one in the list to view its details on a dedicated page.

Top priority

• Show users which indices they have via typeahead, and remove default pattern (Index pattern creation UX: Show users which indices they have #10438)

Initial improvements

Let's improve the UI and UX incrementally and, in the process, simplify the business logic and architecture.

• Simplify index-pattern creation form (Index pattern creation UX: Simplify index-pattern creation form #10444)
• Split up index pattern creation form into 3 forms (Index pattern creation UX: Split the form up into 3 forms #10026)
• Surface "no matching indices" error more clearly (Index pattern creation UX: Surface "no matching indices" error more clearly #10436)
• Improve explanatory copy (Index pattern creation UX: Improve explanatory copy #10437)
• Surface time fields loading state (Index pattern creation UX: Surface time fields loading state #10439)

Follow-up improvements

These are improvements that we can commit to, which build upon the initial improvements.

• Remove "Do not expand index pattern" checkbox (Index pattern creation UX: Remove "Do not expand index pattern" checkbox #10445)
• Implement standard CRUD flow (Index pattern UX: Implement CRUD flow #10442)

Long-term improvements

These are general, broad-stroke ideas which need more exploration before we can commit them.

• Remove UI for creating an index pattern based on event times (Index pattern creation UX: Remove UI for creating index patterns based on event times #10443)
• Rework creation flow to be a wizard, like in Visualize (Index pattern creation UX: Implement wizard flow #10440)

[cjcenizal]

@skearns64 Could you explain your idea regarding index templates (time series vs. rolled over)? I'm not familiar with these terms, so you might need to explain this like you would to a 5th grader (or link to some resources so I can study up).

[skearns64]

@cjcenizal - sure thing.

In ES, in production, it's important to have a good set of mappings (this field is searchable text, that field is a keyword, so we can aggregate on it), and index settings (things like number of shards and replicas) for each index. When you have a set of time series or rollover indexes, you don't want to be manually creating each index and manually specifying the mappings and index settings - you'll forget or get it wrong. Enter: Index Templates. An index template lets you specify a set of mappings and settings that apply to any newly-created index that matches the template pattern. This is true whether it's a time-series index, or whether you're using the roll-over api; anytime you want consistent mappings and settings for a set of indices that get created over time, you should be using an index template in ES. Logstash uses one. Beats use them, so do Watcher, monitoring, etc., for the time-series indexes.

So, what does this have to do with Kibana index patterns? If you look at an index template (see doc link above for example), it has a template field, which contains.. a pattern string that defines which newly-created indices should receive the settings/mappings. It's essentially the same syntax that Kibana Index Patterns support.

So in an ideal world (like phase 3 or beyond), the "index name type-ahead support" of the Kibana Index Pattern page would show you two things: The set of unique Index Template template pattern fields across all index templates.. AND any indexes that do not match any of the index template patterns (to catch those indexes that are not time-series or roll-over). This wouldn't catch all cases, but it would catch the vast majority in an intuitive way, so we could consider how to show the full-set of indexes in the type-ahead box, perhaps grouped - index templates, Indexes without templates, and "All" indexes. The UX might be tricky, but if we could make it work, we could make the common use-cases significantly easier.

[alexfrancoeur]

@cjcenizal Nearly all of the themes I had thought about to improve the getting started / index pattern creation experience were addressed during the UI review. This might be a bit repetitive but as I see it we covered the following:

• Simplifying the UI to focus on what matters - creating an index pattern, regardless of type
• Removing the advertisement of deprecated features
• Making the copy more concise and friendly to both new and power users
• Helping the user create an index pattern utilizing auto-completion and already available indices
• Providing reactions to every action a user has - loading indicators, validation experience, etc.

There were a few other things I'd like to highlight as well. Both of these themes came up during a discussion with a user about some of his frustrations Kibana's getting started experience

Progression

In other products, the getting started experience is a progression. You feel as if you are continually making progress to an end goal. At a high level, we may plan to address this in #10440 but I believe we are missing a step here. This is a great progression through setting up your index pattern - but then what? We should direct a user to one or many next actions they can take. This will expedite the next theme I mention below

Reducing "time to query" for new users

This specific user brought up an interesting concept he referred to as "time to query". This metric is essentially the time it takes for me to set up the stack and play with (any) data. To test out elastic he spun up Elastic cloud, injected some sample data and then got stuck on the index pattern page. Using progression as a theme here, we can certainly make that experience a bit better.

To address "time to query" for net new users, we could consider shipping Kibana with sample data. It would be a small data set preloaded with timestamps relative to the installation or an opt-in approach that easily installs and injects data into the related elasticsearch cluster. We do not want to make this a hinderance for existing users, but it might be something to consider for new users. This would significantly reduce "time to query" by removing needing the getting started guide to inject example data. At the very least, a new user will understand how powerful Kibana can be without needing to do much.

Getting data in

This theme is also related to the two above. This plays both into progression and reducing time to query. We do not have a natural progression for getting data in, we just assume it is already there. Competing UI's do a nice job of this as the first step in their "Wizard" and we could benefit from taking a similar approach. Sample data could simply be an option here in this first "step" to reducing time to query for new users. Along with features such as import CSV.

Also, some related issues I came across previously.

Related Issues

Split up index pattern creation into 3 forms #10026
Index Pattern page to display the error response #10100
Friendly index pattern names #7305
Kibana should suggest index patterns #5053
Tutorial built into Kibana using overlays #3379

There is a lot here and I'd love to have some follow up calls to discuss in more detail whenever you and the team are ready.

[cjcenizal]

Per convo with @skearns64, we should prioritize implementing typeahead support (#10438), which will allow us to remove the logstash-* default pattern, which @jimgoodwin mentioned has been confusing for some users, since it makes them think they need Logstash to continue.

[chrisronline]

@cjcenizal Can this be closed?

[cjcenizal]

@chrisronline Yes it can! Thanks man.