Kibana Discover tab broken the first time I go to it.

[Ahuge]

Hi, my Kibana gives a broken looking page that never loads the first time you go to the app. You can change to Settings, toggle the default indexpattern (or even just switch to another indexpattern) and it will work.

If I make my other index default and launch it, it works everytime. It seems like it is an issue with my index although the kibana log isn't telling my anything is wrong. The stderr one doesn't update and the stdout one is full of responses with 200 and 304 codes

Here is an image of what the page looks like when you load it first : http://imgur.com/U41sS3r

I am getting some errors in my chrome console
TypeError: Cannot read property 'byName' of undefined at isSortable (http://kibana/bundles/kibana.bundle.js:21853:34) at getSort (http://kibana/bundles/kibana.bundle.js:21860:47) at Function.getSort.array (http://kibana/bundles/kibana.bundle.js:21875:15) at getStateDefaults (http://kibana/bundles/kibana.bundle.js:21428:24) at new <anonymous> (http://kibana/bundles/kibana.bundle.js:21424:47) at Object.invoke (http://kibana/bundles/commons.bundle.js:32091:18) at extend.instance (http://kibana/bundles/commons.bundle.js:36749:35) at nodeLinkFn (http://kibana/bundles/commons.bundle.js:35861:37) at compositeLinkFn (http://kibana/bundles/commons.bundle.js:35293:14) at publicLinkFn (http://kibana/bundles/commons.bundle.js:35168:31) <div class="application ng-scope" ng-class="'tab-' + chrome.getActiveTabId('-none-') + ' ' + chrome.getApplicationClasses()" ng-view="" ng-controller="chrome.$$rootControllerConstruct as kibana">

That happens right away as the page loads.

And then a second or so later I get these errors

commons.bundle.js:38335 POST http://kibana/elasticsearch/.kibana/index-pattern/pipeline_tools 400 (Bad Request)(anonymous function) @ commons.bundle.js:38335sendReq @ commons.bundle.js:38128serverRequest @ commons.bundle.js:37835processQueue @ commons.bundle.js:42358(anonymous function) @ commons.bundle.js:42374Scope.$eval @ commons.bundle.js:43602Scope.$digest @ commons.bundle.js:43413(anonymous function) @ commons.bundle.js:43641completeOutstandingRequest @ commons.bundle.js:33120(anonymous function) @ commons.bundle.js:33397 commons.bundle.js:40090 Error: Request to Elasticsearch failed: "Bad Request" at kibana.bundle.js:78762 at processQueue (commons.bundle.js:42358) at commons.bundle.js:42374 at Scope.$eval (commons.bundle.js:43602) at Scope.$digest (commons.bundle.js:43413) at Scope.$apply (commons.bundle.js:43710) at done (commons.bundle.js:38159) at completeRequest (commons.bundle.js:38357) at XMLHttpRequest.requestLoaded (commons.bundle.js:38298)(anonymous function) @ commons.bundle.js:40090(anonymous function) @ commons.bundle.js:36859processQueue @ commons.bundle.js:42366(anonymous function) @ commons.bundle.js:42374Scope.$eval @ commons.bundle.js:43602Scope.$digest @ commons.bundle.js:43413Scope.$apply @ commons.bundle.js:43710done @ commons.bundle.js:38159completeRequest @ commons.bundle.js:38357requestLoaded @ commons.bundle.js:38298

and
commons.bundle.js:40090 Error: Request to Elasticsearch failed: "Bad Request" at kibana.bundle.js:78762 at processQueue (commons.bundle.js:42358) at commons.bundle.js:42374 at Scope.$eval (commons.bundle.js:43602) at Scope.$digest (commons.bundle.js:43413) at Scope.$apply (commons.bundle.js:43710) at done (commons.bundle.js:38159) at completeRequest (commons.bundle.js:38357) at XMLHttpRequest.requestLoaded (commons.bundle.js:38298)(anonymous function) @ commons.bundle.js:40090(anonymous function) @ commons.bundle.js:36859processQueue @ commons.bundle.js:42366(anonymous function) @ commons.bundle.js:42374Scope.$eval @ commons.bundle.js:43602Scope.$digest @ commons.bundle.js:43413Scope.$apply @ commons.bundle.js:43710done @ commons.bundle.js:38159completeRequest @ commons.bundle.js:38357requestLoaded @ commons.bundle.js:38298

Both of the latter two I am assuming are intimately related to it failing with the first error.
I am on Kibana
Version 4.3.3
Build 9542
Commit SHA b883321

[Ahuge]

It is also broken if I directly link to a dashboard like this

http://kibana/app/kibana#/dashboard/Validations
That will give me the byName error in the graphs. http://imgur.com/aJm62D6

It sounds similar to #1561

[epixa]

@Ahuge Sorry for the delay on this one. Are you still encountering this issue?

[Ahuge]

Hey Epixa, no worries and yes we still are. We took a look at our elastic cluster and we think we might know what this issue is. We have a lot of fields in our mapping. Like about 65,000 fields. It's a poor design on our part I know, we are working on transitioning to less fields. We think that is the issue and that when loading the page the first time it times out reading the mapping or something, we aren't totally sure though.
After we go to another page though it seems to pick up everything correctly

[ajodock]

I was having this same issue (or at least I was getting a screen matching your screenshot, I didn't catch the chrome console errors), but unfortunately it quit happening when I rolled off some of my indexes. So I can't do any further testing, but I had done some testing before that, and hopefully this helps.

In my case I was creating a new Kibana instance with its own .kibana index (we have several instances for different teams, each with their own saved searches/visualizations/etc). All of the old instances worked just fine, so I figured the issue must be in the ,kibana index.

Comparing the documents in the new index to the old ones the only difference was that "index-pattern" typed document had an empty/missing "fields" array. Copying the index-pattern document from a working instance to the new one fixed the broken instance.

My best guess was that there was either a issue with a field name (special character or something?), or else querying all of the mappings while creating the index mapping took longer than some timeout. I assume that the same thing would have happened to any of our old instances if someone had gone and done a refresh on the field list.

[epixa]

@Ahuge I'm confident that the number of fields is the issue here. It's a currently-known limitation of Kibana that it can't handle mappings with large numbers of fields: #1540. We want to get it fixed, but it's a hard problem to solve.

@ajodock Thanks for that info. It's possible that many different issues manifest in this same way.

[Ahuge]

Ok thanks @epixa

You can close this if you want, as the issue has already been recorded. We will work on restructuring to have less fields and once we complete that, I will report back confirming that solved it.

Thanks

[epixa]

@Ahuge Sounds good. Good luck!

[visaxin]

@Ahuge hi, I am interesting in your report back. I am faced the same problem now, I think it's probably caused by default-index-pattern.
Looking forward your reply.

Thanks

[ajodock]

@visaxin what version of Kibana are you on? It seems like the newer versions of Kibana still have this issue, but the symptoms have changed a bit.

I assume that you have time based indexes. Are your fields very consistent every day, or is there variation in your field names that is creating a lot of unique names? If your fields vary every day we did find a way to partially work around this issue by switching back to the old deprecated index mapping pattern ([logstash-]YYYY.MM.DD instead of logstash-*).

This helps because of this option: "indexPattern:fieldMapping:lookBack" This setting which is configurable in the advanced settings (defaults to 5) sets how many days back through your pattern that Kibana will look up field names, but it is only used with the old index mapping format.

So imagine you were keeping a whole years worth of indexes with 18 unique field names per day (unique as in never shows up in any other index). If you are using logstash-* you will pick up 6570 fields, but kibana breaks at around 6000, so you will run into the issues above. If you are using [logstash-]YYYY.MM.DD Kibana will only pick up 90 fields which means that Kibana will fully "work" for those 90 fields.

In our case we have a decent number of fields that always stay the same, which most of our visualizations use. However we have some fields that may change on a daily basis like procedurally generated field names. This is a problem because the number of unique field names add up quickly the longer your retention is. These unique fields are just there to be used as logs, and never really need to be mapped in Kibana, but there is no way to filter them out.

Using the deprecated index mapping pattern our static field names always work as expected because the common field names are always included in the last 5 days worth of indexes. The unique fields are visible, but don't generally affect Kibana.

For our use case we have dozens of teams all logging to the same index. We did it this way because in Kibana 3 you could only have one index pattern per dashboard which meant that you couldn't put data together from two different index patterns. This is no longer a problem for kibana 4 because each visualization can use a different mapping.

So the most ideal solution to this is to consider breaking up your logs into multiple indexes. This makes everything easier in Kibana (i.e. sorting through the field list dropdown on the visualizations), but might add some overhead to your ES cluster.