[migration v6.5] recreate .kibana index

[gmeriaux]

Kibana version: 6.5

Elasticsearch version: 6.5

Server OS version: debian 8

Browser version:

Browser OS version:

Original install method (e.g. download page, yum, from source, etc.): apt-get

Describe the bug: Since yesterday, I tried upgrade ELK stack to 6.5 and I am stuck on kibana part. At the beginning, I was thinking this bug this bug was like this but it is strange because I haven't got X-Pack Security.
I tried delete .kibana_1, .kibana_2 without success so I finished by delete .kibana to reset my index.
When I restart kibana I have finally a strange timeout.
In kibana.stdout :
{"type":"log","@timestamp":"2018-11-16T11:20:01Z","tags":["info","migrations"],"pid":18753,"message":"Creating index .kibana_1."} {"type":"log","@timestamp":"2018-11-16T11:20:32Z","tags":["status","plugin:spaces@6.5.0","error"],"pid":18753,"state":"red","message":"Status changed from yellow to red - Request Timeout after 30000ms","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"} {"type":"error","@timestamp":"2018-11-16T11:20:32Z","tags":["fatal","root"],"pid":18753,"level":"fatal","error":{"message":"Request Timeout after 30000ms","name":"Error","stack":"Error: Request Timeout after 30000ms\n at /usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:355:15\n at Timeout.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:384:7)\n at ontimeout (timers.js:498:11)\n at tryOnTimeout (timers.js:323:5)\n at Timer.listOnTimeout (timers.js:290:5)"},"message":"Request Timeout after 30000ms"}

In elasticSearch.log :
[2018-11-16T12:19:45,897][INFO ][o.e.c.m.MetaDataDeleteIndexService] [skNMpPx] [.kibana/7wEYtjwTT3alUWJM9bwClg] deleting index [2018-11-16T12:20:00,734][INFO ][o.e.c.m.MetaDataIndexTemplateService] [skNMpPx] adding template [.management-beats] for index patterns [.management-beats] [2018-11-16T12:20:01,998][INFO ][o.e.c.m.MetaDataCreateIndexService] [skNMpPx] [.kibana_1] creating index, cause [api], templates [], shards [1]/[1], mappings [doc] [2018-11-16T12:20:01,999][INFO ][o.e.c.r.a.AllocationService] [skNMpPx] updating number_of_replicas to [0] for indices [.kibana_1] [2018-11-16T12:20:02,001][INFO ][o.e.c.r.a.AllocationService] [skNMpPx] Cluster health status changed from [YELLOW] to [RED] (reason: [index [.kibana_1] created]).

I didn't success to downgrade also elastic in 6.4.3.

Steps to reproduce:

1. Delete kibana index
2. Restart kibana

Expected behavior:

Screenshots (if relevant):

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

Any additional context:

[tylersmalley]

Are these logs from the original upgrade? Or when you tried resetting your index?

It looks like Elasticsearch became un-available around the same time it began migrating the index. Can you confirm this?

[gmeriaux]

This log arrive when I try reset my index. elasticsearch stay available on my two networks configured (eth1.11 et local)
.kibana index is never created. all shards failed when I try search on .kibana_1.

I don't know if it can help, here is the mapping applied to .kibana_1:

{
".kibana_1": {
    "aliases": {},
    "mappings": {
        "doc": {
            "dynamic": "strict",
            "properties": {
                "canvas-workpad": {
                    "dynamic": "false",
                    "properties": {
                        "@created": {
                            "type": "date"
                        },
                        "@timestamp": {
                            "type": "date"
                        },
                        "id": {
                            "type": "text",
                            "index": false
                        },
                        "name": {
                            "type": "text",
                            "fields": {
                                "keyword": {
                                    "type": "keyword"
                                }
                            }
                        }
                    }
                },
                "config": {
                    "dynamic": "true",
                    "properties": {
                        "buildNum": {
                            "type": "keyword"
                        }
                    }
                },
                "dashboard": {
                    "properties": {
                        "description": {
                            "type": "text"
                        },
                        "hits": {
                            "type": "integer"
                        },
                        "kibanaSavedObjectMeta": {
                            "properties": {
                                "searchSourceJSON": {
                                    "type": "text"
                                }
                            }
                        },
                        "optionsJSON": {
                            "type": "text"
                        },
                        "panelsJSON": {
                            "type": "text"
                        },
                        "refreshInterval": {
                            "properties": {
                                "display": {
                                    "type": "keyword"
                                },
                                "pause": {
                                    "type": "boolean"
                                },
                                "section": {
                                    "type": "integer"
                                },
                                "value": {
                                    "type": "integer"
                                }
                            }
                        },
                        "timeFrom": {
                            "type": "keyword"
                        },
                        "timeRestore": {
                            "type": "boolean"
                        },
                        "timeTo": {
                            "type": "keyword"
                        },
                        "title": {
                            "type": "text"
                        },
                        "uiStateJSON": {
                            "type": "text"
                        },
                        "version": {
                            "type": "integer"
                        }
                    }
                },
                "graph-workspace": {
                    "properties": {
                        "description": {
                            "type": "text"
                        },
                        "kibanaSavedObjectMeta": {
                            "properties": {
                                "searchSourceJSON": {
                                    "type": "text"
                                }
                            }
                        },
                        "numLinks": {
                            "type": "integer"
                        },
                        "numVertices": {
                            "type": "integer"
                        },
                        "title": {
                            "type": "text"
                        },
                        "version": {
                            "type": "integer"
                        },
                        "wsState": {
                            "type": "text"
                        }
                    }
                },
                "index-pattern": {
                    "properties": {
                        "fieldFormatMap": {
                            "type": "text"
                        },
                        "fields": {
                            "type": "text"
                        },
                        "intervalName": {
                            "type": "keyword"
                        },
                        "notExpandable": {
                            "type": "boolean"
                        },
                        "sourceFilters": {
                            "type": "text"
                        },
                        "timeFieldName": {
                            "type": "keyword"
                        },
                        "title": {
                            "type": "text"
                        },
                        "type": {
                            "type": "keyword"
                        },
                        "typeMeta": {
                            "type": "keyword"
                        }
                    }
                },
                "kql-telemetry": {
                    "properties": {
                        "optInCount": {
                            "type": "long"
                        },
                        "optOutCount": {
                            "type": "long"
                        }
                    }
                },
                "migrationVersion": {
                    "type": "object",
                    "dynamic": "true"
                },
                "namespace": {
                    "type": "keyword"
                },
                "search": {
                    "properties": {
                        "columns": {
                            "type": "keyword"
                        },
                        "description": {
                            "type": "text"
                        },
                        "hits": {
                            "type": "integer"
                        },
                        "kibanaSavedObjectMeta": {
                            "properties": {
                                "searchSourceJSON": {
                                    "type": "text"
                                }
                            }
                        },
                        "sort": {
                            "type": "keyword"
                        },
                        "title": {
                            "type": "text"
                        },
                        "version": {
                            "type": "integer"
                        }
                    }
                },
                "server": {
                    "properties": {
                        "uuid": {
                            "type": "keyword"
                        }
                    }
                },
                "space": {
                    "properties": {
                        "_reserved": {
                            "type": "boolean"
                        },
                        "color": {
                            "type": "keyword"
                        },
                        "description": {
                            "type": "text"
                        },
                        "initials": {
                            "type": "keyword"
                        },
                        "name": {
                            "type": "text",
                            "fields": {
                                "keyword": {
                                    "type": "keyword",
                                    "ignore_above": 2048
                                }
                            }
                        }
                    }
                },
                "telemetry": {
                    "properties": {
                        "enabled": {
                            "type": "boolean"
                        }
                    }
                },
                "timelion-sheet": {
                    "properties": {
                        "description": {
                            "type": "text"
                        },
                        "hits": {
                            "type": "integer"
                        },
                        "kibanaSavedObjectMeta": {
                            "properties": {
                                "searchSourceJSON": {
                                    "type": "text"
                                }
                            }
                        },
                        "timelion_chart_height": {
                            "type": "integer"
                        },
                        "timelion_columns": {
                            "type": "integer"
                        },
                        "timelion_interval": {
                            "type": "keyword"
                        },
                        "timelion_other_interval": {
                            "type": "keyword"
                        },
                        "timelion_rows": {
                            "type": "integer"
                        },
                        "timelion_sheet": {
                            "type": "text"
                        },
                        "title": {
                            "type": "text"
                        },
                        "version": {
                            "type": "integer"
                        }
                    }
                },
                "type": {
                    "type": "keyword"
                },
                "updated_at": {
                    "type": "date"
                },
                "url": {
                    "properties": {
                        "accessCount": {
                            "type": "long"
                        },
                        "accessDate": {
                            "type": "date"
                        },
                        "createDate": {
                            "type": "date"
                        },
                        "url": {
                            "type": "text",
                            "fields": {
                                "keyword": {
                                    "type": "keyword",
                                    "ignore_above": 2048
                                }
                            }
                        }
                    }
                },
                "visualization": {
                    "properties": {
                        "description": {
                            "type": "text"
                        },
                        "kibanaSavedObjectMeta": {
                            "properties": {
                                "searchSourceJSON": {
                                    "type": "text"
                                }
                            }
                        },
                        "savedSearchId": {
                            "type": "keyword"
                        },
                        "title": {
                            "type": "text"
                        },
                        "uiStateJSON": {
                            "type": "text"
                        },
                        "version": {
                            "type": "integer"
                        },
                        "visState": {
                            "type": "text"
                        }
                    }
                }
            }
        }
    },
    "settings": {
        "index": {
            "number_of_shards": "1",
            "auto_expand_replicas": "0-1",
            "provided_name": ".kibana_1",
            "creation_date": "1542620415571",
            "number_of_replicas": "0",
            "uuid": "kCZ_KmBnTuua89gp73kCnA",
            "version": {
                "created": "6050099"
            }
        }
    }
}

}

[elasticmachine]

Pinging @elastic/kibana-platform

[elasticmachine]

Pinging @elastic/kibana-operations

[Moireabh]

experiencing the same problem but forgot to set back: "cluster.routing.allocation.enable": null
then all was fine

[gmeriaux]

@LordBonkerz Thanks a lot, everything work better after set back "cluster.routing.allocation.enable": null.

A error message more explicit could be a good improvement.

[tylersmalley]

Thanks all,

This cause is new to me. Is there something we could be checking for in Kibana to identify this? Is disabling shart allocation a normal practice during an upgrade?

[gmeriaux]

Disabling shard allocation is advice for elasticsearch upgrade. When you upgrade ELK stack, you need be very careful to finish elasticsearch upgrade before begin kibana upgrade.
During the upgrade phase, before the migration of .kibana index, is it possible to check the setting "cluster.routing.allocation.enable" and alert user instead of trying to create the index and receive an unfortunately request timeout?

[jparkrr]

This happened to me too! Would be very nice to have a better error message.

[kosxe]

this error message is still the same, and the only way I was able to get here, was by searching the error AFTER the fact. Could this be a warning when upgrade is initiated? - this is really frustrating to deal with