7.12 Upgrade migrations fail if the advanced setting "timepicker:quickRanges" is null

[rudolf]

#93409 Introduced a 7.12 migration for the config saved object that stores advanced settings. But if the advanced settings contains "timepicker:quickRanges": null the migration fails with logs like:

Error: Failed to transform document 7.7.0. Transform: config:7.12.0
Doc: {"type":"config","id":"7.7.0","attributes":{"buildNum":30810,"defaultIndex":"8864-b38e6d72e23e...","dateFormat:dow":"Monday","state:storeInSessionStorage":true,"timepicker:quickRanges":null,"search:includeFrozen":null,"theme:darkMode":true,"courier:ignoreFilterIfFieldNotInIndex":true,"courier:batchSearches":null},"references":[],"migrationVersion":{"config":"7.9.0"},"updated_at":"2020-05-15T15:40:07.716Z"}
at tryTransformDoc (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:562:13)
at migrateProp (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:631:22)
at applyMigrations (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:350:20)

TypeError: Cannot read property 'indexOf' of null at /usr/share/kibana/src/core/server/ui_settings/saved_objects/migrations.js:29:69 at Array.reduce () at 7.12.0 (/usr/share/kibana/src/core/server/ui_settings/saved_objects/migrations.js:28:47) at tryTransformDoc (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:547:22) at migrateProp (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:631:22) at applyMigrations (/usr/share/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:350:20)

Workaround:

1. Stop Kibana
2. Rollback from the failed migration https://www.elastic.co/guide/en/kibana/7.x/upgrade-migrations.html#upgrade-migrations-rolling-back
3. Remove any "timepicker:quickRanges":null keys from the "type":"config" documents:

   POST .kibana/_update_by_query
   {
     "query": {
       "bool": {
         "must": 
           {
             "term": {
               "type": "config"
             }
           }
       }
     },
    "script" : {
      "lang": "painless",
      "source": "if (ctx._source.config['timepicker:quickRanges'] == null) {ctx._source.config.remove('timepicker:quickRanges')} else {ctx.op = \"noop\"}"
      }
   }
   

4. restart Kibana

[elasticmachine]

Pinging @elastic/kibana-app-services (Team:AppServices)

[rudolf]

Although the server-side validation will accept null or an array, it doesn't seem like it's possible to set this value to null with the UI but it's hard to say if this was possible in earlier versions.

[gaby]

This issue is happening with any field in the Advanced Settings section. I'm having it with visualization:tileMap:WMSdefaults. Upgrading to kibana 7.12.0 fails to start. Reverting back to 7.11.1 made our Kibana read-only, we are no longer able to click Save in the settings page.

Our whole ELK installation is now in a limbo state, since Elastic/Logstash are in 7.12.0 and Kibana is stuck in read-only mode with 7.11.1

[rudolf]

@gaby If you don't have a snapshot to restore to and want to rollback to a previous version will need to follow the instructions in https://www.elastic.co/guide/en/kibana/current/upgrade-migrations.html#_not_recommended_rollback_without_a_backup_snapshot

Did you try the suggested workaround, this should enable you to successfully upgrade to 7.12.0

[gaby]

I'm unable to try the work-around using Kibana Dev Tools, because every request gets a HTTP 403. This is happening even using the elastic superuser. I will have to try next week directly from one of the Elastic nodes.

[gaby]

@rudolf I was able to recover my Kibana instance by:

• Exporting all my saved objects
• Stopping Kibana
• Running `curl -k X DELETE https://username:password@hostname:9200/.kibana*
• Starting kibana again with 7.12.0 Docker image
• Importing my saved objects again.
• Several of the Advanced Settings imports failed with HTTP:500, Fix was to manually modify the Saved Objects ndjson and re-import the items.

Ideally Kibana should popup an error an give you an option of Skipping the failed object in the import, instead of throwing an HTTP 500 with no messages.