Skip to content

Commit

Permalink
[docs][migrations v2] Update SO migration docs to include removal of …
Browse files Browse the repository at this point in the history
…index write block when handling corrupt SOs. (elastic#103014) (elastic#103372)

Co-authored-by: Luke Elmers <luke.elmers@elastic.co>
  • Loading branch information
kibanamachine and lukeelmers authored Jun 24, 2021
1 parent f0a29a2 commit 78a030d
Showing 1 changed file with 41 additions and 8 deletions.
49 changes: 41 additions & 8 deletions docs/setup/upgrade/upgrade-migrations.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -55,22 +55,55 @@ This section highlights common causes of {kib} upgrade failures and how to preve
There is a known issue in v7.12.0 for users who tried the fleet beta. Upgrade migrations fail because of a large number of documents in the `.kibana` index.

This can cause Kibana to log errors like:
> Error: Unable to complete saved object migrations for the [.kibana] index. Please check the health of your Elasticsearch cluster and try again. Error: [receive_timeout_transport_exception]: [instance-0000000002][10.32.1.112:19541][cluster:monitor/task/get] request_id [2648] timed out after [59940ms]
> Error: Unable to complete saved object migrations for the [.kibana] index. Please check the health of your Elasticsearch cluster and try again. Error: [timeout_exception]: Timed out waiting for completion of [org.elasticsearch.index.reindex.BulkByScrollTask@6a74c54]

[source,sh]
--------------------------------------------
Error: Unable to complete saved object migrations for the [.kibana] index. Please check the health of your Elasticsearch cluster and try again. Error: [receive_timeout_transport_exception]: [instance-0000000002][10.32.1.112:19541][cluster:monitor/task/get] request_id [2648] timed out after [59940ms]
Error: Unable to complete saved object migrations for the [.kibana] index. Please check the health of your Elasticsearch cluster and try again. Error: [timeout_exception]: Timed out waiting for completion of [org.elasticsearch.index.reindex.BulkByScrollTask@6a74c54]
--------------------------------------------

See https://github.com/elastic/kibana/issues/95321 for instructions to work around this issue.

[float]
===== Corrupt saved objects
We highly recommend testing your {kib} upgrade in a development cluster to discover and remedy problems caused by corrupt documents, especially when there are custom integrations creating saved objects in your environment. Saved objects that were corrupted through manual editing or integrations will cause migration failures with a log message like `Failed to transform document. Transform: index-pattern:7.0.0\n Doc: {...}` or `Unable to migrate the corrupt Saved Object document ...`. Corrupt documents will have to be fixed or deleted before an upgrade migration can succeed.
We highly recommend testing your {kib} upgrade in a development cluster to discover and remedy problems caused by corrupt documents, especially when there are custom integrations creating saved objects in your environment.

Saved objects that were corrupted through manual editing or integrations will cause migration failures with a log message like `Failed to transform document. Transform: index-pattern:7.0.0\n Doc: {...}` or `Unable to migrate the corrupt Saved Object document ...`. Corrupt documents will have to be fixed or deleted before an upgrade migration can succeed.

For example, given the following error message:
> Unable to migrate the corrupt saved object document with _id: 'marketing_space:dashboard:e3c5fc71-ac71-4805-bcab-2bcc9cc93275'. To allow migrations to proceed, please delete this document from the [.kibana_7.12.0_001] index.

The following steps must be followed to allow the upgrade migration to succeed.
Please be aware the Dashboard having ID `e3c5fc71-ac71-4805-bcab-2bcc9cc93275` belonging to the space `marketing_space` will no more be available:
1. Delete the corrupt document with `DELETE .kibana_7.12.0_001/_doc/marketing_space:dashboard:e3c5fc71-ac71-4805-bcab-2bcc9cc93275`
2. Restart {kib}
[source,sh]
--------------------------------------------
Unable to migrate the corrupt saved object document with _id: 'marketing_space:dashboard:e3c5fc71-ac71-4805-bcab-2bcc9cc93275'. To allow migrations to proceed, please delete this document from the [.kibana_7.12.0_001] index.
--------------------------------------------

The following steps must be followed to delete the document that is causing the migration to fail:

. Remove the write block which the migration system has placed on the previous index:
+
[source,sh]
--------------------------------------------
PUT .kibana_7.12.1_001/_settings
{
"index": {
"blocks.write": false
}
}
--------------------------------------------

. Delete the corrupt document:
+
[source,sh]
--------------------------------------------
DELETE .kibana_7.12.0_001/_doc/marketing_space:dashboard:e3c5fc71-ac71-4805-bcab-2bcc9cc93275
--------------------------------------------

. Restart {kib}.

In this example, the Dashboard with ID `e3c5fc71-ac71-4805-bcab-2bcc9cc93275` that belongs to the space `marketing_space` **will no longer be available**.

Be sure you have a snapshot before you delete the corrupt document. If restoring from a snapshot is not an option, it is recommended to also delete the `temp` and `target` indices the migration created before restarting {kib} and retrying.

[float]
===== User defined index templates that causes new `.kibana*` indices to have incompatible settings or mappings
Expand Down

0 comments on commit 78a030d

Please sign in to comment.