Skip to content

Commit

Permalink
update UA readme
Browse files Browse the repository at this point in the history
  • Loading branch information
@alisonelizabeth
alisonelizabeth committed Aug 20, 2021
1 parent a7289d1 commit 12711f8
Showing 1 changed file with 167 additions and 43 deletions.
210 changes: 167 additions & 43 deletions x-pack/plugins/upgrade_assistant/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,66 +2,190 @@

## About

Upgrade Assistant helps users prepare their Stack for being upgraded to the next major. Its primary
purposes are to:
Upgrade Assistant helps users prepare their Stack for being upgraded to the next major. It will only be enabled on the last minor before the next major release. This is controlled via the config: `xpack.upgrade_assistant.readonly` ([#101296](https://github.com/elastic/kibana/pull/101296)).

* **Surface deprecations.** Deprecations are features that are currently being used that will be
removed in the next major. Surfacing tells the user that there's a problem preventing them
from upgrading.
* **Migrate from deprecation features to supported features.** This addresses the problem, clearing
the path for the upgrade. Generally speaking, once all deprecations are addressed, the user can
safely upgrade.
Its primary purposes are to:

* **Surface deprecations.** Deprecations are features that are currently being used that will be removed in the next major. Surfacing tells the user that there's a problem preventing them from upgrading.
* **Migrate from deprecated features to supported features.** This addresses the problem, clearing the path for the upgrade. Generally speaking, once all deprecations are addressed, the user can safely upgrade.

### Deprecations

There are two sources of deprecation information:
There are three sources of deprecation information:

* [**Deprecation Info API.**](https://www.elastic.co/guide/en/elasticsearch/reference/master/migration-api-deprecation.html)
This is information about cluster, node, and index level settings that use deprecated features that
will be removed or changed in the next major version. Currently, only cluster and index deprecations
will be surfaced in the Upgrade Assistant. ES server engineers are responsible for adding
deprecations to the Deprecation Info API.
* [**Deprecation logs.**](https://www.elastic.co/guide/en/elasticsearch/reference/current/logging.html#deprecation-logging)
* [**Elasticsearch Deprecation Info API.**](https://www.elastic.co/guide/en/elasticsearch/reference/master/migration-api-deprecation.html)
This is information about Elasticsearch cluster, node, Machine Learning, and index-level settings that use deprecated features that will be removed or changed in the next major version. ES server engineers are responsible for adding deprecations to the Deprecation Info API.
* [**Elasticsearch deprecation logs.**](https://www.elastic.co/guide/en/elasticsearch/reference/current/logging.html#deprecation-logging)
These surface runtime deprecations, e.g. a Painless script that uses a deprecated accessor or a
request to a deprecated API. These are also generally surfaced as deprecation headers within the
response. Even if the cluster state is good, app maintainers need to watch the logs in case
deprecations are discovered as data is migrated.
deprecations are discovered as data is migrated. Starting in 7.x, deprecation logs can be written to a file or a data stream ([#58924](https://github.com/elastic/elasticsearch/pull/58924)). When the data stream exists, the Upgrade Assistant provides a way to analyze the logs through Observability or Discover ([#106521](https://github.com/elastic/kibana/pull/106521)).
* [**Kibana deprecations API.**](https://github.com/elastic/kibana/blob/master/src/core/server/deprecations/README.mdx) This is information about deprecated features and configs in Kibana. These deprecations are only communicated to the user if the deployment is using these features. Kibana engineers are responsible for adding deprecations to the deprecations API for their respective team.

### Fixing problems

Problems can be fixed at various points in the upgrade process. The Upgrade Assistant supports
various upgrade paths and surfaces various types of upgrade-related issues.

* **Fixing deprecated cluster settings pre-upgrade.** This generally requires fixing some settings
in `elasticsearch.yml`.
* **Migrating indices data pre-upgrade.** This can involve deleting indices so that ES can rebuild
them in the new version, reindexing them so that they're built using a new Lucene version, or
applying a migration script that reindexes them with new settings/mappings/etc.
* **Migrating indices data post-upgrade.** As was the case with APM in the 6.8->7.x upgrade,
sometimes the new data format isn't forwards-compatible. In these cases, the user will perform the
upgrade first and then use the Upgrade Assistant to reindex their data to be compatible with the new
version.

Deprecations can be handled in a number of ways:

* **Reindexing.** When a user's index contains deprecations (e.g. mappings) a reindex solves them.
Upgrade Assistant contains migration scripts that are executed as part of the reindex process.
The user will see a "Reindex" button they can click which will apply this script and perform the
reindex.
#### Elasticsearch

Elasticsearch deprecations can be handled in a number of ways:

* **Reindexing.** When a user's index contains deprecations (e.g. mappings) a reindex solves them. Currently, the Upgrade Assistant only automates reindexing for old indices. For example, if you are currently on 7.x, and want to migrate to 8.0, but you still have indices that were created on 6.x. For this scenario, the user will see a "Reindex" button that they can click, which will perform the reindex.
* Reindexing is an atomic process in Upgrade Assistant, so that ingestion is never disrupted.
It works like this:
* Create a new index with a "reindexed-" prefix ([#30114](https://github.com/elastic/kibana/pull/30114)).
* Create an index alias pointing from the original index name to the prefixed index name.
* Reindex from the original index into the prefixed index.
* Delete the old index and rename the prefixed index.
* Some apps might require custom scripts, as was the case with APM ([#29845](https://github.com/elastic/kibana/pull/29845)).
In that case the migration performed a reindex with a Painless script (covered by automated tests)
that made the required changes to the data.
* **Update index settings.** Some index settings will need to be updated, which doesn't require a
reindex. An example of this is the "Fix" button that was added for metricbeat and filebeat indices
([#32829](https://github.com/elastic/kibana/pull/32829), [#33439](https://github.com/elastic/kibana/pull/33439)).
* **Updating index settings.** Some index settings will need to be updated, which doesn't require a
reindex. An example of this is the "Remove deprecated settings" button, which is shown when [deprecated translog settings](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-translog.html#index-modules-translog-retention) are detected ([#93293](https://github.com/elastic/kibana/pull/93293)).
* **Upgrading or deleting snapshots**. This is specific to Machine Learning. If a user has old Machine Learning job model snapshots, they will need to be upgraded or deleted. The Upgrade Assistant provides a way to resolve this automatically for the user ([#100066](https://github.com/elastic/kibana/pull/100066)).
* **Following the docs.** The Deprecation Info API provides links to the deprecation docs. Users
will follow these docs to address the problem and make these warnings or errors disappear in the
Upgrade Assistant.
* **Stopping/restarting tasks and jobs.** Users had to stop watches and ML jobs and restart them as
soon as reindexing was complete ([#29663](https://github.com/elastic/kibana/pull/29663)).

#### Kibana

Kibana deprecations can be handled in one of two ways:

* **Automatic resolution.** Some deprecations can be fixed automatically through Upgrade Assistant via an API call. When this is possible, users will see a "Quick resolve" button in the Upgrade Assistant.
* **Manual steps.** For deprecations that require the user to address manually, the Upgrade Assistant provides a list of steps to follow as well as a link to documentation. Once the deprecation is addressed, it will no longer appear in the Upgrade Assistant.

### Steps for testing
#### Elasticsearch deprecations

To test the Elasticsearch deprecations page ([#107053](https://github.com/elastic/kibana/pull/107053)), you will first need to create a set of deprecations that will be returned from the deprecation info API.

**1. Reindexing**

The reindex action appears in UA whenever the deprecation `Index created before XX` is encountered. To reproduce, you will need to start up a cluster on the previous major version (e.g., if you are running 7.x, start a 6.8 cluster). Create a handful of indices, for example:

```
PUT my_index
```

Next, point to the 6.x data directory when running from a 7.x cluster.

```
yarn es snapshot -E path.data=./path_to_6.x_indices
```

**2. Upgrading or deleting ML job model snapshots**

Similar to the reindex action, the ML action requires setting up a cluster on the previous major version. It also requires the trial license to be enabled. Then, you will need to create a few ML jobs in order to trigger snapshots.

- Add the Kibana sample data.
- Navigate to Machine Learning > Create new job.
- Select `kibana_sample_data_flights` index.
- Select "Single metric job".
- Add an aggregation, field, and job ID. Change the time range to "Absolute" and select a subset of time.
- Click "Create job"
- View the job created and click the "Start datafeed" action associated with it. Select a subset of time and click "Start". You should now have two snapshots created. If you want to add more, repeat the steps above.

Next, point to the 6.x data directory when running from a 7.x cluster.

```
yarn es snapshot --license trial -E path.data=./path_to_6.x_ml_snapshots
```

**3. Removing deprecated index settings**

The Upgrade Assistant currently only supports fixing deprecated translog index settings. However [the code](https://github.com/elastic/kibana/blob/master/x-pack/plugins/upgrade_assistant/common/constants.ts#L22) is written in a way to add support for more if necessary. Run the following Console command to trigger the deprecation warning:

```
PUT deprecated_settings
{
"settings": {
"translog.retention.size": "1b",
"translog.retention.age": "5m",
"index.soft_deletes.enabled": true,
}
}
```

**4. Other deprecations with no automatic resolutions**

Many deprecations emitted from the deprecation info API are too complex to provide an automatic resolution for in UA. In this case, UA provides details about the deprecation as well as a link to documentation. The following requests will emit deprecations from the deprecation info API. This list is *not* exhaustive of all possible deprecations. You can find the full list of [7.x deprecations in the Elasticsearch repo](https://github.com/elastic/elasticsearch/tree/7.x/x-pack/plugin/deprecation/src/main/java/org/elasticsearch/xpack/deprecation) by grepping `new DeprecationIssue` in the code.

```
PUT /nested_multi_fields
{
"mappings":{
"properties":{
"text":{
"type":"text",
"fields":{
"english":{
"type":"text",
"analyzer":"english",
"fields":{
"english":{
"type":"text",
"analyzer":"english"
}
}
}
}
}
}
}
}
```

```
PUT field_names_enabled
{
"mappings": {
"_field_names": {
"enabled": false
}
}
}
```

```
PUT /_cluster/settings
{
"persistent" : {
"indices.lifecycle.poll_interval" : "500ms"
}
}
```

```
PUT _template/field_names_enabled
{
"index_patterns": ["foo"],
"mappings": {
"_field_names": {
"enabled": false
}
}
}
```

```
// This is only applicable for indices created prior to 7.x
PUT joda_time
{
"mappings" : {
"properties" : {
"datetime": {
"type": "date",
"format": "yyyy/MM/dd HH:mm:ss||yyyy/MM/dd||epoch_millis"
}
}
}
}
```

#### Kibana deprecations
To test the Kibana deprecations page, you will first need to create a set of deprecations that will be returned from the Kibana deprecations API. Refer to the [8.0 Kibana deprecations meta issue](https://github.com/elastic/kibana/issues/109166) for example deprecations and steps to reproduce.

### Errors

This is a non-exhaustive list of different error scenarios in Upgrade Assistant. It's recommended to use the [tweak browser extension](https://chrome.google.com/webstore/detail/tweak-mock-api-calls/feahianecghpnipmhphmfgmpdodhcapi?hl=en), or something similar, to mock the API calls.

- **Error loading deprecation logging status.** Mock a `404` status code to `GET /api/upgrade_assistant/deprecation_logging`. Alternatively, edit [this line](https://github.com/elastic/kibana/blob/545c1420c285af8f5eee56f414bd6eca735aea11/x-pack/plugins/upgrade_assistant/public/application/lib/api.ts#L70) locally and replace `deprecation_logging` with `fake_deprecation_logging`.
- **Error updating deprecation logging status.** Mock a `404` status code to `PUT /api/upgrade_assistant/deprecation_logging`. Alternatively, edit [this line](https://github.com/elastic/kibana/blob/545c1420c285af8f5eee56f414bd6eca735aea11/x-pack/plugins/upgrade_assistant/public/application/lib/api.ts#L77) locally and replace `deprecation_logging` with `fake_deprecation_logging`.
- **Unauthorized error fetching ES deprecations.** Mock a `403` status code to `GET /api/upgrade_assistant/es_deprecations` with the response payload: `{ "statusCode": 403 }`
- **Partially upgraded error fetching ES deprecations.** Mock a `426` status code to `GET /api/upgrade_assistant/es_deprecations` with the response payload: `{ "statusCode": 426, "attributes": { "allNodesUpgraded": false } }`
- **Upgraded error fetching ES deprecations.** Mock a `426` status code to `GET /api/upgrade_assistant/es_deprecations` with the response payload: `{ "statusCode": 426, "attributes": { "allNodesUpgraded": true } }`

0 comments on commit 12711f8

Please sign in to comment.