[Upgrade Assistant] Remove ML/Watcher logic

[sabarasaba]

Summary

Currently UA shows in the ES deprecations page deprecations for system indices, and since for 7.16 system indices will be resolved separately through the system indices migration flow that will be shown in the Overview page (see #110593) we will have to remove them from the deprecations table (see #113280). At the moment since they are shown to the user and we allow them to resolve these deprecations, there were a few considerations we had to have when dealing with the reindexing of ML and Watcher indices:

• ML: If we would detect that a reindexing operation is performed on one of the following system indices '.ml-state', '.ml-anomalies', '.ml-config' then we would attempt to go ahead and set ML upgrade mode flag depending on whether we are starting or stopping the job. This was needed because upgrading ML indices requires setting the mlUpgradeMode setting accordingly, otherwise the re-indexing process would fail.
• Watcher: If we would detect that a reindexing operation is performed on one of the following system indices '.watches', '.triggered-watches' we would start/stop the watcher accordingly depending on whether we are stopping or starting the job. Doing so was crucial because those indices cannot be re-indexed if the watcher is started.


In terms of UX, this had a minimal impact for user since the only difference between a regular deprecation that required re-indexing and one for any of the above mentioned indices was just 2 extra steps for pausing ml_jobs/watcher before doing anything and another one for resuming the ml_job/watcher afterwards:

Therefore this PR removes the logic that was already setup for:

• dealing with stopping/resuming ML jobs when reindexing ML system indices
• dealing with stopping/resuming watcher when reindexing watcher system indices
• showing extra steps for the above cases in progress component shown in the reindexing flyout

How to test

1. Unzip 6.8-data-snapshot.zip and start up ES as follows: yarn es snapshot -E path.data=/path/to/6.8-data-snapshot. This contains indices and ML job model snapshots created on 6.8. This will trigger the reindex and ML deprecations.
2. Start up Kibana, yarn start.
3. Navigate to UA, open ES deprecation issues
4. Verify that for reindex deprecations for the above mentioned system indices there are no extra steps regarding pausing/resuming ml jobs or watcher.

If after following the steps you don't see any deprecations for es in upgrade assistant, try unzipping the folder in the kibana/.es folder and starting up es with: yarn es snapshot -E path.data=../6.8-data-snapshot

Note: We should hide system indices in the deprecations list after the ES team finishes the work for migrating system indices (see #113280)

[cjcenizal]

I see logic on the server regarding indexGroupServices. How is this related to the indexGroup stuff we're removing here? Do we want to remove it, too?

I also grepped for "ml" and found a bunch of hits. Do we want to remove everything ML-related?

[cjcenizal]

@sabarasaba Could you please update this PR with more context? Some questions I have are:

• What is special about the Watcher and ML indices that they require special reindexing?
• What kind of special reindexing did they require?
• How did this affect the UX -- did this require different buttons, flyouts, user flows, and so on? I'd like a complete catalogue of how the reindexing UX for each one diverged from the standard reindexing UX for a 6.x index.
• What is the new flow we expect in 7.16? For example (I'm making this up): "We expect to exclude Watcher and ML indices from the ES deprecation issues table entirely. We expect users to click a Migrate to system indices button which will convert these indices to system indices transparently to the user. As part of this conversion, Elasticsearch will stop and start relevant jobs and handle any necessary reindexing itself."

[sabarasaba]

@elasticmachine merge upstream

[sabarasaba]

Thanks for having a look @cjcenizal! I've updated the summary of the PR to include more context about the changes introduced here.

I see logic on the server regarding indexGroupServices. How is this related to the indexGroup stuff we're removing here? Do we want to remove it, too?

Yes good catch! Should've removed those steps from the state machine also since it was basically only setting the next step. See: c397883

I also grepped for "ml" and found a bunch of hits. Do we want to remove everything ML-related?

Unsure what things in particular you mean, but the code that we are removing now only pertains reindexing of system indices from ML. We should still keep ML deprecations for snapshots for example.

[cjcenizal]

Thanks for the great details you added to the PR description! I now understand that everything revolves around removing logic needed for the indices defined on lines 35 and 35 of reindex_service.ts, because those indices will be hidden from Upgrade Assistant entirely in a subsequent PR.

I had a couple questions regarding tests.

The code looks good to me, but I wasn't sure how to test locally. The last step to test is "Navigate to UA, open ES deprecation issues", but what's the behavior I should expect to see that's different from what exists on 7.x?

[sabarasaba]

Thanks for re-checking @cjcenizal!

The code looks good to me, but I wasn't sure how to test locally. The last step to test is "Navigate to UA, open ES deprecation issues", but what's the behavior I should expect to see that's different from what exists on 7.x?

👍🏼 I've updated the test steps with some more info, but basically just checking that the extra steps for pausing/resuming ml_jobs and watcher should be enough

[cjcenizal]

Thanks @sabarasaba! When I test locally I can confirm that the indices we're now ignoring have only the steps we expect to see.

Note for future work: I see some indices that weren't originally identified by the code as ML indices: .ml-annotations-6, .ml-anomalies-shared, and .ml-notifications. I expect that these will be converted to system indices. We should confirm this assumption.

I'm concerned about ones that have version suffixes, like .ml-annotations-6, or differentiation suffixes, like .ml-anomalies-shared. If we plan to filter out pre-migration system indices based on their name, then we'll need ML and other affected teams to define a contract of the index naming patterns that have been used in 6.x and 7.x so we can capture them all using regular expressions or something similar.

[sabarasaba]

@elasticmachine merge upstream

[sabarasaba]

Note for future work: I see some indices that weren't originally identified by the code as ML indices: .ml-annotations-6, .ml-anomalies-shared, and .ml-notifications. I expect that these will be converted to system indices. We should confirm this assumption.

I also suspect they will be covered by the system indices upgrade process, but I'll double check with the ES team before merging.

I'm concerned about ones that have version suffixes, like .ml-annotations-6, or differentiation suffixes, like .ml-anomalies-shared. If we plan to filter out pre-migration system indices based on their name, then we'll need ML and other affected teams to define a contract of the index naming patterns that have been used in 6.x and 7.x so we can capture them all using regular expressions or something similar.

I'm not sure this would be a problem, the API for getting the system indices that require upgrade returns a list of exact index names and so does the migration.deprecations() API, so it would just be a matter of comparing them before filtering them out.

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 2e54ec7
• Storybooks Preview

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
upgradeAssistant	126.1KB	124.9KB	-1.2KB

History

• 💚 Build #157021 succeeded 77e65ba
• 💚 Build #156872 succeeded c25a569
• 💔 Build #156839 failed ba3ec72
• 💔 Build #156804 failed 9dcec2e
• 💚 Build #156699 succeeded 6076a36

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @sabarasaba

[cjcenizal]

I'm not sure this would be a problem, the API for getting the system indices that require upgrade returns a list of exact index names

Great point!

[sabarasaba]

I see some indices that weren't originally identified by the code as ML indices: .ml-annotations-6, .ml-anomalies-shared, and .ml-notifications. I expect that these will be converted to system indices. We should confirm this assumption.

Confirmed with ES team that .ml-annotations-6, .ml-anomalies and others are covered by the ML plugin’s "associated index" patterns, and they will handle upgrading them.

Still waiting to get an answer to whether the following statement is true: If the system indices upgrade api shows that there are no pending upgrades for the index .ml-config then the migration.deprecations() api shouldn't return any index migration deprecations for it. I should get an answer for this today will will hold merging this PR until then.

[sabarasaba]

Got confirmation that the index deprecations are generated by checking the cluster metadata when the _migration/deprecation endpoint is called, and after the upgrade is completed, those index deprecation checks will not return anything.