Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds an Alias action #304

Closed
wants to merge 1 commit into from
Closed

Conversation

dbbaughe
Copy link
Contributor

@dbbaughe dbbaughe commented Mar 11, 2022

Signed-off-by: Drew Baugher 46505179+dbbaughe@users.noreply.github.com

Issue #, if available:
#35

Description of changes:
Adds support for an alias action in ISM.
Gives complete support for all update alias API options by reusing the alias actions parser internally and explicitly not defining the mappings in our config index.

Coming from the GitHub feature request, the use cases people asked for:

  1. One use case we have is for indices older than a certain age, we allocate them to a cold node, shrink them, delete the original and create an alias with the original name that points to the shrink index. This keeps our shard count down while not breaking queries for older data.
    1.1. As of right now I don't believe this would be doable because we do not know the name of the source index that was used to shrink to this target index. To do so we need to update core to store this information in the IndexMetadata when the shrink operation is performed. This theoretically should be doable though once we release the Shrink action.. in a roundabout kind of way. Essentially once ISM shrinks foo-index to foo-index-shrink there will be two jobs running. One for foo-index and one for foo-index-shrink. If we knew what the source index of the shrink was on foo-index-shrink then foo-index-shrink could call the update alias API and delete the foo-index and add that as an alias to itself in a single API call I believe. Since that's not possible for now.. then going the other way the foo-index after it successfully executes the Shrink action should be able to do the same thing, it can call the update alias API and delete itself while also setting an alias on the foo-index-shrink equal to it's own index name.
    1.2
    PUT foo-index
    PUT foo-index-shrink
    POST /_aliases
    {
      "actions" : [
        { "remove_index": { "index": "foo-index" } },
        { "add" : { "index" : "foo-index-shrink", "alias" : "foo-index" } }
      ]
    }
    
  2. My use case is when an index foo is shrunk, it becomes shrink-foo. An alias action would allow the new shrunk index to also be aliased to foo so applications don't know the difference. When we moved from x-pack and ILM to ISM, the aliasing had to be handled with custom code outside ES. I'd be just as happy with the alias being an optional setting on the shrink operation. Happy to share details, but what we're doing is pretty basic.
    2.1 I'm assuming this is similar to (1) because I don't believe you can add an alias that is a concrete index. So you would need to delete the foo index in this example. So if it's the same as (1) then yes, there would be a way once we support Shrink action.
  3. my use case is around the shrinking of indices as well. But in general I think it would be great to support adding an alias, especially if we get to the stage where we can roll up indices into single indices then alias them with the ones that were rolled up.
    3.1 Should be similar to previous cases where the job that performs the rollup would have to then go and update the target index to have aliases of the source index patterns. The downside being I don't believe this specific policy would be that reusable.. and it wouldn't be possible to go the other way, i.e. a target rollup index comes into being and starts being managed and then figures out which indices it was rolled up from (we might actually store that information in the meta mappings of the rollup target index, so it could be doable in the future). Or if the source index has a wildcard pattern like log* then when updating the alias for the target index.. we can't use that and would have to resolve to concrete indices which could be old or become stale.
  4. My use case is, If we have an alias for an index with rollover ILM policy, can we remove it from alias after some predefined days, let's say 30 days? I know there is a way to delete an index once its old, but in our case we do not want to delete but want the index to be removed from alias and have the alias switch and point to indices whose age is less than 30 days.
    4.1. This should be easily doable with the Alias action, just have a state that gets transitioned into after the index is 30 days old (or 30 days rollover age) and then remove the alias.
  5. My use case is: we provide search_alias to search our dataset. We publish new version of dataset every week as a new index (e.g. dataset-yyyy-MM-dd. We want the search_alias always pointing to the latest published dataset. The process is: ingest new data to a new index, once ingestion finish, re-pointing search_alias to the new index, remove old index from the search_alias
    5.1 Should be possible, the newly created index can remove the alias from all search-index-* indices and then add the alias it itself in a single call. The main thing would be knowing when ingestion finished for the new index.. which you could just use some absolute time using the transition condition if you know it only takes x amount of time.

CheckList:

  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@dbbaughe dbbaughe requested a review from a team March 11, 2022 18:44
Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com>
@codecov-commenter
Copy link

codecov-commenter commented Mar 11, 2022

Codecov Report

Merging #304 (ab81231) into main (4985a18) will decrease coverage by 0.08%.
The diff coverage is 67.53%.

Impacted file tree graph

@@             Coverage Diff              @@
##               main     #304      +/-   ##
============================================
- Coverage     75.75%   75.66%   -0.09%     
- Complexity     2008     2023      +15     
============================================
  Files           253      256       +3     
  Lines         11466    11543      +77     
  Branches       1815     1823       +8     
============================================
+ Hits           8686     8734      +48     
- Misses         1753     1772      +19     
- Partials       1027     1037      +10     
Impacted Files Coverage Δ
...ndexstatemanagement/step/alias/AttemptAliasStep.kt 64.44% <64.44%> (ø)
...t/indexstatemanagement/action/AliasActionParser.kt 64.70% <64.70%> (ø)
...agement/indexstatemanagement/action/AliasAction.kt 78.57% <78.57%> (ø)
...anagement/indexstatemanagement/ISMActionsParser.kt 80.32% <100.00%> (+0.32%) ⬆️
...arch/indexmanagement/rollup/RollupSearchService.kt 55.55% <0.00%> (-5.56%) ⬇️
.../action/explain/TransportExplainTransformAction.kt 67.04% <0.00%> (-2.28%) ⬇️
...xmanagement/rollup/actionfilter/FieldCapsFilter.kt 72.94% <0.00%> (-1.18%) ⬇️
...earch/indexmanagement/transform/TransformRunner.kt 85.71% <0.00%> (-0.72%) ⬇️
.../opensearch/indexmanagement/rollup/model/Rollup.kt 84.43% <0.00%> (-0.48%) ⬇️
...nt/indexstatemanagement/ManagedIndexCoordinator.kt 72.18% <0.00%> (-0.32%) ⬇️
... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4985a18...ab81231. Read the comment docs.

@goyamegh goyamegh mentioned this pull request Oct 20, 2022
1 task
@goyamegh goyamegh mentioned this pull request Nov 1, 2022
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants