Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Opening indices with broken index settings #26998

Open
jasontedor opened this issue Oct 12, 2017 · 5 comments
Open

Opening indices with broken index settings #26998

jasontedor opened this issue Oct 12, 2017 · 5 comments
Assignees
@thecoop
Labels
:Core/Infra/Core Core issues without another label >enhancement help wanted adoptme Team:Core/Infra Meta label for core/infra team

Comments

@jasontedor
Copy link
Member

Today we are lenient with index settings, we do the following things:

  1. We will open an index even if it contains an index setting that we do not recognize because we do not do any index settings validation when opening an index.
  2. We will upgrade an index and archive unrecognized settings; this is bad in the following scenario:
    a. a user installs a plugin that provides an index setting
    b. a user creates an index that uses that index setting
    c. a user stops Elasticsearch to upgrade, so removes the plugin
    d. the user upgrades Elasticsearch, and starts Elasticsearch not having reinstalled the plugin
    e. Elasticsearch will archive the index setting and happily open the index

This leniency is dangerous and we should stop.

To be fair, the reason for the existence of this leniency in the upgrade case is for when we first introduced strict settings, we knew that we would see unrecognized settings on indices coming from 2.x into 5.x. We are past that point now. It's time to reconsider this leniency.

Relates #26995
@jasontedor jasontedor added :Core/Infra/Core Core issues without another label discuss labels Oct 12, 2017
@jasontedor
Copy link
Member Author

I opened #26955 to address the first case. I would like to have a broader discussion about this.

@jasontedor
Copy link
Member Author

We are going to proceed with the following plan:

  • remove the leniency on opening indices with unrecognized settings and instead such an index will be closed with the unrecognized settings archived
  • we will not open any index with archived settings
  • users can remove archived settings via the wildcard archived.*

@mayya-sharipova mayya-sharipova self-assigned this Jan 31, 2018
@mayya-sharipova
Copy link
Contributor

@jasontedor Jason, I was studying your corresponding PR: https://github.com/elastic/elasticsearch/pull/26995/files

and I don't understand that why when we create an IndexService in IndicesService.createIndexService, we don't validate archive settings as well. Current code:
https://github.com/elastic/elasticsearch/pull/26995/files#diff-97e5abcaadbacda4084c902214cd3654R437

indexScopedSettings.validate(indexMetaData.getSettings(), true, true, true); //ignoreArchiveSetting: true

Setting ignoreArchiveSetting to false, will validate archive settings as well, and will ensure that we never open an index with archived settings. No?

Thanks in advance for clarification.

@mayya-sharipova mayya-sharipova mentioned this issue Feb 8, 2018
Closed
mayya-sharipova added a commit to mayya-sharipova/elasticsearch that referenced this issue Feb 8, 2018
@mayya-sharipova
Do not open indices with broken index settings
4e0722f 
- remove the leniency on opening indices with unrecognized settings and
instead such an index will be closed with the unrecognized settings archived
- do not open any index with archived settings
- users can remove archived settings via the wildcard archived
- return "index_open_exception" 400 status code when trying to open an
index with broken settings

Relates to elastic#26995

Closes elastic#26998
mayya-sharipova added a commit to mayya-sharipova/elasticsearch that referenced this issue Mar 15, 2018
@mayya-sharipova
Do not open indices with broken index settings
e6d9325 
- remove the leniency on opening indices with unrecognized settings and
instead such an index will be closed
with the unrecognized settings archived
- do not open any index with archived settings
- users can remove archived settings via the wildcard archived
- return "index_open_exception" 400 status code when trying to open an
index with broken settings

Relates to elastic#26995

Closes elastic#26998
mayya-sharipova added a commit to mayya-sharipova/elasticsearch that referenced this issue Mar 16, 2018
@mayya-sharipova
Do not open indices with broken index settings
209afa4 
- remove the leniency on opening indices with unrecognized settings and
instead such an index will be closed
with the unrecognized settings archived
- do not open any index with archived settings
- users can remove archived settings via the wildcard archived
- return illegal_argument_exception  400 status code when trying
to open an index with broken/archived settings

Relates to elastic#26995

Closes elastic#26998
mayya-sharipova added a commit to mayya-sharipova/elasticsearch that referenced this issue Mar 16, 2018
@mayya-sharipova
Do not open indices with broken index settings
08daa7c 
- remove the leniency on opening indices with unrecognized settings and
instead such an index will be closed
with the unrecognized settings archived
- do not open any index with archived settings
- users can remove archived settings via the wildcard archived
- return illegal_argument_exception 400 status code when trying
to open an index with broken/archived settings

Relates to elastic#26995

Closes elastic#26998
@jasontedor jasontedor mentioned this issue Apr 15, 2019
Merged
@jasontedor jasontedor mentioned this issue Feb 22, 2020
Merged
@rjernst rjernst added the Team:Core/Infra Meta label for core/infra team label May 4, 2020
@rjernst rjernst added the needs:triage Requires assignment of a team area label label Dec 3, 2020
@jtibshirani jtibshirani mentioned this issue Dec 10, 2020
Closed
@jaymode jaymode added help wanted adoptme and removed needs:triage Requires assignment of a team area label labels Dec 14, 2020
@jtibshirani jtibshirani mentioned this issue Jan 16, 2021
Open
@thecoop thecoop assigned thecoop and unassigned mayya-sharipova Nov 1, 2024
@thecoop
Copy link
Member

thecoop commented Nov 1, 2024

Is this still the case?

@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-core-infra (Team:Core/Infra)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@thecoop thecoop

Labels
:Core/Infra/Core Core issues without another label >enhancement help wanted adoptme Team:Core/Infra Meta label for core/infra team
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Do not open indices with broken index settings mayya-sharipova/elasticsearch
7 participants
@colings86 @rjernst @thecoop @jaymode @jasontedor @mayya-sharipova @elasticsearchmachine