Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Allocation Index Priority #11787

Closed
pickypg opened this issue Jun 20, 2015 · 3 comments
Closed

Add Allocation Index Priority #11787

pickypg opened this issue Jun 20, 2015 · 3 comments
Assignees
Labels
:Distributed Indexing/Recovery Anything around constructing a new shard, either from a local or a remote source. >feature release highlight v1.7.0 v2.0.0-beta1

Comments

@pickypg
Copy link
Member

pickypg commented Jun 20, 2015

The idea is allow a simple, dynamically modifiable priority for indices. Something akin to:

$ curl -XPUT host:9200/my-index/_settings -d '{
  "index.priority" : 1
}'

By default, all priorities would default with index.priority : 0, thus being of equal priority unless modified. From there, the comparison could go to index creation date/time as the next priority level, then finally by name for the final order.

index.priority > index.creation_date > reverse sorted index name

This allows pretty much every use case to specify recovery order for best-case recovery order for their use case. Time-based solutions would pretty much work out of the box because the creation date would prioritize recovery of today's index first, thus giving the best chance to return to fully replicated indexing as soon as possible. Similarly, falling back to the reverse sorted index name gives arbitrary--yet predictable--sorting for simultaneously-created indices (however unlikely) that have no time basis (index-z, index-y, ...), but YMD-formatted date/time strings will properly sort (logstash-2015.05.03 will always come before logstash-2015.05.02 or earlier dates). Finally, the arbitrary priority allows users to take control of the process for cases where there is no time basis, such as when a "main" index is created, but then minor indices are created from time-to-time. Even better, this would be as simple as setting it once with a high value, then not setting or selectively setting the minor indices.

@pickypg pickypg added discuss :Core/Infra/Settings Settings infrastructure and APIs :Allocation :Distributed Indexing/Recovery Anything around constructing a new shard, either from a local or a remote source. :Data Management/Indices APIs APIs to create and manage indices and templates labels Jun 20, 2015
@pickypg
Copy link
Member Author

pickypg commented Jun 20, 2015

This is a duplicate of #10069, but I had been thinking about it a bit over the course of the past few weeks and I wanted to write it down. We can close this one in favor of that one (or vice versa), but I wanted to get it written down.

@clintongormley clintongormley added >feature and removed :Allocation :Data Management/Indices APIs APIs to create and manage indices and templates :Core/Infra/Settings Settings infrastructure and APIs labels Jun 23, 2015
@avleen
Copy link

avleen commented Jun 23, 2015

I like the defaults proposed here :)

@s1monw s1monw self-assigned this Jul 1, 2015
s1monw added a commit to s1monw/elasticsearch that referenced this issue Jul 6, 2015
This commit adds logic to prefer shards with higher priority
or from newer indicse to be allocated first if they are unallocated post API.

This commit allows users to set `index.priority` to a non-negative integer to
prioritize index recovery for certain indices. This setting is dynamically updateable
and defaults to `0`. If two indices have the same priority this change takes the creation
date into account to prioritize shards from newer indices which is important in the time-based
indices usecase.

Closes elastic#11787
s1monw added a commit that referenced this issue Jul 7, 2015
This commit adds logic to prefer shards with higher priority
or from newer indicse to be allocated first if they are unallocated post API.

This commit allows users to set `index.priority` to a non-negative integer to
prioritize index recovery for certain indices. This setting is dynamically updateable
and defaults to `0`. If two indices have the same priority this change takes the creation
date into account to prioritize shards from newer indices which is important in the time-based
indices usecase.

Closes #11787
@s1monw s1monw added v1.7.0 and removed help wanted adoptme labels Jul 7, 2015
@s1monw
Copy link
Contributor

s1monw commented Jul 7, 2015

FYI pushed the #11975 to 1.x and master

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Distributed Indexing/Recovery Anything around constructing a new shard, either from a local or a remote source. >feature release highlight v1.7.0 v2.0.0-beta1
Projects
None yet
Development

No branches or pull requests

4 participants