-
Notifications
You must be signed in to change notification settings - Fork 2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improve auto-sharding documentation #1559
Improve auto-sharding documentation #1559
Conversation
@@ -43,7 +43,7 @@ are deleted they are no longer visible on the `/metrics` endpoint. | |||
- [kube-state-metrics vs. metrics-server](#kube-state-metrics-vs-metrics-server) | |||
- [Scaling kube-state-metrics](#scaling-kube-state-metrics) | |||
- [Resource recommendation](#resource-recommendation) | |||
- [Horizontal scaling (sharding)](#horizontal-scaling-sharding) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think horizontal scaling can be a bit confusing since it might indicate that all KSM replicas are the same.
README.md
Outdated
|
||
There is also an experimental feature, that allows kube-state-metrics to auto discover its nominal position if it is deployed in a StatefulSet, in order to automatically configure sharding. This is an experimental feature and may be broken or removed without notice. | ||
KSM supports a feature which allows each shard to discover its nominal position when deployed in a StatefulSet. which is useful for automatically configuring sharding. This is an experimental feature and may be broken or removed without notice. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@brancz do you have any background information on why this feature was marked as experimental? Is it simply because it never got promoted to stable?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@kubernetes/kube-state-metrics-maintainers should we start marking this as non-experimental with the next release to get more users test it? Or do we have plans to remove it at some point?
/assign @mrueg |
Thanks for working on this @fpetkovski! I added some comments @SuperQ as you asked about more information, is this doc change helpful for you? |
README.md
Outdated
|
||
There is also an experimental feature, that allows kube-state-metrics to auto discover its nominal position if it is deployed in a StatefulSet, in order to automatically configure sharding. This is an experimental feature and may be broken or removed without notice. | ||
KSM supports a feature which allows each shard to discover its nominal position when deployed in a StatefulSet. which is useful for automatically configuring sharding. This is an experimental feature and may be broken or removed without notice. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we want to introduce KSM as an abbreviation for the project or stick with "kube-state-metrics"?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good point, I used it as second nature 😄. I added the abbreviation next to the first reference of kube-state-metrics
at the beginning of the readme. I think that it is used commonly used in the community so IMO it makes sense to have it in the docs. Wdyt?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I you change the sentence structure a bit, you could write it like this.
Automatic sharding allows each shard each shard to discover its nominal position
This avoids needing to be overly self-referential about the fact that this is a kube-state-metrics feature.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the suggestion, applied
README.md
Outdated
|
||
* `--shard` (zero indexed) | ||
* `--total-shards` | ||
|
||
Sharding is done by taking an md5 sum of the Kubernetes Object's UID and performing a modulo operation on it, with the total number of shards. The configured shard decides whether the object is handled by the respective instance of kube-state-metrics or not. Note that this means all instances of kube-state-metrics even if sharded will have the network traffic and the resource consumption for unmarshaling objects for all objects, not just the ones it is responsible for. To optimize this further, the Kubernetes API would need to support sharded list/watch capabilities. Overall memory consumption should be 1/n th of each shard compared to an unsharded setup. Typically, kube-state-metrics needs to be memory and latency optimized in order for it to return its metrics rather quickly to Prometheus. | ||
Sharding is done by taking an md5 sum of the Kubernetes Object's UID and performing a modulo operation on it with the total number of shards. Each shard decides whether the object is handled by the respective instance of kube-state-metrics or not. Note that this means all instances of kube-state-metrics, even if sharded, will have the network traffic and the resource consumption for unmarshaling objects for all objects, not just the ones they are responsible for. To optimize this further, the Kubernetes API would need to support sharded list/watch capabilities. In the optimal case, memory consumption for each shard will be 1/n compared to an unsharded setup. Typically, kube-state-metrics needs to be memory and latency optimized in order for it to return its metrics rather quickly to Prometheus. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we mention --use-apiserver-cache
as a way to reduce load on etcd here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good idea, added the suggestion
c7184e0
to
99dcf27
Compare
Signed-off-by: fpetkovski <filip.petkovsky@gmail.com>
99dcf27
to
8eecd40
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
/approve |
/lgtm Thanks for improving the doc @fpetkovski ! |
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: fpetkovski, mrueg, SuperQ The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
What this PR does / why we need it:
This PR improves the documentation on auto-sharding. It explains the pros and cons of using auto-sharding vs manual sharding.
How does this change affect the cardinality of KSM: (increases, decreases or does not change cardinality)
Does not affect it
Which issue(s) this PR fixes:
Fixes #1546