-
Notifications
You must be signed in to change notification settings - Fork 74
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
K8SSAND-995 ⁃ Make the management-api heap size configurable #212
Comments
➤ Jeff DiNoto commented: Notes
|
cass-operator This was easier to solve than I expected. Preliminary thoughts on the cass-operator side of things:
Is there any reason why this is an inappropriate solution?
|
Configuring via env vars is what I had in mind :) |
K8ssandra operator If we can configure the env var via the podTemplateSpec in cass-operator, we need to figure out how to pass that configuration down to the CassandraDatacenter resource from k8ssandra-operator. I can see here that we have a I'd do the following to clarify the schema and allow for better composition of the objects:
I appreciate that the above proposal risks hitting known issues with kubebuilder. I am hoping that by turning off descriptions for nested objects we can avoid these, or that we can find another workaround if that doesn't work (perhaps SSA can play a role). If all else fails, I suggest that we either
Option 2 is a more permanent solution, if we anticipate that issues around max CRD size will be persistent once SSA changes have settled. |
I went ahead and updated the description. After rereading it I realized it was lacking some detail. As we discussed in k8ssandra/k8ssandra#1143 I am not in favor of exposing the PodTemplateSpec. We made the conscious decision to not expose it in k8ssandra 1.x, and we are following the same approach for k8ssandra-operator. With this in mind, the
Take a look at how the heap is configured for Stargate. We should be able to do something very similar for the management-api. I think it would be ideal to first expose the property in cass-operator as it might be generally beneficial to users. If we go ahead and implement it through the PodTemplateSpec property, it will be much, much less obvious to users. |
I very much agree that all of these things are evil and should not be in the I am concerned that creating a new struct to configure environment variables in pods from within the I believe this will lead to poor maintainability and a poor user experience.
This appears to contradict your earlier comment? (NB: it should go without saying that k8ssandra 1.x was a Helm chart and k8ssandra 2.x is an operator, and that any decisions which were taken in the former should have no bearing upon design approaches in the latter.) |
What is missing?
The heap size for the management-api had been hard coded to either 128M or 256M. Changes were made to make it configurable via an env var. The default was reduced to 16M which helped with the resource challenges in e2e tests. The default setting caused some OutOfMemoryErrors in non-test environments. The default has been reverted back to 128M.
I was doing some manual testing and having problems with my Cassandra pods becoming unstable. Then I remembered about the heap size change. After increasing the memory limits things became stable again in my cluster.
Why do we need it?
We need to make the management-api's heap size configurable so that we can reduce the memory requirements of the
cassandra
container in dev/test enviromments. This may be generally useful as well in non-test environments as folks look to tune things.Goals
The expectation is that properties are added to the K8ssandraCluster spec for configuring the heap size. The fact that the heap size is actually configured via env vars on the
cassandra
container should be considered an implementation detail with which the user need not be concerned.To be consistent with the approach taken thus far it probably makes sense to make the heap size configurable both at the cluster and dc levels. This means properties should be added in both
CassandraClusterTemplate
and inCassandraDatacenterTemplate
with the latter taking precedence over the former.We can implement this without any changes in cass-operator by configuring the heap size via the PodTemplateSpec property. My preference however would be to first implement the above changes in cass-operator and then have k8ssandra-operator simply set the new property in the CassandraDatacenterSpec.
v1.0.0-alpha.1
┆Issue is synchronized with this Jira Task by Unito
┆epic: Cassandra Configuration
┆fixVersions: k8ssandra-operator-v1.0.0
┆friendlyId: K8SSAND-995
┆priority: Medium
The text was updated successfully, but these errors were encountered: