Way to change index.store.type for ElasticSearch chart

[g00ntar]

Is there way to change index.store.type like on https://www.elastic.co/guide/en/cloud-on-k8s/master/k8s-virtual-memory.html in ElasticSearch chart ?

[carrodher]

It is possible to use the config parameter from the values.yaml in order to provide a custom ES configuration, see

## Customize elasticsearch configuration
## ref: https://www.elastic.co/guide/en/elasticsearch/reference/current/settings.html
##
# config:

which it's going to create a ConfigMap with the data provided in this section and mounted to /opt/bitnami/elasticsearch/config/elasticsearch.yml. You should be able to set any parameter supported by ES in this config file

[g00ntar]

I looking to change this parameter because this allow to run ElasticSearch pod as unprivileged and allow to install this helm on GKE Autopilot. After your answer I created myvalues.yml with content

sysctlImage:
  enabled: true
config:
  index.store.type: niofs
  node.store.allow_mmap: false

and after running helm install -f myvars.yml test-es-1 bitnami/elasticsearch container dies in pod with error complaining about Invalid kernel settings. Elasticsearch requires at least: vm.max_map_count = 262144 which should not be a case.

Helm chart provided by ElasticSearch itself with values

sysctlInitContainer:
  enabled: false
esConfig:
  elasticsearch.yml: |
    index.store.type: niofs
    node.store.allow_mmap: false

works normally - it skips checking value of vm.max_map_count

[carrodher]

Currently, Elasticsearch requires some changes in the kernel of the host machine to work as expected. If those values are not set in the underlying operating system, the ES containers fail to boot with ERROR messages. Here you can find more info about the specific case of Virtual Memory.

The Bitnami Helm Chart try to solve this issue by adding an initContainer to change those settings in the Kernel by running: sysctl -w vm.max_map_count=262144 && sysctl -w fs.file-max=65536. This initContainer should be executed as a privileged container since it is needed to change the kernel settings. This container is enabled by default, but it can be disabled by using sysctlImage.enabled=false, for example, if the kernel already has the proper settings.

[g00ntar]

Yes, you are right and I'm aware of this need, but...
After https://www.elastic.co/guide/en/cloud-on-k8s/master/k8s-virtual-memory.html

Note that this requires the ability to run privileged containers, which is likely not the case on many secure clusters.

and there (on the same website) are solutions/workaround to make this works without need to change kernel settings.
To run elesticsearch without need max_map_count=262144 setting (and at the same time drop depency of privileged initialContainer ) config of elesticsearch must contain index.store.type: niofs or node.store.allow_mmap: false (and thats why I provided working config for elasticsearch helm chart).
I understand that maybe bitnami are not interested to make changes for this sepecyfic use case (GKE) but still changes in config are not respected by the chart.
I'm not native speaker and maybe I'm not quite clear, so to help understand the case I found similar issue on elasticsearch github #32267

[carrodher]

The privileged initContainer is enabled by default, in order to disable it, you need to set sysctlImage.enabled=false when installing the chart. Then, you should be able to set the desired parameters in the configuration file, something like

sysctlImage:
  enabled: false
config:
  index:
    store:
      type: niofs
  node:
    store:
      allow_mmap: false

In that case, are you able to see the proper configuration in the config file (elasticsearch.yml)? You can check it by accessing the pod (if it is up and running) or taking a look at the ConfigMap with kubectl.

What is the exact error you are seen in the logs when deploying the chart with the above configuration?

[g00ntar]

I'm little ambarased because I made mistake in config in my second post, of course there should be sysctlImage.enabled=false and I tried this from begining.

So below steps to reproduce on GKE with some output - helm chart elastic/elasticsearch with same config works on GKE

cat <<EOF > myvalues.yml
sysctlImage:
  enabled: false
config:
  index.store.type: niofs
  node.store.allow_mmap: false
EOF

helm install -f myvalues.yml bitnami-es bitnami/elasticsearch

kubectl get pods

NAME                                                          READY   STATUS              RESTARTS   AGE
bitnami-es-elasticsearch-coordinating-only-85cd75d5b4-ftptg   0/1     CrashLoopBackOff    3          105s
bitnami-es-elasticsearch-coordinating-only-85cd75d5b4-lt2bj   0/1     ContainerCreating   0          105s
bitnami-es-elasticsearch-data-0                               0/1     Pending             0          105s
bitnami-es-elasticsearch-data-1                               0/1     ContainerCreating   0          104s
bitnami-es-elasticsearch-master-0                             0/1     ContainerCreating   0          105s
bitnami-es-elasticsearch-master-1                             0/1     ContainerCreating   0          104s

kubectl logs bitnami-es-elasticsearch-coordinating-only-85cd75d5b4-ftptg

elasticsearch 13:58:00.84 
elasticsearch 13:58:00.90 Welcome to the Bitnami elasticsearch container
elasticsearch 13:58:00.90 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-elasticsearch
elasticsearch 13:58:00.90 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-elasticsearch/issues
elasticsearch 13:58:00.91 
elasticsearch 13:58:00.91 INFO  ==> ** Starting Elasticsearch setup **
elasticsearch 13:58:01.02 ERROR ==> Invalid kernel settings. Elasticsearch requires at least: vm.max_map_count = 262144

[carrodher]

Thanks for the detailed explanation, I was taking a look and found where the issue comes from.

During the bitnami/elasticsearch container image initialization, there are different scripts executed, one of those scripts is setup.sh, in this script there is the following function (link):

# Ensure kernel settings are valid
elasticsearch_validate_kernel

This function is defined at libelasticsearch.sh (link):

########################
# Validate kernel settings
# Arguments:
#   None
# Returns:
#   None
#########################
elasticsearch_validate_kernel() {
    # Auxiliary functions
    validate_sysctl_key() {
        local key="${1:?key is missing}"
        local value="${2:?value is missing}"
        local current_value
        current_value="$(sysctl -n "$key")"
        if [[ "$current_value" -lt "$value" ]]; then
            error "Invalid kernel settings. Elasticsearch requires at least: $key = $value"
            exit 1
        fi
    }

    debug "Validating Kernel settings..."
    validate_sysctl_key "vm.max_map_count" 262144
    validate_sysctl_key "fs.file-max" 65536
}

This validation was implemented in the past because we were having several support cases related to that and probably modifying the storage type was not an option. I just created an internal task to investigate if we can improve the current container logic to support this use case.

In the meantime, as a workaround, you can build your own image and use it in the Helm chart:

• Build the image removing the kernel validation from the setup.sh

$ git clone https://github.com/bitnami/bitnami-docker-elasticsearch.git
$ cd bitnami-docker-elasticsearch/7/debian-10
$ vim rootfs/opt/bitnami/scripts/elasticsearch/setup.sh # Removing the elasticsearch_validate_kernel line
$ docker build -t myuser/custom-es:latest .
$ docker push myuser/custom-es:latest

• Use the custom image in the Helm Chart:

image:
  registry: docker.io
  repository: myuser/custom-es
  tag: latest
sysctlImage:
  enabled: false
config:
  index:
    store:
      type: niofs
  node:
    store:
      allow_mmap: false

Following the previous approach, you should be able to deploy ES in an unprivileged cluster by disabling the initcontainer and using a custom config file, thanks to the removal of the validation in your custom image.

[carrodher]

I'm setting the "on-hold" label to prevent the stale bot from closing the issue while we work on the internal task to investigate what is the real purpose of the validation in the image and if there is a chance of getting rid of it

[carrodher]

@g00ntar we are working on some changes in the image to run the kernel settings validation only if node.store.allow_mmap and/or index.store.type are not set in the config file. Apart from that, I created the above PR adding more info to the NOTES.txt.

Were you able to solve the issue with the proposed workaround? Basically, the solution we are going to provide is similar. In the proposed workaround the kernel validation was directly removed while in the long-term solution we are going to check the config file to determine if the validation should be executed or not, but the result should be the same

[carrodher]

This issue was automatically closed because the changes in the image were already implemented, see tag 7.10.2-debian-10-r56 (https://github.com/bitnami/bitnami-docker-elasticsearch/commit/348956597b96af42be41f8f1d1e07ddb3b4dd648) and the Helm Chart was updated to use this image among other changes related to the docs.

Now using custom config file containing node.store.allow_mmap and/or index.store.type in one of the following ways:

• Standard

sysctlImage:
  enabled: false
config:
  index:
    store:
      type: niofs
  node:
    store:
      allow_mmap: false

• Flatten

sysctlImage:
  enabled: false
config:
  index.store.type: niofs
  node.store.allow_mmap: false

the kernel setting validations shouldn't be executed during the container initialization. That means the deployment is not going to fail due to this reason, but then ES should handle properly the configured settings to deploy the Helm Chart in an unprivileged cluster, but this is something not checked/handled by the container image nor the Helm Chart.