fix(helm): Add extra args to zfsController containers and leader election inteligence

[trunet]

Added zfsController extraArgs variable to pass extra argumenta to containers command line.

Also adding inteligence to enable leader election only when number of replicas is more than one.

Fixes #486

Pull Request template

Please, go through these steps before you submit a PR.

Why is this PR required? What issue does it fix?: It'll fix a high number of restarts on slow clusters by disabling leader election when replicas < 2

What this PR does?: It'll add option to add extra arguments to zfsController containers and enable leader election only when replicas > 1

Does this PR require any upgrade changes?: No

If the changes in this PR are manually verified, list down the scenarios covered::

• helm install with default values
• helm install --set zfsController.replicas=2
• helm install --set zfsController.replicas=2 --set zfsController.resizer.extraArgs={--leader-election-lease-duration 60}

Any additional information for your reviewer? :
Mention if this PR is part of any design or a continuation of previous PRs

Checklist:

• Fixes ZFS Controller leader election arguments in helm charts #486
• PR Title follows the convention of <type>(<scope>): <subject>
• Has the change log section been updated?
• Commit has unit tests
• Commit has integration tests
• (Optional) Are upgrade changes included in this PR? If not, mention the issue/PR to track:
• (Optional) If documentation changes are required, which issue on https://github.com/openebs/openebs-docs is used to track them:

[sinhaashish]

Thanks for contributing @trunet. Can you please share the test result with the newly added command. @pawanpraka1 It would be great if you too can review it.

[trunet]

@sinhaashish what do you expect from "test results"? What I did was helm template with the variables I sent and checking the output. If I share, will be a very long comment because each run will be a lot of resources.

[trunet]

These are the tests I did. Hope it's enough.

# helm template deploy/helm/charts
...
      containers:
        - name: csi-resizer
          image: "registry.k8s.io/sig-storage/csi-resizer:v1.8.0"
          args:
            - "--v=5"
            - "--csi-address=$(ADDRESS)"
...
        - name: csi-snapshotter
          image: "registry.k8s.io/sig-storage/csi-snapshotter:v6.2.2"
          imagePullPolicy: IfNotPresent
          args:
            - "--csi-address=$(ADDRESS)"
...
        - name: snapshot-controller
          image: "registry.k8s.io/sig-storage/snapshot-controller:v6.2.2"
          args:
            - "--v=5"
...
        - name: csi-provisioner
          image: "registry.k8s.io/sig-storage/csi-provisioner:v3.5.0"
          imagePullPolicy: IfNotPresent
          args:
            - "--csi-address=$(ADDRESS)"
            - "--v=5"
            - "--feature-gates=Topology=true"
            - "--strict-topology"
            - "--enable-capacity=true"
            - "--extra-create-metadata=true"
            - "--default-fstype=ext4"
...

# helm template deploy/helm/charts --set zfsController.replicas=2
...
      containers:
        - name: csi-resizer
          image: "registry.k8s.io/sig-storage/csi-resizer:v1.8.0"
          args:
            - "--v=5"
            - "--csi-address=$(ADDRESS)"
            - "--leader-election"
...
        - name: csi-snapshotter
          image: "registry.k8s.io/sig-storage/csi-snapshotter:v6.2.2"
          imagePullPolicy: IfNotPresent
          args:
            - "--csi-address=$(ADDRESS)"
            - "--leader-election"
...
        - name: snapshot-controller
          image: "registry.k8s.io/sig-storage/snapshot-controller:v6.2.2"
          args:
            - "--v=5"
            - "--leader-election"
          imagePullPolicy: IfNotPresent
        - name: csi-provisioner
          image: "registry.k8s.io/sig-storage/csi-provisioner:v3.5.0"
          imagePullPolicy: IfNotPresent
          args:
            - "--csi-address=$(ADDRESS)"
            - "--v=5"
            - "--feature-gates=Topology=true"
            - "--strict-topology"
            - "--enable-capacity=true"
            - "--extra-create-metadata=true"
            - "--default-fstype=ext4"
            - "--leader-election"
...

# helm template deploy/helm/charts --set zfsController.replicas=2 --set zfsController.resizer.extraArgs={"--leader-election-lease-duration 60"}
         containers:
        - name: csi-resizer
          image: "registry.k8s.io/sig-storage/csi-resizer:v1.8.0"
          args:
            - "--v=5"
            - "--csi-address=$(ADDRESS)"
            - "--leader-election"
            - "--leader-election-lease-duration 60"
...
        - name: csi-snapshotter
          image: "registry.k8s.io/sig-storage/csi-snapshotter:v6.2.2"
          imagePullPolicy: IfNotPresent
          args:
            - "--csi-address=$(ADDRESS)"
            - "--leader-election"
...
        - name: snapshot-controller
          image: "registry.k8s.io/sig-storage/snapshot-controller:v6.2.2"
          args:
            - "--v=5"
            - "--leader-election"
          imagePullPolicy: IfNotPresent
        - name: csi-provisioner
          image: "registry.k8s.io/sig-storage/csi-provisioner:v3.5.0"
          imagePullPolicy: IfNotPresent
          args:
            - "--csi-address=$(ADDRESS)"
            - "--v=5"
            - "--feature-gates=Topology=true"
            - "--strict-topology"
            - "--enable-capacity=true"
            - "--extra-create-metadata=true"
            - "--default-fstype=ext4"
            - "--leader-election"
...

[trunet]

@sinhaashish and @pawanpraka1 Is there anything else I need to do to have this merged?

[trunet]

@sinhaashish @pawanpraka1 could you give a feedback on this?

[sinhaashish]

@trunet Can you please resolve the conflict so that it can be merged.

[trunet]

It's resolved, but if you take that long again, any new release will keep causing a new conflict.

[niladrih]

@trunet -- Changes look good. However, we require DCO signs on all of your commits. The git pull-based Merge commit doesn't have a signature. You'd want to rebase your branch instead.

git checkout leader-election-helm-options
git fetch upstream develop
git rebase upstream/develop
git push origin leader-election-helm-options

Let me know if you need help with that. I'm going to have to hold off on the merge until this is fixed.

[sinhaashish]

@trunet can you please do the needful here

[trunet]

@sinhaashish I was off-grid, on holidays, just fixed.