Unfinished elasticsearch cluster upgrade

[cehoffman]

Is this a BUG REPORT or FEATURE REQUEST?:

/kind bug

What happened:

Cluster was upgrade from 6.2.3 to 6.2.4. Master node pool was upgraded then data pool started. 3 of the 5 data pods were upgraded and then upgrade stopped.

What you expected to happen:

All pods were upgraded.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

There were no errors in the controller manager logs and no events on the elasticsearch cluster resource. It just stopped executing the UpdateVersion action. This is with versions v0.1.0-alpha.1 of all components. It is stuck at partition 2 in the statefulset.

Cluster configuration:

apiVersion: navigator.jetstack.io/v1alpha1
kind: ElasticsearchCluster
metadata:
  creationTimestamp: 2018-04-04T09:15:36Z
  name: logging
  namespace: logging
  resourceVersion: "1279998"
  selfLink: /apis/navigator.jetstack.io/v1alpha1/namespaces/logging/elasticsearchclusters/logging
  uid: bc1c02eb-37e8-11e8-bd07-56967cf91fd8
spec:
  image:
    pullPolicy: ""
    repository: cr.vertiv.life/library/elasticsearch
    tag: 6.2.4
  nodePools:
  - name: master
    nodeSelector: null
    persistence:
      enabled: true
      size: 1Gi
      storageClass: ext4-2x-high
    replicas: 3
    resources:
      limits:
        cpu: "1"
        memory: 3Gi
      requests:
        cpu: 50m
        memory: 3Gi
    roles:
    - master
    schedulerName: stork
  - name: d
    nodeSelector: null
    persistence:
      enabled: true
      size: 250Gi
      storageClass: ext4-1x-high
    replicas: 5
    resources:
      limits:
        cpu: "5"
        memory: 6Gi
      requests:
        cpu: 300m
        memory: 6Gi
    roles:
    - data
    - ingest
    schedulerName: stork
  pilotImage:
    pullPolicy: Always
    repository: quay.io/jetstack/navigator-pilot-elasticsearch
    tag: v0.1.0-alpha.1
  plugins: null
  securityContext:
    runAsUser: 1000
  version: 6.2.4
status:
  health: Green
  nodePools: null

Environment:

• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration**:
• Install tools:
• Others: