Add replicas configurability

[miguelsorianod]

This PR objective is to add replicas configurability on scalable DeploymentConfigs.
We define scalable DeploymentConfigs as the DeploymentConfigs that can have more than one replica deployed without causing data inconsistency / problems. DB Deployments are usually non-scalable DeploymentConfigs for example.

[miguelsorianod]

@eguzki still at the beginning but I have an approach to how to give them the user so I ask for your input and we can iterate over that.

Summarized, the way I'm organizing the CRD changes are:

• Create a new section for every DeploymentConfig that we decide to be able to configure replicas
• Not breaking schema compatibility (only additions, no reorganization or removal of fields)

The downsides of not breaking schema compatibility and this approach is that we will end up with fields that should belong inside the new section that will still be in a more general section. For example, the 'includeRespnseCodes' field of the apicast section should ideally be inside the ApicastProduction and ApicastStaging new sections related to DeploymentConfigs but we are not able to do that because with this approach I decided not breaking compatibility.

The other option would be put it all inside the more "general" sections but with that approach we would have to namespace the attribute names itself and we could end up with lots of fields in the same section and make it less maintainable too so I see good and bad parts with both approaches.

What do you think?

[miguelsorianod]

There's no DeploymentConfig name "zync-app" but I could not choose ZyncSpec because ZyncSpec already exists as a section that contains this field.

[eguzki]

Added some comments @miguelsorianod

I agree about adding those new fields in the CRD, I think it is the right direction to go.

Please, rebase the branch so the PR only shows changes related to the PR.

Hint: try not to include refactors in this PR, only include required changes for the replicas. Add your refactors in other PR.

[miguelsorianod]

I've renamed the fields removing redundant parts (for example, ApicastProductionSpec field inside the Apicast section has been renamed to ProductionSpec).

Default replicas when setting CR defaults is now always set to 1 instead of relying on the HA field. This means that now is up to the user to correctly specify more than one replica through the CR in the desired configurable DCs in case they want HA for those DCs at pod level.
Also updated Updated deployment_config_reconciler.go to always set the desired replicas instead of only when HA is used with a specific condition.

The component object builders themselves (in pkg/3scale/amp/component) have also default replicas to 1 (was already set that way) in case they are not set through the builder (for the templates scenario)

@eguzki can you review it again?

Thank you.

[miguelsorianod]

I've tried to perform an upgrade of the 3scale version including the changes here for the operator and the new CRD, through OLM, and it seems to work correctly.

The test performed has been:
1 - Install 3scale 2.6 from a custom application repository containing csv 0.3.0 (that contains the crds of 3scale 2.6)
2 - Upload a new package to the custom application repository, as version 0.4.0 containing a csv 0.4.0 that replaces the 0.3.0 csv, has 0.4.0 as "currentCSV", and has "master" images for the operands
3 - This triggered a new installplan for apimanager and I've observed that a new operator pod has been deployed with the new operator image, and the CRD has been updated in OpenShift with the new "replica" fields. I've also observed that the defaults (replicas set to 1) have been set for the configurable DCs in the APIManager resource after the upgrade

Can you review it @eguzki ?

[miguelsorianod]

Detail of outputs

Before upgrade (3scale 2.6 installed)

APIManager

msoriano@localhost:~/go/src/github.com/3scale/3scale-operator/deploy/olm-catalog/3scale-operator (add-replicas-configurability)$ oc get apimanager example-apimanager -o yaml
apiVersion: apps.3scale.net/v1alpha1
kind: APIManager
metadata:
  annotations:
    apps.3scale.net/threescale-operator-version: 0.3.0
  creationTimestamp: "2019-09-19T09:08:11Z"
  generation: 2
  name: example-apimanager
  namespace: msoriano-test
  resourceVersion: "723678"
  selfLink: /apis/apps.3scale.net/v1alpha1/namespaces/msoriano-test/apimanagers/example-apimanager
  uid: 015cfa41-dabd-11e9-8fc7-0efb785ddbd2
spec:
  apicast:
    managementAPI: status
    openSSLVerify: false
    registryURL: http://apicast-staging:8090/policies
    responseCodes: true
  appLabel: 3scale-api-management
  imageStreamTagImportInsecure: false
  resourceRequirementsEnabled: false
  system:
    database:
      mysql: {}
    fileStorage:
      amazonSimpleStorageService:
        awsBucket: enterprise-onprem.3sca.net
        awsCredentialsSecret:
          name: s3-bucket-credentials
        awsRegion: eu-west-1
  tenantName: 3scale
  wildcardDomain: test.apps.miguel.testing.3sca.net
status:
  deployments:
    ready:
    - apicast-production
    - apicast-staging
    - backend-cron
    - backend-listener
    - backend-redis
    - backend-worker
    - system-app
    - system-memcache
    - system-mysql
    - system-redis
    - system-sidekiq
    - system-sphinx
    - zync
    - zync-database
    - zync-que

APIManager CRD:

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  creationTimestamp: "2019-09-19T09:07:03Z"
  generation: 1
  name: apimanagers.apps.3scale.net
  resourceVersion: "720352"
  selfLink: /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/apimanagers.apps.3scale.net
  uid: d8bd2193-dabc-11e9-85f9-0a5f891f16a0
spec:
  conversion:
    strategy: None
  group: apps.3scale.net
  names:
    kind: APIManager
    listKind: APIManagerList
    plural: apimanagers
    singular: apimanager
  scope: Namespaced
  subresources:
    status: {}
  validation:
    openAPIV3Schema:
      properties:
        apiVersion:
          description: 'APIVersion defines the versioned schema of this representation
            of an object. Servers should convert recognized schemas to the latest
            internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources'
          type: string
        kind:
          description: 'Kind is a string value representing the REST resource this
            object represents. Servers may infer this from the endpoint the client
            submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds'
          type: string
        metadata:
          type: object
        spec:
          properties:
            apicast:
              properties:
                image:
                  type: string
                managementAPI:
                  type: string
                openSSLVerify:
                  type: boolean
                registryURL:
                  type: string
                responseCodes:
                  type: boolean
              type: object
            appLabel:
              type: string
            backend:
              properties:
                image:
                  type: string
                redisImage:
                  type: string
              type: object
            highAvailability:
              properties:
                enabled:
                  type: boolean
              type: object
            imageStreamTagImportInsecure:
              type: boolean
            resourceRequirementsEnabled:
              type: boolean
            system:
              properties:
                database:
                  properties:
                    mysql:
                      description: Union type. Only one of the fields can be set
                      properties:
                        image:
                          type: string
                      type: object
                    postgresql:
                      properties:
                        image:
                          type: string
                      type: object
                  type: object
                fileStorage:
                  properties:
                    amazonSimpleStorageService:
                      properties:
                        awsBucket:
                          type: string
                        awsCredentialsSecret:
                          properties:
                            name:
                              type: string
                          type: object
                        awsRegion:
                          type: string
                      required:
                      - awsBucket
                      - awsRegion
                      - awsCredentialsSecret
                      type: object
                    persistentVolumeClaim:
                      description: Union type. Only one of the fields can be set.
                      properties:
                        storageClassName:
                          type: string
                      type: object
                  type: object
                image:
                  type: string
                memcachedImage:
                  type: string
                redisImage:
                  type: string
              type: object
            tenantName:
              type: string
            wildcardDomain:
              type: string
            zync:
              properties:
                image:
                  type: string
                postgreSQLImage:
                  type: string
              type: object
          required:
          - wildcardDomain
          type: object
        status:
          properties:
            conditions:
              items:
                properties:
                  status:
                    type: string
                  type:
                    type: string
                required:
                - type
                - status
                type: object
              type: array
            deployments:
              properties:
                ready:
                  description: Deployments are ready to serve requests
                  items:
                    type: string
                  type: array
                starting:
                  description: Deployments are starting, may or may not succeed
                  items:
                    type: string
                  type: array
                stopped:
                  description: Deployments are not starting, unclear what next step
                    will be
                  items:
                    type: string
                  type: array
              type: object
          required:
          - deployments
          type: object
  version: v1alpha1
  versions:
  - name: v1alpha1
    served: true
    storage: true
status:
  acceptedNames:
    kind: APIManager
    listKind: APIManagerList
    plural: apimanagers
    singular: apimanager
  conditions:
  - lastTransitionTime: "2019-09-19T09:07:03Z"
    message: no conflicts found
    reason: NoConflicts
    status: "True"
    type: NamesAccepted
  - lastTransitionTime: null
    message: the initial names have been accepted
    reason: InitialNamesAccepted
    status: "True"
    type: Established
  storedVersions:
  - v1alpha1

After upgrade

APIManager

msoriano@localhost:~/go/src/github.com/3scale/3scale-operator (add-replicas-configurability)$ oc get apimanager example-apimanager -o yaml
apiVersion: apps.3scale.net/v1alpha1
kind: APIManager
metadata:
  annotations:
    apps.3scale.net/apimanager-threescale-version: "2.7"
    apps.3scale.net/threescale-operator-version: 0.4.0
  creationTimestamp: "2019-09-19T09:08:11Z"
  generation: 3
  name: example-apimanager
  namespace: msoriano-test
  resourceVersion: "749337"
  selfLink: /apis/apps.3scale.net/v1alpha1/namespaces/msoriano-test/apimanagers/example-apimanager
  uid: 015cfa41-dabd-11e9-8fc7-0efb785ddbd2
spec:
  apicast:
    managementAPI: status
    openSSLVerify: false
    productionSpec:
      replicas: 1
    registryURL: http://apicast-staging:8090/policies
    responseCodes: true
    stagingSpec:
      replicas: 1
  appLabel: 3scale-api-management
  backend:
    cronSpec:
      replicas: 1
    listenerSpec:
      replicas: 1
    workerSpec:
      replicas: 1
  imageStreamTagImportInsecure: false
  resourceRequirementsEnabled: false
  system:
    appSpec:
      replicas: 1
    database:
      mysql: {}
    fileStorage:
      amazonSimpleStorageService:
        awsBucket: enterprise-onprem.3sca.net
        awsCredentialsSecret:
          name: s3-bucket-credentials
        awsRegion: eu-west-1
    sidekiqSpec:
      replicas: 1
  tenantName: 3scale
  wildcardDomain: test.apps.miguel.testing.3sca.net
  zync:
    appSpec:
      replicas: 1
    queSpec:
      replicas: 1
status:
  deployments:
    ready:
    - apicast-production
    - apicast-staging
    - backend-cron
    - backend-listener
    - backend-redis
    - backend-worker
    - system-app
    - system-memcache
    - system-mysql
    - system-redis
    - system-sidekiq
    - system-sphinx
    - zync
    - zync-database
    - zync-que

APIManager CRD

msoriano@localhost:~/go/src/github.com/3scale/3scale-operator (add-replicas-configurability)$ oc get crd apimanagers.apps.3scale.net -o yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  creationTimestamp: "2019-09-19T09:07:03Z"
  generation: 1
  name: apimanagers.apps.3scale.net
  resourceVersion: "720352"
  selfLink: /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/apimanagers.apps.3scale.net
  uid: d8bd2193-dabc-11e9-85f9-0a5f891f16a0
spec:
  conversion:
    strategy: None
  group: apps.3scale.net
  names:
    kind: APIManager
    listKind: APIManagerList
    plural: apimanagers
    singular: apimanager
  scope: Namespaced
  subresources:
    status: {}
  validation:
    openAPIV3Schema:
      properties:
        apiVersion:
          description: 'APIVersion defines the versioned schema of this representation
            of an object. Servers should convert recognized schemas to the latest
            internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources'
          type: string
        kind:
          description: 'Kind is a string value representing the REST resource this
            object represents. Servers may infer this from the endpoint the client
            submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds'
          type: string
        metadata:
          type: object
        spec:
          properties:
            apicast:
              properties:
                image:
                  type: string
                managementAPI:
                  type: string
                openSSLVerify:
                  type: boolean
                registryURL:
                  type: string
                responseCodes:
                  type: boolean
              type: object
            appLabel:
              type: string
            backend:
              properties:
                image:
                  type: string
                redisImage:
                  type: string
              type: object
            highAvailability:
              properties:
                enabled:
                  type: boolean
              type: object
            imageStreamTagImportInsecure:
              type: boolean
            resourceRequirementsEnabled:
              type: boolean
            system:
              properties:
                database:
                  properties:
                    mysql:
                      description: Union type. Only one of the fields can be set
                      properties:
                        image:
                          type: string
                      type: object
                    postgresql:
                      properties:
                        image:
                          type: string
                      type: object
                  type: object
                fileStorage:
                  properties:
                    amazonSimpleStorageService:
                      properties:
                        awsBucket:
                          type: string
                        awsCredentialsSecret:
                          properties:
                            name:
                              type: string
                          type: object
                        awsRegion:
                          type: string
                      required:
                      - awsBucket
                      - awsRegion
                      - awsCredentialsSecret
                      type: object
                    persistentVolumeClaim:
                      description: Union type. Only one of the fields can be set.
                      properties:
                        storageClassName:
                          type: string
                      type: object
                  type: object
                image:
                  type: string
                memcachedImage:
                  type: string
                redisImage:
                  type: string
              type: object
            tenantName:
              type: string
            wildcardDomain:
              type: string
            zync:
              properties:
                image:
                  type: string
                postgreSQLImage:
                  type: string
              type: object
          required:
          - wildcardDomain
          type: object
        status:
          properties:
            conditions:
              items:
                properties:
                  status:
                    type: string
                  type:
                    type: string
                required:
                - type
                - status
                type: object
              type: array
            deployments:
              properties:
                ready:
                  description: Deployments are ready to serve requests
                  items:
                    type: string
                  type: array
                starting:
                  description: Deployments are starting, may or may not succeed
                  items:
                    type: string
                  type: array
                stopped:
                  description: Deployments are not starting, unclear what next step
                    will be
                  items:
                    type: string
                  type: array
              type: object
          required:
          - deployments
          type: object
  version: v1alpha1
  versions:
  - name: v1alpha1
    served: true
    storage: true
status:
  acceptedNames:
    kind: APIManager
    listKind: APIManagerList
    plural: apimanagers
    singular: apimanager
  conditions:
  - lastTransitionTime: "2019-09-19T09:07:03Z"
    message: no conflicts found
    reason: NoConflicts
    status: "True"
    type: NamesAccepted
  - lastTransitionTime: null
    message: the initial names have been accepted
    reason: InitialNamesAccepted
    status: "True"
    type: Established
  storedVersions:
  - v1alpha1

[eguzki]

Few comments pending to be reviewed, but generally, LGTM

[eguzki]

rebase and ready to be merged

[codeclimate]

Code Climate has analyzed commit 25c6d11 and detected 10 issues on this pull request.

Here's the issue category breakdown:

Category	Count
Complexity	1
Style	9

View more on Code Climate.