Proposal: NifiReplicaCluster -- Unclustered NiFi Deployment

[mh013370]

Purpose

This proposal captures the high level design of a new NiFiKop resource called NifiReplicaCluster, which complements NifiCluster in the sense that it deploys NiFi in a different operating mode: unclustered.

One feature lacking in NiFiKop is the ability to treat a NiFi cluster like a set of unrelated replicas (i.e. cattle) instead of as a clustered set of nodes which require special attention (i.e. pets) during upgrades or calculated NiFi cluster operations. For the remainder of this document, NiFi clusters will be referred to as either clustered or unclustered, where clustered is the default behavior and is the same as NifiCluster. An unclustered NiFi is one where none of the nodes are aware of each other, but they are collectively running the same flow.

Benefits of clustering

• You may interact with every node in the NiFi cluster via the Cluster Coordinator (e.g. the zookeeper-elected leader). All REST API requests are internally replicated to every node by the coordinator.
• You gain coordination features like:
• flow election, guaranteeing that every node is running the exact same flow
• primary node only processors, guaranteeing that only one node in the cluster will run a particular processor. Useful for data pulls or scenarios where you only want 1 node to do some work.
• As of NiFi 1.16.0 , flows can now be edited while one cluster node is disconnected (previously not possible). Disconnected nodes can now rejoin the cluster, updating their flow to match whatever the cluster gives it. Even if that flow is drastically different.

Drawbacks of clustering

• Upgrades are challenging, requiring a careful process to disconnect nodes from the cluster and then draining, stopping, upgrading, and then restarting them.
• It's not possible to upgrade a cluster with zero downtime. Nor is it possible to run a cluster with a mixed set of NiFi versions.
• In large cluster scenarios, nodes tend to disconnect from the cluster in production and it can be painful getting the node re-added to the cluster.

Benefits of unclustered NiFis

• You can trivially rolling upgrade each NiFi node
• You can support a mixed set of NiFi versions in the "cluster"
• There's no complicated leader election or cluster connectivity problems
• Easily horizontally and vertically scalable

Drawbacks of unclustered NiFis

• There's no flow election, so there will be periods of time during flow upgrades where nodes in the "cluster" will be running different flows.
• You lose the ability to drain/offload flowfiles from one NiFi to another. This could be problematic if the NiFi can't successfully process a particular flowfile and gets hung up.
• In order to command & control a "cluster" RESTfully, you need to issue REST requests to each node instead of just the cluster coordinator.

There are benefits and drawbacks to each of these deployment methods, but NiFiKop only provides the ability to deployed clustered NiFis. This is why we are introducing the NifiReplicaCluster concept.

In depth documentation can be found here and in neighboring pages: https://konpyutaika.github.io/nifikop/docs/1_concepts/3_features

NiFiKop API

NiFiKop exposes its API through several Kubernetes custom resource definitions (CRDs):

• NifiCluster - This is the primary specification which defines a clustered NiFi deployment. This resource contains Node specifications and all of the necessary NiFi application configuration.
• NifiDataflow - This resource captures a version controlled flow which has been persisted in NiFi Registry, the component application which handles saving/retrieving flows and nar bundle resources. NiFiKop deploys this versioned flow to the NiFi cluster. One or more may be deployed to a cluster.
• NifiUser - This resources captures the accesses and permissions a particular user identity has in a NifiCluster. NiFiKop configures the user in the NiFi application.
• NifiUserGroup - This resource captures the accesses and permissions a user group has in a NiFi cluster. A NifiUser may belong to zero or more groups.
• NifiParameterContext - This resource captures the information necessary to populate a ParameterContext in NiFi. A NifiDataflow may have zero or more ParameterContexts configured.
• NifiRegistryClient - The configuration for a remote NiFi Registry that the operator configures each NiFi pod in a cluster to communicate with for fetching versioned dataflows.

Together, these resources allow you to declaratively define a clustered NiFi configuration. NiFiKop processes these resources and translates them into a deployment in Kubernetes. NiFiKop elects to oversee the Pod lifecycle instead of leveraging a Kubernetes Deployment or ReplicaSet. This is because NiFiKop allows granular and varying node configurations within a NiFi cluster, which isn't possible with a Deployment/ReplicaSet. With the latter resources, the same spec is applied to every Pod in the deployment.

NifiReplicaCluster Concept

Where NiFiKop's NifiCluster deploys NiFi in a clustered configuration and implements various logic to ensure that each node in the cluster remains that way, NifiReplicaCluster deploys unclustered NiFi pods that are entirely isolated from each other but each share the following resources:

• NifiDataflow
• NifiUser
• NifiUserGroup
• NifiRegistryClient
• NifiParameterContext

This enables us to deploy NiFi in an unclustered configuration, but configure each NiFi pod with the same users, user groups, dataflows, parameter contexts, and NiFi Registry client. In this way, we treat each node as a logical cluster, but they are unclustered in the NiFi sense. This deployment method enables us to perform the following operations more simply than in the clustered NiFi case:

• "Zero downtime" rolling upgrades
  • zero downtime is in quotes, because when a single NiFi pod is restarted none of the flowfiles its processing will be offloaded. If the NiFi repositories are held in persistent volumes, then they will still be there when the pod comes back up. In this way, it's no different than restarting the NiFi process on a VM. Zero downtime refers to the fact that it's not possible to upgrade NiFi clusters without completely shutting them off. Unclustered NiFis may be rolling upgraded and the disruption spread over time while maintaining availability.
• Trivial horizontal autoscaling
• Avoid common clustering-related operational issues like node disconnections, flow inconsistencies

These benefits comes with some sacrifices:

• You lose all clustered NiFi coordination features like
  • flow election & voting
  • the primary node only processor feature
  • active node draining/off-loading
  • cluster coordinator to orchestrate & replicate REST requests
• When RESTfully interacting with a clustered NiFi, you only need to send a request to the cluster coordinator. The request is then automatically replicated to each node in the cluster. In an unclustered NiFi scenario, a REST request must be made to each node in the replica cluster by the client.

As previously mentioned, there are benefits and drawbacks to each approach. For users who do not need cluster coordination features or users who wish to perform zero downtime upgrades, then a NifiReplicaCluster might be a suitable alternative.

NifiReplicaCluster Design

A new NiFiKop API resource NifiReplicaCluster is introduced, which is quite similar to NifiCluster:

• NifiReplicaCluster
  • NifiReplicaClusterSpec
    • replicas int32
    • stateful bool
    • serviceSpec
      • headlessEnabled bool
      • annotations map[string]string
      • labels map[string]string
    • podSpec
      • annotations map[string]string
      • labels map[string]string
      • zkAddress string
      • zkPath string
      • clusterImage string
      • oneNifiPerNode bool
      • autoscalerConfig
        • nodeConfigSpec
      • externalVolumeConfigs list
        • volumeMount
        • volumeSource
      • storageConfigs list
        • name string
        • mountPath
        • PVCSpec
      • serviceAccountName string
      • resources
        • resourcesRequirements
      • propagateLabels bool
      • listenersConfig
        • internalListeners
      • externalServices
  • NifiReplicaClusterStatus
    • rootProcessGroupId string
    • clusterState string
    • creationTime time

The benefit of this resource being so similar to NifiCluster and re-using the NifiDataflow, NifiUser, NifiUserGroup, NifiRegistryClient, and NifiParameterContext resources is that it can re-use the logic written in controllers to configure the NiFi pods in the deployment with all of this information. The lone difference is that the controllers won't be able to use the NiFi cluster coordinator to replicate requests on their behalf. They will need to issue requests to each node in the cluster and ensure that the request to each node was successful to ensure state.

We need not concern ourselves with embedded/remote zookeeper since we're not using any cluster coordination features.

A controller will be created for this API resource that performs the following actions as a part of its reconciliation loop:

• Create a NifiReplicaCluster Deployment or StatefulSet depending on whether the spec.stateful flag is set with a PodSpec composed of:
  • the clusterImage
  • oneNifiPerNode
  • NifiReplicaCluster podSpec
  • nodeConfigSpec
  • listenersConfig
  • all of the nifi application configuration (nifi.properties, bootstrap.properties, zookeeper.properties, etc)
• Configure the NiFi PrometheusReportingTask for each pod
• Configure PKI for each of the NiFI pods using the existing pki manager in NiFiKop
• Create a Service(s) for the Deployment/StatefulSet according to the serviceSpec and externalServices config

Existing Controller Updates

Each of the below controllers will be updated to handle two separate cases. One where the resources have a clusterRef and one where they have a replicaClusterRef. For each controller, nothing will change in the existing clusterRef scenario. For a replicaClusterRef, the implementation will be the same except for any interaction with NiFi via a REST client must now interact with every pod in the deployment. The controller should ensure that the request with each pod is successful:

• NifiDataflow
• NifiUser
• NifiUserGroup
• NifiRegistryClient
• NifiParameterContext

Deployment/StatefulSet Update Strategies

Because we're using k8s native Deployment and/or StatefulSets, we can take advantage of native Kubernetes update strategies.

• https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#update-strategies
• https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#strategy

There is also interest in implementing various upgrade strategies for unclustered NiFis. These are the scenarios we wish to support, perhaps through the use of finalizers:

• Drain : Mark pod for deletion and allow it to drain naturally before killing it. Finalizers may be used to accomplish this
• Offload : Mark pod for deletion and prompt the nifi to push all of its data to a remote store
• Drop : Mark pod for deletion and immediately kill it, dropping any data the pod was processing

[mh013370]

Continuing this idea, this could perhaps be extended to deploy NiFi in another operating mode: StatelessNiFi

• https://bryanbende.com/development/2021/11/10/apache-nifi-stateless
• https://github.com/pvillard31/nifi-stateless-demo

[mh013370]

Continuing this idea, this could perhaps be extended to deploy NiFi in another operating mode: StatelessNiFi

• https://bryanbende.com/development/2021/11/10/apache-nifi-stateless
• https://github.com/pvillard31/nifi-stateless-demo

Ah, nevermind. This all changed to a processor called ExecuteStateless instead of a dedicated binary in 1.15.0. 😄

[mh013370]

I explored this idea and compared it to just extending the NifiCluster controller to handle a different cluster type. I don't believe this use case warrants an entirely new CRD (e.g. a NifiReplicaSet) as a cluster of standalone NiFis are just a different operating mode of a NiFi cluster.

I found that for standalone clusters, if the operator acts like the cluster coordinator (in the clustered NiFi case), then we can achieve nifikop driving a set of standalone NiFis.

For this reason, I'm going to close this discussion and raise a new one to discuss extending NiFiKOp to drive a standalone cluster.