[Feature/Operator] Enable resource lifecycle management for external NiFi cluster

CHANGELOG.md

@@ -10,15 +10,47 @@
 
 ### Fixed Bugs
 
+## v0.7.0
+
+### Added
+
+- [PR #132](https://github.com/Orange-OpenSource/nifikop/pull/132) - **[Operator]** Add the ability to manage dataflow lifecycle on non managed NiFi Cluster.
+- [PR #132](https://github.com/Orange-OpenSource/nifikop/pull/132) - **[Operator]** Operator can interact with the NiFi cluster using basic authentication in addition to tls.
+
+### Changed
+
+- [PR #132](https://github.com/Orange-OpenSource/nifikop/pull/132) - **[Operator]** Enabling the ability to move a resource from one cluster to another by just changing the clusterReference.
+- [PR #132](https://github.com/Orange-OpenSource/nifikop/pull/132) - **[Operator]** Improves the performances by reducing the amont of errors when interacting with then NiFi cluster API, checking cluster readiness before applying actions.
+- [PR #132](https://github.com/Orange-OpenSource/nifikop/pull/132) - **[Operator/NiFiCluster]** Support `evicted` and `shutdown` pod status as terminating.
+
+### Deprecated
+
+### Removed
+
+### Fixed Bugs
+
+- [PR #132](https://github.com/Orange-OpenSource/nifikop/pull/132) - **[Operator/NiFiCluster]** Fix the downscale issue ([PR #131](https://github.com/Orange-OpenSource/nifikop/issues/131)) by removing references to configmap
+- [PR #132](https://github.com/Orange-OpenSource/nifikop/pull/132) - **[Helm Chart]** Fix the RBAC definition for configmap and lease generated by operator-sdk with some mistakes.
+- [PR #132](https://github.com/Orange-OpenSource/nifikop/pull/132) - **[Helm Chart]** Add corect CRDs in the chart helm.
+- [PR #132](https://github.com/Orange-OpenSource/nifikop/pull/132) - **[Operator/NiFiUser]** Fix policy check conflict between user and group scope policy.
+
+## v0.6.4
+
+### Fixed Bugs
+
+- [COMMIT #d98eb15fb3a74a1be17be5d456b02bd6a2d333cd](https://github.com/Orange-OpenSource/nifikop/tree/d98eb15fb3a74a1be17be5d456b02bd6a2d333cd) - **[Fix/NiFiCluster]** Fix external service port configuration being ignore [#133](https://github.com/Orange-OpenSource/nifikop/issues/133)
+- [PR #134](https://github.com/Orange-OpenSource/nifikop/pull/134) - **[Operator/NifiCluster]** corrected typo in the nifi configmap for bootstrap-notification-service.
+- [PR #119](https://github.com/Orange-OpenSource/nifikop/pull/119) - **[Helm Chart]** bring nificlusters crd in helm chart to spec with rest of repo.
+
 ## v0.6.3
 
 ### Added
 
-- [PR #114](https://github.com/Orange-OpenSource/nifikop/pull/114) - **[Fix/NiFiCluster]** Additionals environment variables.
+- [PR #114](https://github.com/Orange-OpenSource/nifikop/pull/114) - **[Operator/NiFiCluster]** Additionals environment variables.
 
 ### Fixed Bugs
 
-- [PR #113](https://github.com/Orange-OpenSource/nifikop/pull/113) - **[Fix/NiFiDataflow]** Simple work around to avoid null pointer dereferencing on nifi side.
+- [PR #113](https://github.com/Orange-OpenSource/nifikop/pull/113) - **[Operator/NiFiDataflow]** Simple work around to avoid null pointer dereferencing on nifi side.
 
 ## v0.6.2
 
@@ -36,7 +68,6 @@
 - [PR #93](https://github.com/Orange-OpenSource/nifikop/pull/93) - **[Helm Chart]** Included securityContext and custom service account in helm chart for NiFiKop deployment.
 - [PR #100](https://github.com/Orange-OpenSource/nifikop/pull/100) - **[Helm Chart]** Add nodeSelector, affinty and toleration in helm chart for NiFiKop deployment.
 
-
 ## v0.6.0
 
 ### Added
@@ -116,7 +147,7 @@
 
 ### Fixed Bugs
 
-- [PR #53](https://github.com/Orange-OpenSource/nifikop/pull/53) - **[Operator]** Upgrade k8s dependencies to match with new version requirement : [#52](https://github.com/Orange-OpenSource/nifikop/issues/52) [#51](https://github.com/Orange-OpenSource/nifikop/issues/51) [#33](https://github.com/Orange-OpenSource/nifikop/issues/33) 
+- [PR #53](https://github.com/Orange-OpenSource/nifikop/pull/53) - **[Operator]** Upgrade k8s dependencies to match with new version requirement : [#52](https://github.com/Orange-OpenSource/nifikop/issues/52) [#51](https://github.com/Orange-OpenSource/nifikop/issues/51) [#33](https://github.com/Orange-OpenSource/nifikop/issues/33)
 - [PR #53](https://github.com/Orange-OpenSource/nifikop/pull/53) - **[Operator]** Fix the users used into Reader user group
 - [PR #53](https://github.com/Orange-OpenSource/nifikop/pull/53) - **[Documentation]** Fix the chart version informations : [#51](https://github.com/Orange-OpenSource/nifikop/issues/51)
 
@@ -132,7 +163,6 @@
 - [PR #41](https://github.com/Orange-OpenSource/nifikop/pull/41) - **[Operator/NifiCluster]** Create three defaults groups : admins, readers, nodes
 - [PR #41](https://github.com/Orange-OpenSource/nifikop/pull/41) - **[Operator/NifiCluster]** Add pod disruption budget support
 
-
 ### Changed
 
 - [PR #41](https://github.com/Orange-OpenSource/nifikop/pull/41) - **[Helm Chart]** Add CRDs
@@ -143,7 +173,6 @@
 
 - [PR #41](https://github.com/Orange-OpenSource/nifikop/pull/41) - **[Operator/NifiCluster]** Remove `ClusterSecure` and `SiteToSiteSecure` by only checking if `SSLSecret` is set.
 
-
 ### Fixed Bugs
 
 - [PR #30](https://github.com/Orange-OpenSource/nifikop/pull/40) - **[Documentation]** Fix getting started
@@ -168,7 +197,7 @@
 
 ### Added
 
-- [PR #25](https://github.com/Orange-OpenSource/nifikop/pull/25) -  [Helm Chart] Add support for iterating over namespaces
+- [PR #25](https://github.com/Orange-OpenSource/nifikop/pull/25) - [Helm Chart] Add support for iterating over namespaces
 - [PR #18](https://github.com/Orange-OpenSource/nifikop/pull/18) - [Operator] NiFiKop CRDs in version `v1beta1` of CustomResourceDefinition object.
 
 ### Changed
@@ -200,8 +229,8 @@
 
 - [MR #17](https://github.com/Orange-OpenSource/nifikop/-/merge_requests/17) - Upgrade dependencies
 - [MR #17](https://github.com/Orange-OpenSource/nifikop/-/merge_requests/17) - CRD generated under `apiextensions.k8s.io/v1`
-- [MR #16](https://github.com/Orange-OpenSource/nifikop/-/merge_requests/16) - Set binami zookeeper helm chart as recommended solution for 
-ZooKeeper.
+- [MR #16](https://github.com/Orange-OpenSource/nifikop/-/merge_requests/16) - Set binami zookeeper helm chart as recommended solution for
+  ZooKeeper.
 - [MR #16](https://github.com/Orange-OpenSource/nifikop/-/merge_requests/16) - Improve terraform setup for articles.
 - [MR #18](https://gitlab.si.francetelecom.fr/kubernetes/nifikop/-/merge_requests/18) - Add ability to define if cert-manager is cluster scoped or not.
 - [MR #18](https://gitlab.si.francetelecom.fr/kubernetes/nifikop/-/merge_requests/18) - Open source changes
@@ -266,4 +295,4 @@ ZooKeeper.
 
 ### Removed
 
-### Fixed Bugs
+### Fixed Bugs

api/v1alpha1/common_types.go

@@ -14,7 +14,9 @@
 
 package v1alpha1
 
-import "fmt"
+import (
+	"fmt"
+)
 
 // DataflowState defines the state of a NifiDataflow
 type DataflowState string
@@ -46,6 +48,12 @@ type InitClusterNode bool
 // PKIBackend represents an interface implementing the PKIManager
 type PKIBackend string
 
+// ClientConfigType represents an interface implementing the ClientConfigManager
+type ClientConfigType string
+
+// ClusterType represents an interface implementing the  ClientConfigManager
+type ClusterType string
+
 // AccessPolicyType represents the type of access policy
 type AccessPolicyType string
 
@@ -82,6 +90,10 @@ func (r State) Complete() State {
 	}
 }
 
+func (r ClusterState) IsReady() bool {
+	return r == NifiClusterRunning || r == NifiClusterReconciling
+}
+
 // NifiAccessType hold info about Nifi ACL
 type NifiAccessType string
 
@@ -162,13 +174,13 @@ type AccessPolicy struct {
 	ComponentId string `json:"componentId,omitempty"`
 }
 
-func (a *AccessPolicy) GetResource(cluster *NifiCluster) string {
+func (a *AccessPolicy) GetResource(rootProcessGroupId string) string {
 	if a.Type == GlobalAccessPolicyType {
 		return string(a.Resource)
 	}
 	componentId := a.ComponentId
 	if a.ComponentType == "process-groups" && componentId == "" {
-		componentId = cluster.Status.RootProcessGroupId
+		componentId = rootProcessGroupId
 	}
 	resource := a.Resource
 	if a.Resource == ComponentsAccessPolicyResource {
@@ -244,6 +256,16 @@ const (
 	//PKIBackendVault PKIBackend = "vault"
 )
 
+const (
+	ClientConfigTLS   ClientConfigType = "tls"
+	ClientConfigBasic ClientConfigType = "basic"
+)
+
+const (
+	ExternalCluster ClusterType = "external"
+	InternalCluster ClusterType = "internal"
+)
+
 const (
 	// DataflowStateCreated describes the status of a NifiDataflow as created
 	DataflowStateCreated DataflowState = "Created"
@@ -315,6 +337,8 @@ type NodeState struct {
 	ConfigurationState ConfigurationState `json:"configurationState"`
 	// InitClusterNode contains if this nodes was part of the initial cluster
 	InitClusterNode InitClusterNode `json:"initClusterNode"`
+	// PodIsReady whether or not the associated pod is ready
+	PodIsReady bool `json:"podIsReady"`
 }
 
 // RackAwarenessState holds info about rack awareness status
@@ -381,3 +405,36 @@ const (
 	// NotInitClusterNode states the node is not part of initial cluster setup
 	NotInitClusterNode InitClusterNode = false
 )
+
+func ClusterRefsEquals(clusterRefs []ClusterReference) bool {
+	c1 := clusterRefs[0]
+	name := c1.Name
+	ns := c1.Namespace
+
+	for _, cluster := range clusterRefs {
+		if name != cluster.Name || ns != cluster.Namespace {
+			return false
+		}
+	}
+
+	return true
+}
+
+func SecretRefsEquals(secretRefs []SecretReference) bool {
+	name := secretRefs[0].Name
+	ns := secretRefs[0].Namespace
+	for _, secretRef := range secretRefs {
+		if name != secretRef.Name || ns != secretRef.Namespace {
+			return false
+		}
+	}
+	return true
+}
+
+type DataflowSyncMode string
+
+const (
+	SyncNever DataflowSyncMode  = "never"
+	SyncOnce DataflowSyncMode   = "once"
+	SyncAlways DataflowSyncMode = "always"
+)

api/v1alpha1/nificluster_types.go

@@ -17,11 +17,12 @@ limitations under the License.
 package v1alpha1
 
 import (
+	"strings"
+
 	cmmeta "github.com/jetstack/cert-manager/pkg/apis/meta/v1"
 	corev1 "k8s.io/api/core/v1"
 	"k8s.io/apimachinery/pkg/api/resource"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
-	"strings"
 )
 
 const (
@@ -37,14 +38,30 @@ const (
 
 // NifiClusterSpec defines the desired state of NifiCluster
 type NifiClusterSpec struct {
+	// clientType defines if the operator will use basic or tls authentication to query the NiFi cluster.
+	// +kubebuilder:validation:Enum={"tls","basic"}
+	ClientType ClientConfigType `json:"clientType,omitempty"`
+	// type defines if the cluster is internal (i.e manager by the operator) or external.
+	// +kubebuilder:validation:Enum={"external","internal"}
+	Type ClusterType `json:"type,omitempty"`
+	// nodeURITemplate used to dynamically compute node uri (used if external type)
+	NodeURITemplate string `json:"nodeURITemplate,omitempty"`
+	// nifiURI used access through a LB uri (used if external type)
+	NifiURI string `json:"nifiURI,omitempty"`
+	// rootProcessGroupId contains the uuid of the root process group for this cluster (used if external type)
+	RootProcessGroupId string `json:"rootProcessGroupId,omitempty"`
+	// secretRef reference the secret containing the informations required to authentiticate to the cluster (used if external type)
+	SecretRef SecretReference `json:"secretRef,omitempty"`
+	// proxyUrl defines the proxy required to query the NiFi cluster (used if external type)
+	ProxyUrl string `json:"proxyUrl,omitempty"`
 	// Service defines the policy for services owned by NiFiKop operator.
 	Service ServicePolicy `json:"service,omitempty"`
 	// Pod defines the policy for  pods owned by NiFiKop operator.
 	Pod PodPolicy `json:"pod,omitempty"`
 	// zKAddress specifies the ZooKeeper connection string
 	// in the form hostname:port where host and port are those of a Zookeeper server.
 	// TODO: rework for nice zookeeper connect string =
-	ZKAddress string `json:"zkAddress"`
+	ZKAddress string `json:"zkAddress,omitempty"`
 	// zKPath specifies the Zookeeper chroot path as part
 	// of its Zookeeper connection string which puts its data under same path in the global ZooKeeper namespace.
 	ZKPath string `json:"zkPath,omitempty"`
@@ -58,7 +75,7 @@ type NifiClusterSpec struct {
 	// oneNifiNodePerNode if set to true every nifi node is started on a new node, if there is not enough node to do that
 	// it will stay in pending state. If set to false the operator also tries to schedule the nifi node to a unique node
 	// but if the node number is insufficient the nifi node will be scheduled to a node where a nifi node is already running.
-	OneNifiNodePerNode bool `json:"oneNifiNodePerNode"`
+	OneNifiNodePerNode bool `json:"oneNifiNodePerNode,omitempty"`
 	// propage
 	PropagateLabels bool `json:"propagateLabels,omitempty"`
 	// managedAdminUsers contains the list of users that will be added to the managed admin group (with all rights)
@@ -81,7 +98,7 @@ type NifiClusterSpec struct {
 	// TODO : add vault
 	//VaultConfig         	VaultConfig         `json:"vaultConfig,omitempty"`
 	// listenerConfig specifies nifi's listener specifig configs
-	ListenersConfig ListenersConfig `json:"listenersConfig"`
+	ListenersConfig *ListenersConfig `json:"listenersConfig,omitempty"`
 	// SidecarsConfig defines additional sidecar configurations
 	SidecarConfigs []corev1.Container `json:"sidecarConfigs,omitempty" patchStrategy:"merge" patchMergeKey:"name" protobuf:"bytes,2,rep,name=containers"`
 	// ExternalService specifies settings required to access nifi externally
@@ -654,3 +671,48 @@ func (nSpec *NifiClusterSpec) GetMetricPort() *int {
 
 	return nil
 }
+
+func (cluster *NifiCluster) RootProcessGroupId() string {
+	return cluster.Status.RootProcessGroupId
+}
+
+func (c *NifiCluster) GetClientType() ClientConfigType {
+	if c.Spec.ClientType == "" {
+		return ClientConfigTLS
+	}
+	return c.Spec.ClientType
+}
+
+func (c *NifiCluster) GetType() ClusterType {
+	if c.Spec.Type == "" {
+		return InternalCluster
+	}
+	return ExternalCluster
+}
+
+func (c *NifiCluster) IsSet() bool {
+	return (c.GetType() == InternalCluster && len(c.Name) != 0) ||
+		(c.GetType() != ExternalCluster && len(c.Spec.NodeURITemplate) != 0 && len(c.Spec.RootProcessGroupId) != 0)
+}
+
+func (c *NifiCluster) IsInternal() bool {
+	return c.GetType() == InternalCluster
+}
+
+func (c NifiCluster) IsExternal() bool {
+	return c.GetType() != InternalCluster
+}
+
+func (cluster NifiCluster) IsReady() bool {
+	for _, nodeState := range cluster.Status.NodesState {
+		if nodeState.ConfigurationState != ConfigInSync || nodeState.GracefulActionState.State != GracefulUpscaleSucceeded ||
+			!nodeState.PodIsReady {
+			return false
+		}
+	}
+	return cluster.Status.State.IsReady()
+}
+
+func (cluster *NifiCluster) Id() string {
+	return cluster.Name
+}

api/v1alpha1/nifidataflow_types.go

@@ -35,8 +35,9 @@ type NifiDataflowSpec struct {
 	FlowVersion *int32 `json:"flowVersion,omitempty"`
 	// contains the reference to the ParameterContext with the one the dataflow is linked.
 	ParameterContextRef *ParameterContextReference `json:"parameterContextRef,omitempty"`
-	// if the flow will be ran once or continuously checked
-	RunOnce *bool `json:"runOnce,omitempty"`
+	// if the flow will be synchronized once, continuously or never
+	// +kubebuilder:validation:Enum={"never","always","once"}
+	SyncMode *DataflowSyncMode `json:"syncMode,omitempty"`
 	// whether the flow is considered as ran if some controller services are still invalid or not.
 	SkipInvalidControllerService bool `json:"skipInvalidControllerService,omitempty"`
 	// whether the flow is considered as ran if some components are still invalid or not.
@@ -143,16 +144,37 @@ func init() {
 	SchemeBuilder.Register(&NifiDataflow{}, &NifiDataflowList{})
 }
 
-func (d *NifiDataflowSpec) GetRunOnce() bool {
-	if d.RunOnce != nil {
-		return *d.RunOnce
+func (d *NifiDataflowSpec) GetSyncMode() DataflowSyncMode {
+	if d.SyncMode == nil {
+		return SyncAlways
 	}
-	return true
+	return *d.SyncMode
 }
 
-func (d *NifiDataflowSpec) GetParentProcessGroupID(cluster *NifiCluster) string {
+func (d *NifiDataflowSpec) SyncOnce() bool {
+	if d.GetSyncMode() == SyncOnce {
+		return true
+	}
+	return false
+}
+
+func (d *NifiDataflowSpec) SyncAlways() bool {
+	if d.GetSyncMode() == SyncAlways {
+		return true
+	}
+	return false
+}
+
+func (d *NifiDataflowSpec) SyncNever() bool {
+	if d.GetSyncMode() == SyncNever {
+		return true
+	}
+	return false
+}
+
+func (d *NifiDataflowSpec) GetParentProcessGroupID(rootProcessGroupId string) string {
 	if d.ParentProcessGroupID == "" {
-		return cluster.Status.RootProcessGroupId
+		return rootProcessGroupId
 	}
 	return d.ParentProcessGroupID
 }