feat: Add a Kubernetes Operator for the Feast Feature Server

[tchughesiv]

What this PR does / why we need it:

This PR wraps the existing feast-feature-server helm chart into a simple helm-based operator.

We should leverage the power of kubernetes controllers to manage the deployment state of the feast-feature-server. This is more operationally sound, as it can “repair” issues with deployments in a cluster. It also simplifies the management of multiple deployments within a cluster and opens the door to integrations with other operator-based projects.

This will require the creation of a new image repo @ docker.io/feastdev/feast-operator. Builds are automated, same as the existing feature server images.

While this solution will leverage a few kustomize manifests to deploy the operator itself, these manifests will require little to no maintenance. They should not change often (if at all) as they are not impacted by changes to the helm chart, nor do they impact how the feature server is deployed.

By using a symlink to the feast-feature-server helm directory, future operator images will always have the latest chart version enhancements.

Below is a sample FeastFeatureServer Custom Resource. As you can see, the fields in the FeastFeatureServer.spec directly match existing helm chart parameters -

apiVersion: charts.feast.dev/v1alpha1
kind: FeastFeatureServer
metadata:
  name: feastfeatureserver-sample
spec:
  # Default values copied from helm-charts/feast-feature-server/values.yaml
  affinity: {}
  # base64 encoding of `sdk/python/feast/templates/local/feature_repo/feature_store.yaml`
  feature_store_yaml_base64: "cHJvamVjdDogbXlfcHJvamVjdAojIEJ5IGRlZmF1bHQsIHRoZSByZWdpc3RyeSBpcyBhIGZpbGUgKGJ1dCBjYW4gYmUgdHVybmVkIGludG8gYSBtb3JlIHNjYWxhYmxlIFNRTC1iYWNrZWQgcmVnaXN0cnkpCnJlZ2lzdHJ5OiBkYXRhL3JlZ2lzdHJ5LmRiCiMgVGhlIHByb3ZpZGVyIHByaW1hcmlseSBzcGVjaWZpZXMgZGVmYXVsdCBvZmZsaW5lIC8gb25saW5lIHN0b3JlcyAmIHN0b3JpbmcgdGhlIHJlZ2lzdHJ5IGluIGEgZ2l2ZW4gY2xvdWQKcHJvdmlkZXI6IGxvY2FsCm9ubGluZV9zdG9yZToKICAgIHR5cGU6IHNxbGl0ZQogICAgcGF0aDogZGF0YS9vbmxpbmVfc3RvcmUuZGIKZW50aXR5X2tleV9zZXJpYWxpemF0aW9uX3ZlcnNpb246IDIK"
  fullnameOverride: ""
  image: {}
  imagePullSecrets: []
  livenessProbe:
    initialDelaySeconds: 30
    periodSeconds: 30
  nameOverride: ""
  nodeSelector: {}
  podAnnotations: {}
  podSecurityContext: {}
  readinessProbe:
    initialDelaySeconds: 20
    periodSeconds: 10
  replicaCount: 1
  resources: {}
  securityContext: {}
  service:
    port: 80
    type: ClusterIP
  tolerations: []

NOTE: This operator could easily be expanded in the future to also handle the release of the feast-release helm chart

Which issue(s) this PR fixes:

Fixes #4144

[dmartinol]

shouldn't we document somewhere what's the actual feature_store.yaml that this is representing? (and maybe in the future we can also add a more complex sample with integrated psql DB deployment)

[tchughesiv]

it's currently documented in the comment above field... that said, i can certainly add something additional to the getting started docs i'll be adding.

[tchughesiv]

fwiw, to keep maintenance low... i'd like to lean heavily on existing feast docs and samples. there's no need/reason for us to duplicate anything. for example -
https://github.com/feast-dev/feast/blob/master/sdk/python/feast/templates/postgres/feature_repo/feature_store.yaml
https://docs.feast.dev/project/development-guide#feast-python-feature-server-helm-chart

[tchughesiv]

i've added some additional docs, including how one might pass in their own feature_store.yaml during a deployment.

[jeremyary]

@tokoko @shuchu @HaoXuAI @sudohainguyen hey all, may I kindly ask for your considerations of the effort here? We'd love to hear any feedback or questions/convo around technical and/or objective aspects from some of our heavier contributors, so if you're able, we'd really appreciate it!

[tokoko]

This is more operationally sound, as it can “repair” issues with deployments in a cluster. It also simplifies the management of multiple deployments within a cluster and opens the door to integrations with other operator-based projects.

I'll be completely honest, this is my first time hearing of a helm-based operator and I'm not sure I understand the point. 😄What's the difference with normal helm deployment in any practical sense other than the fact that parameters are used to populate a CRD instead of helm values file? Previously I thought (maybe just my limited experience) that controllers came in handy when you needed some very custom behavior in cluster or maybe even orchestrating stuff outside the cluster. So my questions right now are 1) Is something like that even possible with helm-based operators? and 2) do we have any use case for it (actual or expected)? java feature server deployment doesn't look that unusual to me. I'm probably unaware of some other benefits, so some more context there will be appreciated...

[HaoXuAI]

Nice. I personally think the feast deployment process is not devops friendly. Hope this feature streamlines the process better

[tokoko]

Another slightly unrelated point to note here. I think feast has bit of an identity crisis when it comes to deployments like this. For example we have lambda feature server which has it's own registry object and is actually deployed when user runs feast apply (not a fan of this myself). At the same time we have ways to start servers either with cli or deploy with helm. I think we should try to make all this less confusing. Personally I think deploying anything anywhere with a feast apply needs to be killed off.

[franciscojavierarceo]

Another slightly unrelated point to note here. I think feast has bit of an identity crisis when it comes to deployments like this. For example we have lambda feature server which has it's own registry object and is actually deployed when user runs feast apply (not a fan of this myself). At the same time we have ways to start servers either with cli or deploy with helm. I think we should try to make all this less confusing. Personally I think deploying anything anywhere with a feast apply needs to be killed off.

I think we need to be able to allow users to pick and choose based on tradeoffs. Being able to deploy feature changes via feast apply is a choice that has been very successful for some and not within the risk tolerance of others. So I don't think we should be overly prescriptive on that.

[tokoko]

@franciscojavierarceo sorry, let me clarify. I wasn't referring to feature changes, I meant deployments like aws lambda feature server.

[tchughesiv]

@tokoko let me know if you have additional Qs

This is more operationally sound, as it can “repair” issues with deployments in a cluster. It also simplifies the management of multiple deployments within a cluster and opens the door to integrations with other operator-based projects.

I'll be completely honest, this is my first time hearing of a helm-based operator and I'm not sure I understand the point. 😄What's the difference with normal helm deployment in any practical sense other than the fact that parameters are used to populate a CRD instead of helm values file?

• The Operator’s controller will automatically recover missing objects in the cluster, and fix misconfigured ones. For example, if a feature server’s k8s Service or Deployment were accidentally deleted, the operator will recreate it. In critical environments, this is invaluable.
• Once an admin installs the operator to a cluster, all users can take advantage using native k8s tools to deploy their own feature servers, with the use of the new FeastFeatureServer CR. No helm knowledge or installation is required. This will be identical to how a user might create a Deployment object in their own namespace.
• Changes made to a FeastFeatureServer CR in-cluster will automatically be reflected in the feature server deployment. Nothing else has to be done.

Previously I thought (maybe just my limited experience) that controllers came in handy when you needed some very custom behavior in cluster or maybe even orchestrating stuff outside the cluster. So my questions right now are 1) Is something like that even possible with helm-based operators?

• Helm-based operators are limited in that respect. We would have to use a Go-based operator instead. But, as I stated in the related issue (Add a Kubernetes Operator for the Feast Feature Server, to the infra directory #4144), a go-based operator would add significant complexity to the development and maintenance of feast. Given that feast and its users already utilize, and prefer, helm… this seemed unnecessary. My intention with this approach was to continue to funnel future deployment contributions to the helm charts… rather than other deployment methods. Any future changes made to the helm charts will be automatically reflected in this helm-based operator.
• That said, do we think the community would prefer a Go-based operator instead? They can be a lot to take on and maintain, but i have significant experience with them.

and 2) do we have any use case for it (actual or expected)?

• IMO, aside from the controller benefits stated above, it's a great way to potentially consolidate all of the helm charts into a single, optional deployment mechanism. In the future, should the community so choose, all helm chart installs could be handled by this same feast-operator. Each would have its own CRD. e.g. FeastJavaFeatureServer or FeastTransformationService, etc.
• Also, should the community choose to, in the future this feast-operator could be added to https://operatorhub.io/ for greater visibility to potential k8s users… or to OpenShift’s internal OperatorHub, which is then made available to all OpenShift & OKD cluster users.
• More immediately, we’d be able to easily integrate the feast-feature-server into the Open Data Hub (ODH) Operator.

java feature server deployment doesn't look that unusual to me. I'm probably unaware of some other benefits, so some more context there will be appreciated...

[sudohainguyen]

totally support this, since there's a lot of room to consolidate deployment for feast
I would prefer going with the helm-based operator first due to its simplification 🙌

[tokoko]

@tchughesiv thanks for the lengthy response. I do see the benefits and since people will also be able to use normal helm charts as well, I don't see why we shouldn't do this. as for the go-based approach, I agree with Harry, right now helm is enough. If a suitable use case arises for a go-based one, we can revisit this later.

[tchughesiv]

thanks @tokoko @sudohainguyen @HaoXuAI @franciscojavierarceo
if you get a minute, there's a related PR #4161 that affects the full functionality of this operator. very simple helm fix.