[POC] Can serverless-webhook be removed ?

[kwiatekus]

Description

This exercise it to verify if we can get rid of serverless-webhook in favor of validation rules that can be defined in the Function CRD itself.
Find out which features of serverless webhook would be sacrified if only validation rules on CRD would be used.
Could we introduce the rules and cover the same validation/defaulting rules w/o api breaking changes?

Reasons

Webhooks are intercepting calls to apiserver and introduce another point of failure.
By getting rid of our webhook we can improve resiliency of serverless module and remove the problem of CA generation.
Also, atm user can apply functions CR only after webhook is up, and it takes it a while until its up.
If we get rid of webhooks we will enable true declarative way of defining function.

Attachments

https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#validation-rules

[kyma-bot]

This issue or PR has been automatically marked as stale due to the lack of recent activity.
Thank you for your contributions.

This bot triages issues and PRs according to the following rules:

• After 60d of inactivity, lifecycle/stale is applied
• After 7d of inactivity since lifecycle/stale was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Close this issue or PR with /close

If you think that I work incorrectly, kindly raise an issue with the problem.

/lifecycle stale

[dbadura]

Our current validation rules for function CR:

• user can choose resoruce profile or custom resources.
• when nothing is passed, the default profile is set.
• default profile in case of empty profile and resoruces.

We can replace this logic in controller and put the the used profile into status. Checking if one of profile or resouces is set and in case none of them is set, setting defualt value sounds very complicated

• resoruce request has to be smaller than resource limits.

It's not possible because values are strings and comparing string is not the same as comparing the resources.

Opened ticket: kubernetes-sigs/controller-tools#827

• restriced ENVs

It's possible to achieve by using regex and all() fn.

• git url validation. It has to be http(s) or git.
• runtime validation.

It can be an enum.

• Reastrictied labels and annotations.
• replicas has to be between ScaleConfig min and max if ScaleConfig is defined.
• default profil for replicas.

Currently we don't have such field in spec. I would consider to remove it or use the same mechanism as in resource profile

• secret mounts has to be k8s compatible (e.g.: size) and mountpath cannot me empty.

• function is inline or git.

• baseDir and reference cannot be empty in case of git function.

We can consider to use default values like `master` and `/`

• gitAuth can be Key or Basic.
• Secret reference with git credentials.

Similar validation to k8s secret

PoC PR: #18034

Webhook secret:
It's executed agains secret with label: "serverless.kyma-project.io/remote-registry": "config"
and extracts:

• username
• password
• serverAddress
  then it create .dockerconfigjson and put it in secret.

Out secrer is created in chart and has .dockerconfigjson no matter if is internal or not. Consits of following fields:

• registryAddress
• serverAddress (it's used to create dockerconfigjson)

Our controller uses registryAddress to push by kaniko and set image in pod. dockerconfigjson is used to pull image by kubelet.

The distinction between serverAddress and registryAddress is because docker.hub is edge case. To pull images from dockerhub the https://index.docker.io/v1/ address is used and it's in dockerconfigjson. To pull images you have to type image name like e.g.: my-user/my-image.

User can create registry secret by kubectl:

  kubectl create secret docker-registry my-secret --docker-server=DOCKER_REGISTRY_SERVER --docker-username=DOCKER_USER
--docker-password=DOCKER_PASSWORD --docker-email=DOCKER_EMAIL

From out perspective this webhook is unnecesary if we point users to create registry secret using kubectl.

[kwiatekus]

The following steps would allow us to remove serverless webhook

• Check if build job can push to docker registry identified by secret of docker-registry type (w/o any additions like registry address)

• For defaulting we could use controller logic. I.e if user doesnt provide runtime/buildtime profile, the controller would use the currently configured default profile and inform about used profile in the status.

• Comparing if Limits > Requests could be dropped. Misconfigured function would show an error message in Deployed condition

• Reject not allowed changes via CEL for

  • ENVs
  • git url
  • runtime (CEL preferred over enum)
  • labels & annotations
  • secret mounts ( secret name regexp & mountPath )
  • function source type ( inline/git)
  • git references, secret validation

• Get rid of validation for spec.replicas :

• between ScaleConfig.min/max values

• Get rid of profile for replicas