title | authors | sponsors | reviewers | approvers | creation-date | ||||
---|---|---|---|---|---|---|---|---|---|
Server-Side Apply |
|
|
|
|
2022-03-17 |
Server-Side Apply (SSA) allows calculating the final patch to update resources in Kubernetes in the server instead of the client. This proposal describes how ArgoCD can leverage SSA during syncs.
- Open Questions
- Summary
- Motivation
- Goals
- Non-Goals
- Proposal
- Drawbacks
When SSA is enabled, the server may return field conflicts with other managers. What ArgoCD controller should do in case of conflict? Just force the sync and log warnings (like some other controllers do?)
The first version should use the force flag and override even if there are conflicts. We could improve and add other options once there is a use case.
Should Server-Side Apply support in ArgoCD be implemented allowing multiple managers for the same controller? (more details)
ArgoCD can benefit from Server-Side Apply during syncs. A few improvements to consider:
- More reliable dry-runs (as admission controller is executed) (ISSUE-804)
- Syncs always run mutating webhooks (even without diff)
- Fix big CRD sync issues
- Better interoperability with different controllers
Kubernetes SSA Proposal (KEP-555) has more details about how it works.
ArgoCD uses kubectl library while syncing resources in the cluster. Kubectl uses
by default a 3-way-merge logic between the live state (in k8s), desired state
(in git) and the previous state (last-applied-configuration
annotation) to
calculate diffs and patch resources in the cluster. This logic is executed in
the client (ArgoCD) and once the patch is calculated it is then sent to the
server.
This strategy works well in the majority of the use cases. However, there are some scenarios where calculating patches in the client side can cause problems.
Some of the known problems about using client-side approach:
More and more users are adopting and configuring Admission Controllers in Kubernetes with different flavors of Validating Webhooks and Mutating Webhooks. Admission Controllers will only execute in server-side. In cases where users rely on dry-run executions to decide if they should proceed with a deployment, having the patch calculated at the client side might provide undesired results. Having SSA enabled syncs also guaranties that Admission Controllers are always executed, even when there is no diff in the resource.
Server-Side Apply will better handle and identify conflicts during syncs by
analyzing the managedFields
metadata available in all Kubernetes resources
(since kubernetes 1.18).
By not having to rely on the last-applied-configuration
annotation, SSA would
help with failing syncs caused by exceeded annotation size limit. This is a
common issue when syncing CRDs with large schemas.
All following goals should be achieve in order to conclude this proposal:
- Provide the ability for users to define if they want to use SSA during syncs
- Users should be able to enable SSA at the controller level (via binary flag) (see UC-1)
- Users should be able to enable SSA for a given Application (via syncOptions) (see UC-2)
- Users should be able to enable SSA at resource level (via annotation) (see UC-3
- Relates to ISSUE-2267
- Diffing needs to support strategic merge patch (see ISSUE-2268)
- Make sure Services can be patched correctly (more details)
- Allow Admission Controllers to execute even when there is no diff for a particular resource. (Needs investigation) (more details)
- ArgoCD should respect field ownership and provide a configuration to allow users to define the behavior in case of conflicts (see Q-1 outcome)
- ArgoCD must register itself with a pre-defined manager (suggestion:
argocd-controller
). It shouldn't rely on the default value defined in the kubectl code. (more details)
TBD
Change ArgoCD controller to accept new parameter to enable Server-Side Apply during syncs. Changes are necessary in ArgoCD as well as in gitops-engine library.
The following use cases should be implemented:
[UC-1]: As a user, I would like enable SSA at the controller level so all Application are applied server-side
Implement a binary flag to configure ArgoCD to run all syncs using SSA.
(suggestion: --server-side-apply=true
). Default value should be false
.
[UC-2]: As a user, I would like enable SSA at the Application level so all resources are applied server-side
Implement a new syncOption to allow users to enable SSA at the application
level (Suggestion ServerSideApply=true
). UI needs to be updated to support
this new sync option. If not informed, the controller should keep the current
behaviour (client-side).
[UC-3]: As a user, I would like enable SSA at the resource level so only a single manifest is applied server-side
Leverage the existing argocd.argoproj.io/sync-options
annotation allowing the
ServerSideApply=true
to be informed at the resource level. Must not impact
other sync-options informed in the annotation (make sure this annotation
supports providing multiple options).
TBD
ArgoCD must check if the target Kubernetes cluster has full support for SSA. The feature turned GA in Kubernetes 1.22. Full support for managed fields was introduced as beta in Kubernetes 1.18. The implementation must check that the target kubernetes cluster is running at least version 1.18. If SSA is enabled and target cluster version < 1.18 ArgoCD should log warning and fallback to client sync.
Kubernetes SSA proposal (KEP-555) mentions about alternating between
server-side and client-side applies in the Upgrade/Downgrade Strategy
section. It is stated that Kubernetes will verify the incoming apply request
validating if the user-agent is kubectl
to decide if the
last-applied-configuration
annotation should be updated. ArgoCD relies on this
annotation and the implementation must make sure that this agent is correctly
informed when changing to server-side apply and specifying a manager different
than kubectl
. This is mainly to make sure that
G-5 isn't impacting the
client-side/server-side compatibility.
No CRD update necessary as syncOption
field in Application resource is non-typed
(string array). Upgrade will only require ArgoCD controller update.
Slight increase in ArgoCD code base complexity.