How are Kubernetes objects deployed in MetalK8s?

[NicolasT]

(Based on a question sent to the Salt mailing-list by @Baitanik here)

In MetalK8s there are several SLS files starting with a #! metalk8s_kubernetes line. There's a salt/_renderers/metalk8s_kubernetes.py module, and salt/_modules/metalk8s_kubernetes.py. How do these relate, and how are Kubernetes objects defined in SLS files applied in the MetalK8s cluster?

[NicolasT]

To answer the question, we'll focus on one particular SLS file: metalk8s.addons.prometheus-operator.deployed.namespace.

This is not a 'standard' Jinja/YAML SLS template, but more alike a standard Kubernetes manifest file, listing objects in their serialized YAML form (in this case, only a single object). The shebang at the top, #! metalk8s_kubernetes, causes a custom renderer shipped with MetalK8s (though not MetalK8s-specific in any way) to be used to render the SLS. Let's see what this looks like:

# salt-call --out=yaml state.show_sls metalk8s.addons.prometheus-operator.deployed.namespace saltenv=metalk8s-2.7.0 
local:
  Apply v1/Namespace 'metalk8s-monitoring':
    __env__: metalk8s-2.7.0
    __sls__: metalk8s.addons.prometheus-operator.deployed.namespace
    metalk8s_kubernetes:
    - name: Apply v1/Namespace 'metalk8s-monitoring'
    - kubeconfig: null
    - context: null
    - manifest:
        apiVersion: v1
        kind: Namespace
        metadata:
          labels:
            app.kubernetes.io/managed-by: metalk8s
            app.kubernetes.io/part-of: metalk8s
            heritage: metalk8s
          name: metalk8s-monitoring
    - object_present
    - order: 10000

This is a state definition which uses a custom state, metalk8s_kubernetes.object_present. This is also implemented in MetalK8s (though again not MetalK8s-specific) in salt/_states/metalk8s_kubernetes.py. This is similar to the functions available in the kubernetes state as shipped with SaltStack. However, since we ran into maintenance issues with the upstream state (needing to add new executor functions, new state functions,... whenever we need to support a new object type), we decided to take a completely different approach.

Unlike the Salt kubernetes states, where there are multiple functions defined for all supported object types, and the expected input sometimes being different from the 'standard' Kubernetes object structures, we built generic state functions which can take any Kubernetes object, in their 'native' structure.

These state in turn rely on some generic functions defined in salt/_modules/metalk8s_kubernetes.py. These use runtime introspection of type information provided by the official Kubernetes Python client to validate objects (as defined in your original SLS), and to apply operations through the Kubernetes API.

In essence, the renderer is unrelated to the metalk8s_kubernetes state, which can be used in 'normal' SLS mode as well. This state uses some functions from the metalk8s_kubernetes execution module to 'get things done' (as any state implementation). The renderer simply turns a Kubernetes-native object defined as a YAML object in some SLS file into an application of the metalk8s_kubernetes.object_present state.

Hope this clarifies things a bit 😄

[TeddyAndrieux]

The implementation changed a bit in a recent PR (#3510).

Because of the python-kubernetes lib that starts to be a bit late on the latest Kubernetes versions, we are no longer able to manage all Kubernetes objects we want using models and APIs class provided by the official Python Kubernetes client, that's why we decided to use directly the DynamicClient provided by Kubernetes python-base lib.
For more information about the various options we got and the reason why we choose this one take a look at the design about interaction with Kubernetes from Salt.

This new rework makes the Salt execution module really simpler and flexible compare to the old approach, as we manage all kinds of Kubernetes objects by design.

The usage of this execution module did not change we have the exact same functions as before it's only the way to interact with Kubernetes that changed. The renderer and salt states work the same way as @NicolasT described above.

Feel free to take a look at the Salt execution module implementation and post a question, if any 😃