plugins: add sgx-epc plugin.

[klihub]

plugins: add sgx-epc plugin.

Add a plugin for limiting SGX encrypted page cache usage with
pod annotations. Note that the plugin requires a patched cgroup
v2 misc controller with support for something like
echo 'sgx_epc 65536' > /sys/fs/cgroup/$CGRP/misc.max
to work.

[klihub]

@mythi Here is the sgx-config. It is the original NRI PoC sample plugin. The code is updated for latest NRI main HEAD, retrofitted to this repo, and extended with the necessary bits for image building and a kustomizable Helm chart for easier deployment. PTAL.

[askervin]

This is as clean as it gets with the current stuff.

Maybe the name of the plugin could be just nri-sgx, if we can see this plugin to be extended to handle other SGX properties later on. Or nri-sgx-epc it's going to be EPC and nothing else.

The current project structure makes you write three daemonset specs for a single plugin. I guess we could make this somehow simpler, but did not get any good ideas when writing memory plugins. Yet another layer of templating on top of templating did not feel good idea at that point, but this repetition with slight differences is not very nice either.

[klihub]

The current project structure makes you write three daemonset specs for a single plugin. I guess we could make this somehow simpler, but did not get any good ideas when writing memory plugins. Yet another layer of templating on top of templating did not feel good idea at that point, but this repetition with slight differences is not very nice either.

@askervin I think we should be able to get this down to one per plugin, the one in the helm chart templates, with relative ease. We have now extra copies to be able to wire up plugins differently for e2e tests which tend to use config files and not ConfigMaps. I think we should (be able to) change this once we reworked the configuration handling bits (IOW, manage to iron out the worst monkey puke from #164 and add the missing bits for e2e test there). That will get us rid of the fallback configuration file altogether.

At that point it would be nice to use helm and a local unpacked helm chart directory for e2e tests, too (and make sure that all the necessary {k,c}ustomizability and other templating is correctly in place to support the e2e test cases).

[klihub]

I think the SGX EPC NRI plugin should also have an UpdateContainer method (similar to the already implemented CreateContainer method).

The use case for this is that a user defines an EPC limit for a pod after it has already been created. Or a user updates a defined limit because it was too low/high.

So does that really work already without recreating the whole pod and container ? With in-place vertical pod scaling ?

[mythi]

I think the SGX EPC NRI plugin should also have an UpdateContainer method (similar to the already implemented CreateContainer method).
The use case for this is that a user defines an EPC limit for a pod after it has already been created. Or a user updates a defined limit because it was too low/high.

So does that really work already without recreating the whole pod and container ? With in-place vertical pod scaling ?

To me it looks this is not possible right now. I tried patching sgx.intel.com/epc and I get an error:

* spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`,`spec.initContainers[*].image`,`spec.activeDeadlineSeconds`,`spec.tolerations` (only additions to existing tolerations),`spec.terminationGracePeriodSeconds` (allow it to be set to 1 if it was previously negative)

[ffuerste]

I think the SGX EPC NRI plugin should also have an UpdateContainer method (similar to the already implemented CreateContainer method).
The use case for this is that a user defines an EPC limit for a pod after it has already been created. Or a user updates a defined limit because it was too low/high.

So does that really work already without recreating the whole pod and container ? With in-place vertical pod scaling ?

After thinking it completely through, I don't think so. If the annotations change on the pod level (e.g. in .spec.template.metadata.annotations for a Deployment) then Kubernetes is creating a new ReplicaSet. I forgot that when I was writing, sorry. So please ignore my comment. I also deleted it.

[mythi]

@ffuerste I think that would be a reasonable feature for the future if we can get rid of the annotations and in-place autoscaling can be made to work etc but for know I'd keep the focus on the CreateContainer flows

[askervin]

LGTM.

[marquiz]

Thanks @klihub
LGTM

[askervin]

A typo... otherwise LGTM.

[askervin]

s/nri-sg-epc/nri-sgx-epc

[mythi]

users are expected to set per-container sgx.intel.com/epc limits which we'd take to epc-limit.nri.io/container.c0 annotations. Are these about pod-level misc setting (e.g., a sum of all per-container requests within the pod)?

[klihub]

No, the '/pod' (and its alternative) notation applies to any container in the pod. It's simply a shorthand notation that can be used instead of adding for every container CTR in the pod a container-specific epc-limit.nri.io/container.$CTR: $limit.

[mythi]

OK, hmm. There's probably no use for that functionality.

I could think of a case where we'd allow a policy that takes the sum of epc resource requests and adds that to a sandbox level shared resource...