Plots: start-up latency

README.md

@@ -74,6 +74,13 @@ If your app uses Knative, you will have to install it first:
 inv knative.install
 ```
 
+## Evaluation
+
+The goal of the project is to measure the performance of Knative with CoCo,
+and compare it to other isolation mechanisms using standarised benchmarks. To
+This extent, we provide a thorough evaluation in the [evaluation](./eval)
+directory.
+
 ## Uninstall
 
 In order to uninstall components for debugging purposes, you may un-install the CoCo runtime, and then the operator as follows:

apps/helloworld-py/deployment.yaml

@@ -15,10 +15,10 @@ spec:
         apps.coco-serverless/name: helloworld-py
         io.katacontainers.config.pre_attestation.enabled: "false"
     spec:
-      runtimeClassName: kata-qemu-sev
+      runtimeClassName: kata-qemu
       containers:
       - name: helloworld-py
-        image: csegarragonz/coco-helloworld-py:latest
-          imagePullPolicy: Always
+        image: ghcr.io/csegarragonz/coco-helloworld-py:unencrypted
+        imagePullPolicy: Always
         ports:
         - containerPort: 8080

eval/README.md

@@ -0,0 +1,71 @@
+# Evaluation
+
+This directory summarizes the different evaluation efforts to measure the
+performance of Knative using confidential containers, and place it in relation
+to well-known serverless benchmarks.
+
+The evaluation of the project is divided in two parts:
+* [Performance Measurements](#performance-measurements) - performance (overheads) of Knative on CoCo.
+* [Benchmarks](#benchmarks) - evaluating Knative + CoCo on standarised benchmarks.
+
+In general, we compare Knative running on regular containers, on VMs (aka
+Knative + Kata) and with Knative + CoCo with different levels of security: (i)
+no attestaion, (ii) only guest FW attestation, (ii) image signature, and (iii)
+image signature + encryption.
+
+## Performance Measurements
+
+In order to execute any of the performance measurement experiments, it is
+expected that you have a functional system as described in the [Quick Start](
+https://github.com/csegarragonz/coco-serverless#quick-start) guide.
+
+Then, start the KBS:
+
+```bash
+inv kbs.start
+
+# If the KBS is already running, clear the DB contents
+inv kbs.clear-db
+```
+
+you must also sign and encrypt all the images used in the performance tests.
+Signing and encryption is an interactive process, hence why we do it once,
+in advance of the evaluation:
+
+```bash
+# First encrypt (and sign) the image
+inv skopeo.encrypt-container-image "ghcr.io/csegarragonz/coco-helloworld-py:unencrypted" --sign
+
+# Then sign the unencrypted images used
+inv cosign.sign-container-image "ghcr.io/csegarragonz/coco-helloworld-py:unencrypted"
+inv cosign.sign-container-image "ghcr.io/csegarragonz/coco-knative-sidecar:unencrypted"
+```
+
+Now you are ready to run one of the experiments:
+* [Start-Up Costs](#start-up-costs) - time required to spin-up a Knative service.
+
+### Start-Up Costs
+
+This benchmark compares the time required to spin-up a pod as measured from
+Kubernetes. This is the higher-level (user-facing) measure we can take.
+
+In order to run the experiment, just run:
+
+```
+inv eval.startup.run
+```
+
+you may then plot the results by using:
+
+```
+inv eval.startup.plot
+```
+
+which generates a plot in [`./plots/startup/startup.png`](
+./plots/sartup/startup.png). You can also see the plot below:
+
+![plot](./plots/startup/startup.png)
+
+## Benchmarks
+
+TODO

eval/apps/startup/service.yaml.j2

@@ -0,0 +1,26 @@
+apiVersion: serving.knative.dev/v1
+kind: Service
+metadata:
+  name: helloworld-knative
+  annotations:
+    "features.knative.dev/podspec-runtimeclassname": "enabled"
+spec:
+  template:
+    metadata:
+      labels:
+        apps.coco-serverless/name: helloworld-py
+        io.katacontainers.config.pre_attestation.enabled: "false"
+    spec:
+      {% if runtime_class is defined %}
+      runtimeClassName: {{ runtime_class }}
+      # coco-knative: need to run user container as root
+      securityContext:
+        runAsUser: 1000
+      {% endif %}
+      containers:
+        - image: {{ image_repo }}/{{ image_name }}:{{ image_tag }}
+          ports:
+            - containerPort: 8080
+          env:
+            - name: TARGET
+              value: "World"

eval/results/startup/coco-fw-sig-enc_cold.csv

@@ -0,0 +1,16 @@
+Run,Event,TimeStampMs
+0,Start,1697221341.961345
+0,Initialized,1697221342.0
+0,PodScheduled,1697221342.0
+0,Ready,1697221358.0
+0,ContainersReady,1697221358.0
+1,Initialized,1697221365.0
+1,PodScheduled,1697221365.0
+1,Start,1697221365.0416353
+1,Ready,1697221382.0
+1,ContainersReady,1697221382.0
+2,Initialized,1697221388.0
+2,PodScheduled,1697221388.0
+2,Start,1697221388.1206732
+2,Ready,1697221405.0
+2,ContainersReady,1697221405.0

eval/results/startup/coco-fw-sig-enc_warm.csv

@@ -0,0 +1,16 @@
+Run,Event,TimeStampMs
+0,Start,1697221255.6528904
+0,Initialized,1697221256.0
+0,PodScheduled,1697221256.0
+0,Ready,1697221272.0
+0,ContainersReady,1697221272.0
+1,Start,1697221278.7323177
+1,Initialized,1697221279.0
+1,PodScheduled,1697221279.0
+1,Ready,1697221295.0
+1,ContainersReady,1697221295.0
+2,Start,1697221301.8187425
+2,Initialized,1697221302.0
+2,PodScheduled,1697221302.0
+2,Ready,1697221336.0
+2,ContainersReady,1697221336.0

eval/results/startup/coco-fw-sig_cold.csv

@@ -0,0 +1,16 @@
+Run,Event,TimeStampMs
+0,Initialized,1697221114.0
+0,PodScheduled,1697221114.0
+0,Start,1697221114.1425405
+0,Ready,1697221131.0
+0,ContainersReady,1697221131.0
+1,Initialized,1697221137.0
+1,PodScheduled,1697221137.0
+1,Start,1697221137.219481
+1,Ready,1697221154.0
+1,ContainersReady,1697221154.0
+2,Initialized,1697221160.0
+2,PodScheduled,1697221160.0
+2,Start,1697221160.308928
+2,Ready,1697221177.0
+2,ContainersReady,1697221177.0

eval/results/startup/coco-fw-sig_warm.csv

@@ -0,0 +1,16 @@
+Run,Event,TimeStampMs
+0,Start,1697221046.878591
+0,Initialized,1697221047.0
+0,PodScheduled,1697221047.0
+0,Ready,1697221063.0
+0,ContainersReady,1697221063.0
+1,Start,1697221069.967245
+1,Initialized,1697221070.0
+1,PodScheduled,1697221070.0
+1,Ready,1697221085.0
+1,ContainersReady,1697221085.0
+2,Initialized,1697221091.0
+2,PodScheduled,1697221091.0
+2,Start,1697221091.0533175
+2,Ready,1697221108.0
+2,ContainersReady,1697221108.0

eval/results/startup/coco-fw_cold.csv

@@ -0,0 +1,16 @@
+Run,Event,TimeStampMs
+0,Start,1697220961.5506494
+0,Initialized,1697220962.0
+0,PodScheduled,1697220962.0
+0,Ready,1697220975.0
+0,ContainersReady,1697220975.0
+1,Start,1697220981.6300936
+1,Initialized,1697220982.0
+1,PodScheduled,1697220982.0
+1,Ready,1697220996.0
+1,ContainersReady,1697220996.0
+2,Start,1697221002.709317
+2,Initialized,1697221003.0
+2,PodScheduled,1697221003.0
+2,Ready,1697221017.0
+2,ContainersReady,1697221017.0

eval/results/startup/coco-fw_warm.csv

@@ -0,0 +1,16 @@
+Run,Event,TimeStampMs
+0,Initialized,1697220855.0
+0,PodScheduled,1697220855.0
+0,Start,1697220855.17601
+0,Ready,1697220869.0
+0,ContainersReady,1697220869.0
+1,Initialized,1697220875.0
+1,PodScheduled,1697220875.0
+1,Start,1697220875.2426844
+1,Ready,1697220935.0
+1,ContainersReady,1697220935.0
+2,Initialized,1697220940.0
+2,PodScheduled,1697220940.0
+2,Start,1697220940.4730253
+2,Ready,1697220955.0
+2,ContainersReady,1697220955.0

eval/results/startup/coco_cold.csv

@@ -0,0 +1,16 @@
+Run,Event,TimeStampMs
+0,Start,1697220756.8188488
+0,Initialized,1697220757.0
+0,PodScheduled,1697220757.0
+0,Ready,1697220770.0
+0,ContainersReady,1697220770.0
+1,Start,1697220776.8993905
+1,Initialized,1697220777.0
+1,PodScheduled,1697220777.0
+1,Ready,1697220791.0
+1,ContainersReady,1697220791.0
+2,Start,1697220796.9742057
+2,Initialized,1697220797.0
+2,PodScheduled,1697220797.0
+2,Ready,1697220811.0
+2,ContainersReady,1697220811.0

eval/results/startup/coco_warm.csv

@@ -0,0 +1,16 @@
+Run,Event,TimeStampMs
+0,Start,1697220694.552954
+0,Initialized,1697220695.0
+0,PodScheduled,1697220695.0
+0,Ready,1697220708.0
+0,ContainersReady,1697220708.0
+1,Start,1697220714.630086
+1,Initialized,1697220715.0
+1,PodScheduled,1697220715.0
+1,Ready,1697220729.0
+1,ContainersReady,1697220729.0
+2,Start,1697220735.7185054
+2,Initialized,1697220736.0
+2,PodScheduled,1697220736.0
+2,Ready,1697220750.0
+2,ContainersReady,1697220750.0

eval/results/startup/docker_cold.csv

@@ -0,0 +1,16 @@
+Run,Event,TimeStampMs
+0,Initialized,1697220557.0
+0,PodScheduled,1697220557.0
+0,Start,1697220557.0543413
+0,Ready,1697220559.0
+0,ContainersReady,1697220559.0
+1,Initialized,1697220593.0
+1,PodScheduled,1697220593.0
+1,Start,1697220593.3947294
+1,Ready,1697220599.0
+1,ContainersReady,1697220599.0
+2,Start,1697220633.5336018
+2,Initialized,1697220634.0
+2,PodScheduled,1697220634.0
+2,Ready,1697220640.0
+2,ContainersReady,1697220640.0

eval/results/startup/docker_warm.csv

@@ -0,0 +1,16 @@
+Run,Event,TimeStampMs
+0,Start,1697220448.6815393
+0,Initialized,1697220449.0
+0,PodScheduled,1697220449.0
+0,Ready,1697220450.0
+0,ContainersReady,1697220450.0
+1,Start,1697220484.8129041
+1,Initialized,1697220485.0
+1,PodScheduled,1697220485.0
+1,Ready,1697220486.0
+1,ContainersReady,1697220486.0
+2,Start,1697220520.9298782
+2,Initialized,1697220521.0
+2,PodScheduled,1697220521.0
+2,Ready,1697220522.0
+2,ContainersReady,1697220522.0

requirements.txt

@@ -3,6 +3,8 @@ black>=23.9.1
 invoke>=2.1.0
 Jinja2>=3.1.2
 jinja2-ansible-filters>=1.3.2
+matplotlib>=3.8.0
+pandas>=2.1.1
 pymysql>=1.1.0
 python-language-server[all]
 sev-snp-measure>=0.0.7

tasks/__init__.py

@@ -15,6 +15,8 @@
 from . import sev
 from . import skopeo
 
+from tasks.eval import ns as eval_ns
+
 ns = Collection(
     apps,
     coco,
@@ -31,3 +33,5 @@
     sev,
     skopeo,
 )
+
+ns.add_collection(eval_ns, name="eval")

tasks/coco.py

@@ -1,66 +1,21 @@
 from invoke import task
-from os.path import join
-from tasks.util.env import KATA_CONFIG_DIR, KBS_PORT, get_kbs_url
-from tasks.util.toml import read_value_from_toml, update_toml
+from tasks.util.coco import (
+    guest_attestation as do_guest_attestation,
+    signature_verification as do_signature_verification,
+)
 
 
 @task
 def guest_attestation(ctx, mode="off"):
     """
     Toggle guest attestation for CoCo: -attestation --mode=[on,off]
     """
-    conf_file_path = join(KATA_CONFIG_DIR, "configuration-qemu-sev.toml")
-
-    # Update the pre_attestation flag
-    att_val = str(mode == "on").lower()
-    updated_toml_str = """
-    [hypervisor.qemu]
-    guest_pre_attestation = {att_val}
-    """.format(
-        att_val=att_val
-    )
-    update_toml(conf_file_path, updated_toml_str)
-
-    # We also update the KBS URI if pre_attestation is enabled
-    if mode == "on":
-        # We need to set the KBS URL to something that is reachable both from
-        # the host _and_ the guest
-        updated_toml_str = """
-        [hypervisor.qemu]
-        guest_pre_attestation_kbs_uri = "{kbs_url}:{kbs_port}"
-        """.format(
-            kbs_url=get_kbs_url(), kbs_port=KBS_PORT
-        )
-        update_toml(conf_file_path, updated_toml_str)
+    do_guest_attestation(mode)
 
 
 @task
 def signature_verification(ctx, mode="off"):
     """
     Toggle signature verification for CoCo's agent: --mode=[on,off]
     """
-    conf_file_path = join(KATA_CONFIG_DIR, "configuration-qemu-sev.toml")
-    att_val = str(mode == "on").lower()
-
-    # We need to update the kernel parameters, which is a string, so we are
-    # particularly careful
-    original_kernel_params = read_value_from_toml(
-        conf_file_path, "hypervisor.qemu.kernel_params"
-    )
-    # Whenever I learn regex, this will be less hacky
-    pattern = "enable_signature_verification="
-    value_beg = original_kernel_params.find(pattern) + len(pattern)
-    value_end = original_kernel_params.find(" ", value_beg)
-    updated_kernel_params = (
-        original_kernel_params[:value_beg]
-        + att_val
-        + original_kernel_params[value_end:]
-    )
-
-    updated_toml_str = """
-    [hypervisor.qemu]
-    kernel_params = "{updated_kernel_params}"
-    """.format(
-        updated_kernel_params=updated_kernel_params
-    )
-    update_toml(conf_file_path, updated_toml_str)
+    do_signature_verification(mode)