Fix security-context for apiserver - audit logs are supported only in Enterprise version #2906

mihivagyok · 2023-09-29T08:32:47Z

Description

kind/bug

It fixes the case when the operator installs apiserver OSS version. In case of OSS Calico, it shall run as non-root as audit logs are only supported in Enterprise version.
The change makes apiserver deployment more secure (least privilege).

For PR author

Tests for change.
If changing pkg/apis/, run make gen-files
If changing versions, run make gen-versions

For PR reviewers

A note for code reviewers - all pull requests must have the following:

Milestone set according to targeted release.
Appropriate labels:
- kind/bug if this is a bugfix.
- kind/enhancement if this is a a new feature.
- enterprise if this PR applies to Calico Enterprise only.

CLAassistant · 2023-09-29T08:32:52Z

All committers have signed the CLA.

tmjd · 2024-02-06T14:07:27Z

Have you tested this out in a real system (I'm guessing so because of your other PRs)?
I'm asking because I thought the User and Group IDs had to match the one built into the image but I am by no means confident in that thought. So if you've tried it out and it works then I'm obviously wrong.

pkg/render/apiserver.go

…, it shall run as non-root Signed-off-by: Adam Mihelcsik <18672841+mihivagyok@users.noreply.github.com>

mihivagyok · 2024-02-06T19:00:46Z

@tmjd I will get back to you with some test result tomorrow! Thanks!

tmjd

LGTM

tmjd · 2024-02-13T14:55:52Z

/sem-approve

mihivagyok · 2024-02-16T17:50:48Z

E0216 17:42:12.047708       1 run_server.go:77] open /tmp/ready: permission denied
F0216 17:42:12.047724       1 hooks.go:203] PostStartHook "apiserver-autoregistration" failed: open /tmp/ready: permission denied

Seems it is not really working. Try to figure out why:
https://github.com/projectcalico/calico/blob/release-v3.27/apiserver/cmd/apiserver/server/run_server.go#L75

hjiawei · 2024-02-16T18:34:27Z

E0216 17:42:12.047708       1 run_server.go:77] open /tmp/ready: permission denied
F0216 17:42:12.047724       1 hooks.go:203] PostStartHook "apiserver-autoregistration" failed: open /tmp/ready: permission denied
Seems it is not really working. Try to figure out why: https://github.com/projectcalico/calico/blob/release-v3.27/apiserver/cmd/apiserver/server/run_server.go#L75

That permission error could be caused by https://github.com/projectcalico/calico/blob/release-v3.27/apiserver/docker-image/Dockerfile.amd64#L11 so the recreated /tmp folder is 0755. Non-root apiserver process won't be able to write to that directory.

The /tmp/ready file is removed and /tmp folder is reworked from calico/base. However, these are only in the coming Calico v3.28. I believe this change will likely to work with latest Calico master apiserver.

mihivagyok · 2024-02-16T19:09:59Z

@hjiawei Thanks for pointing out!
On the master branch, the Dockerfile is quite different:
https://github.com/projectcalico/calico/blob/master/apiserver/Dockerfile

Also, I don't really understand the comment here. https://github.com/projectcalico/calico/blob/release-v3.27/apiserver/docker-image/Dockerfile.amd64#L8 For me, it seems it does nothing special with /code.

Thanks!

hjiawei · 2024-02-20T07:25:22Z

Also, I don't really understand the comment here. https://github.com/projectcalico/calico/blob/release-v3.27/apiserver/docker-image/Dockerfile.amd64#L8 For me, it seems it does nothing special with /code.

Yeah. You are right about this @mihivagyok . It is related to /tmp folder permission and not /code. We reworked Dockerfiles on the main branch so it is different / unified for multi-arch.

pkg/render/apiserver.go

mihivagyok · 2024-02-20T16:41:51Z

With latest changes:

deployment:

➜ kubectl get deployment -n calico-apiserver calico-apiserver -o yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
  creationTimestamp: "2024-02-20T16:35:18Z"
  generation: 1
  labels:
    apiserver: "true"
    app.kubernetes.io/name: calico-apiserver
    k8s-app: calico-apiserver
  name: calico-apiserver
  namespace: calico-apiserver
  ownerReferences:
  - apiVersion: operator.tigera.io/v1
    blockOwnerDeletion: true
    controller: true
    kind: APIServer
    name: default
    uid: c580d54c-30d8-449b-a0bf-5205c90296b2
  resourceVersion: "400266"
  uid: 87600bdc-7bf8-4f50-9c54-d93306caaefc
spec:
  progressDeadlineSeconds: 600
  replicas: 2
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      apiserver: "true"
  strategy:
    type: Recreate
  template:
    metadata:
      annotations:
        tigera-operator.hash.operator.tigera.io/calico-apiserver-certs: 5f64c431bd397c8b87c8ebcc7721ac1c9f7206dc
      creationTimestamp: null
      labels:
        apiserver: "true"
        app.kubernetes.io/name: calico-apiserver
        k8s-app: calico-apiserver
      name: calico-apiserver
      namespace: calico-apiserver
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchLabels:
                  k8s-app: calico-apiserver
              namespaces:
              - calico-apiserver
              topologyKey: kubernetes.io/hostname
            weight: 100
          - podAffinityTerm:
              labelSelector:
                matchLabels:
                  k8s-app: calico-apiserver
              namespaces:
              - calico-apiserver
              topologyKey: topology.kubernetes.io/zone
            weight: 100
      containers:
      - args:
        - --secure-port=5443
        - --tls-private-key-file=/calico-apiserver-certs/tls.key
        - --tls-cert-file=/calico-apiserver-certs/tls.crt
        env:
        - name: DATASTORE_TYPE
          value: kubernetes
        - name: KUBERNETES_SERVICE_HOST
          value: 172.21.0.1
        - name: KUBERNETES_SERVICE_PORT
          value: "443"
        - name: MULTI_INTERFACE_MODE
          value: none
        image: us.icr.io/armada-master/calico/apiserver:v3.27.0
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /version
            port: 5443
            scheme: HTTPS
          initialDelaySeconds: 90
          periodSeconds: 60
          successThreshold: 1
          timeoutSeconds: 10
        name: calico-apiserver
        readinessProbe:
          exec:
            command:
            - /code/filecheck
          failureThreshold: 5
          initialDelaySeconds: 5
          periodSeconds: 30
          successThreshold: 1
          timeoutSeconds: 10
        resources: {}
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          privileged: false
          runAsGroup: 10001
          runAsNonRoot: true
          runAsUser: 10001
          seccompProfile:
            type: RuntimeDefault
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /calico-apiserver-certs
          name: calico-apiserver-certs
          readOnly: true
        - mountPath: /tmp
          name: cache-volume
      dnsPolicy: ClusterFirst
      nodeSelector:
        kubernetes.io/os: linux
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: calico-apiserver
      serviceAccountName: calico-apiserver
      terminationGracePeriodSeconds: 30
      tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/master
      - effect: NoSchedule
        key: node-role.kubernetes.io/control-plane
      volumes:
      - name: calico-apiserver-certs
        secret:
          defaultMode: 420
          secretName: calico-apiserver-certs
      - emptyDir: {}
        name: cache-volume
status:
  availableReplicas: 2
  conditions:
  - lastTransitionTime: "2024-02-20T16:35:50Z"
    lastUpdateTime: "2024-02-20T16:35:50Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  - lastTransitionTime: "2024-02-20T16:35:19Z"
    lastUpdateTime: "2024-02-20T16:35:50Z"
    message: ReplicaSet "calico-apiserver-58fb6b4bd7" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  observedGeneration: 1
  readyReplicas: 2
  replicas: 2
  updatedReplicas: 2

status

➜ kubectl get pods -n calico-apiserver
NAME                                READY   STATUS    RESTARTS   AGE
calico-apiserver-58fb6b4bd7-66crp   1/1     Running   0          3m28s
calico-apiserver-58fb6b4bd7-w8ls4   1/1     Running   0          3m28s

➜ kubectl get tigerastatus apiserver
NAME        AVAILABLE   PROGRESSING   DEGRADED   SINCE
apiserver   True        False         False      2m58s

This change backports changes from [1] for `calico/apiserver` only to fix the `/tmp` folder permission. Currently, when `/tmp` folder is copied form ubi stage to scratch, the folder permission is 0755 so it isn't writable for non-root processes. With the changes in [2], apiserver won't be ready because `/tmp/ready` flag can't be written. [1] #8299 [2] tigera/operator#2906

tmjd · 2024-03-29T20:14:00Z

AFAICT we're still waiting to drop the last/2nd commit that added an emptyDir volume.

mihivagyok · 2024-03-29T20:19:30Z

@tmjd Let me revert those. Sorry for the delay!

tmjd · 2024-03-29T20:27:36Z

/sem-approve

mihivagyok · 2024-03-30T17:43:54Z

All tests are passed :) Thank you!

tmjd · 2024-04-01T15:23:48Z

Thank you @mihivagyok for this PR

…-oss-openshift Fix security-context for apiserver - audit logs are supported only in Enterprise version

…penshift (#3310) Fix security-context for apiserver - audit logs are supported only in Enterprise version Co-authored-by: Erik Stidham <erik@tigera.io>

mihivagyok requested a review from a team as a code owner September 29, 2023 08:32

marvin-tigera added this to the v1.32.0 milestone Sep 29, 2023

marvin-tigera added docs-pr-required release-note-required labels Sep 29, 2023

mihivagyok changed the title ~~Fix security-context - audit logs are supported only in Enterprise version~~ Fix security-context for apiserver - audit logs are supported only in Enterprise version Sep 29, 2023

mihivagyok force-pushed the run-apiserver-non-root-oss-openshift branch 2 times, most recently from 7e081ef to 6427e23 Compare September 29, 2023 09:09

danudey modified the milestones: v1.32.0, v1.32.1, v1.32.2 Dec 2, 2023

danudey modified the milestones: v1.32.2, v1.32.3 Dec 15, 2023

danudey modified the milestones: v1.32.3, v1.32.4 Jan 18, 2024

tmjd reviewed Feb 6, 2024

View reviewed changes

pkg/render/apiserver.go Outdated Show resolved Hide resolved

audit logs are supported only in Enterprise version - in case of OSS…

f962a05

…, it shall run as non-root Signed-off-by: Adam Mihelcsik <18672841+mihivagyok@users.noreply.github.com>

mihivagyok force-pushed the run-apiserver-non-root-oss-openshift branch from 6427e23 to f962a05 Compare February 6, 2024 18:59

mihivagyok requested review from tmjd and hjiawei February 6, 2024 18:59

tmjd approved these changes Feb 13, 2024

View reviewed changes

radTuti modified the milestones: v1.32.4, v1.32.5 Feb 16, 2024

radTuti modified the milestones: v1.32.5, v1.32.6 Feb 16, 2024

hjiawei reviewed Feb 20, 2024

View reviewed changes

pkg/render/apiserver.go Show resolved Hide resolved

hjiawei mentioned this pull request Feb 28, 2024

[release-v3.27] Fix calico/apiserver /tmp folder permission projectcalico/calico#8570

Merged

3 tasks

hjiawei mentioned this pull request Feb 29, 2024

Run calico/apiserver as non-root by default projectcalico/calico#8576

Merged

3 tasks

radTuti modified the milestones: v1.32.6, v1.32.7 Mar 25, 2024

mihivagyok force-pushed the run-apiserver-non-root-oss-openshift branch from 6c3ecea to f962a05 Compare March 29, 2024 20:22

tmjd merged commit 21356ae into tigera:master Apr 1, 2024
7 checks passed

mihivagyok pushed a commit to mihivagyok/operator that referenced this pull request Apr 12, 2024

Merge pull request tigera#2906 from mihivagyok/run-apiserver-non-root…

c6a1b75

…-oss-openshift Fix security-context for apiserver - audit logs are supported only in Enterprise version

mihivagyok mentioned this pull request Apr 12, 2024

[release-v1.32][cherry-pick] Fix security-context for apiserver - audit logs are supported only in Enterprise version #3308

Merged

5 tasks

mihivagyok pushed a commit to mihivagyok/operator that referenced this pull request Apr 12, 2024

Merge pull request tigera#2906 from mihivagyok/run-apiserver-non-root…

2193ab1

…-oss-openshift Fix security-context for apiserver - audit logs are supported only in Enterprise version

mihivagyok mentioned this pull request Apr 12, 2024

[release-v1.33][cherry-pick] Fix security-context for apiserver - audit logs are supported only in Enterprise version #3309

Merged

5 tasks

mihivagyok pushed a commit to mihivagyok/operator that referenced this pull request Apr 12, 2024

Merge pull request tigera#2906 from mihivagyok/run-apiserver-non-root…

8a991d1

…-oss-openshift Fix security-context for apiserver - audit logs are supported only in Enterprise version

mihivagyok mentioned this pull request Apr 12, 2024

[release-v1.34][cherry-pick] Fix security-context for apiserver - audit logs are supported only in Enterprise version #3310

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix security-context for apiserver - audit logs are supported only in Enterprise version #2906

Fix security-context for apiserver - audit logs are supported only in Enterprise version #2906

mihivagyok commented Sep 29, 2023 •

edited

Loading

CLAassistant commented Sep 29, 2023 •

edited

Loading

tmjd commented Feb 6, 2024

mihivagyok commented Feb 6, 2024

tmjd left a comment

tmjd commented Feb 13, 2024

mihivagyok commented Feb 16, 2024

hjiawei commented Feb 16, 2024 •

edited

Loading

mihivagyok commented Feb 16, 2024 •

edited

Loading

hjiawei commented Feb 20, 2024 •

edited

Loading

mihivagyok commented Feb 20, 2024

tmjd commented Mar 29, 2024

mihivagyok commented Mar 29, 2024

tmjd commented Mar 29, 2024

mihivagyok commented Mar 30, 2024

tmjd commented Apr 1, 2024

Fix security-context for apiserver - audit logs are supported only in Enterprise version #2906

Fix security-context for apiserver - audit logs are supported only in Enterprise version #2906

Conversation

mihivagyok commented Sep 29, 2023 • edited Loading

Description

For PR author

For PR reviewers

CLAassistant commented Sep 29, 2023 • edited Loading

tmjd commented Feb 6, 2024

mihivagyok commented Feb 6, 2024

tmjd left a comment

Choose a reason for hiding this comment

tmjd commented Feb 13, 2024

mihivagyok commented Feb 16, 2024

hjiawei commented Feb 16, 2024 • edited Loading

mihivagyok commented Feb 16, 2024 • edited Loading

hjiawei commented Feb 20, 2024 • edited Loading

mihivagyok commented Feb 20, 2024

tmjd commented Mar 29, 2024

mihivagyok commented Mar 29, 2024

tmjd commented Mar 29, 2024

mihivagyok commented Mar 30, 2024

tmjd commented Apr 1, 2024

mihivagyok commented Sep 29, 2023 •

edited

Loading

CLAassistant commented Sep 29, 2023 •

edited

Loading

hjiawei commented Feb 16, 2024 •

edited

Loading

mihivagyok commented Feb 16, 2024 •

edited

Loading

hjiawei commented Feb 20, 2024 •

edited

Loading