K8s-athenz-syncer is a controller that synchronizes the Athenz domain data including the roles, services and policies into corresponding Kubernetes AthenzDomain custom resources.
Athenz is a generic RBAC provider for Kubernetes resource access management and Istio service-service authentication and authorization. An Athenz domain contains a set of roles and policies defined by service admins. The policies can grant or deny an Athenz role with permissions to perform specific actions on services or resources. An Athenz role can comprise of a set of principals which could represent end users or other services.
Every Kubernetes namespace is mapped to an Athenz domain and the Athenz roles and policies defined within each domain are used to express access control rules to Kubernetes resources such as deployments, services, ingresses, etc. associated with the namespace.
A kubernetes namespace is mapped to an Athenz domain with the following pattern:
| Kubernetes | Athenz |
|---|---|
| Namespace | Domain |
| {user-namespace} | {user-managed-domain} |
| {system-namespace} | {special-cluster-admin-managed-domain} |
| - dashes '-' in namespace replaced with '.' in domain | |
| - double dashes '--' in namespace replaced with '-' in domain | |
| e.g. sports-frontend | sports.frontend |
| Resources | Resources |
| - {namespace-scoped} | {user-managed-domain-prefixed} |
| e.g. sports-frontend/fantasy-prod-deployment | sports.frontend:fantasy-dashboard |
| - {cluster-scoped} | {special-admin-domain-prefixed} |
| Service | Service |
| e.g. sports-frontend/fantasy-dashboard | sports.frontend::fantasy-dashboard |
| RBAC - Role and RoleBindings | Roles and Policies |
| - {Role.rule} | {Policy.assertion} |
| - {RoleBinding.Subjects} | {Role} |
Note: Kubernetes system namespaces such as "kube-system", "istio-system" are mapped to an equivalent Athenz domain
with the format: <cluster-admin-domain>.<cluster-namespace>
While Athenz ZMS provides APIs to perform resource access checks against user/client credentials, caching the relevant policies as Kubernetes resources allows any in-cluster auth provider service to perform in-memory validation of user or client credentials without worrying about the Athenz ZMS/ZTS service availability.
K8s-athenz-syncer calls Athenz ZMS API GetSignedDomains() API to fetch the entire contents of an Athenz domain signed by the ZMS including roles, principals and policies with signatures and creates Kubernetes Custom Resources that store the domain data in the cluster so that applications can do the security checks based on local cached data.
The controller also runs a cron that periodically fetches the list of Athenz domains that were modified during the cron interval and then fetches the signed contents for each domain and stores them as the AthenzDomain Custom Resource in the cluster in order to keep all policies in local cache updated. There is also a full resync cron that adds all the watched namespaces to the controller work queue so that all of Kubernetes AthenzDomains Custom Resources are resynced after a full resync interval.
apiVersion: v1
items:
- apiVersion: athenz.io/v1
kind: AthenzDomain
metadata:
creationTimestamp: 2019-01-01T00:00:00Z
generation:
name: home.test
namespace: home-test
resourceVersion:
selfLink:
uid:
spec:
domains:
domain:
name: home.test
ypmId: 0
roles:
name: home.test:role.admin
modified: '2019-01-01T00:00:00.000Z'
members:
user.tester
roleMembers:
memberName: user.tester
policies:
contents:
domain: home.test
policies:
name: home.test:policy.admin
modified: '2019-01-01T00:00:00.000Z'
assertions:
role: home.test:role.admin
resource: home.test:*
action: "*"
effect: ALLOW
signature:
keyId: xyz
services: []
entities:
modified: '2019-01-01T00:00:00.000Z'
signature:
keyId: xyz
There are a variety of prerequisites required in order to run this controller, they are specified below.
- Kubernetes cluster - A running Kubernetes cluster is required to run the controller. More information on how to setup a cluster can be found in the official documentation here. This controller was developed and tested with the 1.13 release.
- Athenz - Athenz is required for the controller to fetch policy and domain data. More information and setup steps can be found here. The authorization management service (ZMS) and its apis are primarily used for this controller.
- SIA Provider - A service identity agent (SIA) must be running in the Kubernetes cluster in order to provision X.509 certificates to instances in order to authenticate with Athenz. The approach we currently use in production can be found here.
Configuration files which must be applied to run k8s-athenz-syncer which can be found in the k8s directory.
The Athenz Domain custom resource definition must be first created in order for the controller to sync the custom resource. Run the following command:
kubectl apply -f k8s/athenzdomain.yaml
In order to tell SIA which service to provide an X.509 certificate to, a service account must be present. This is required for the controller to authenticate with ZMS for api calls. Run the following command:
kubectl apply -f k8s/serviceaccount.yaml
or
kubectl create serviceaccount k8s-athenz-syncer
This controller requires RBAC to create, update, delete, watch and list all Athenzdomains Custom Resources in the cluster. It also has a watch and list on namespaces in order to know which domains to look up from Athenz.
NOTE: If you are deploying to a non-default namespace, make sure to update the k8s/clusterrolebinding.yaml subject namespace accordingly.
kubectl apply -f k8s/clusterrole.yaml
kubectl apply -f k8s/clusterrolebinding.yaml
The deployment for the controller contains three containers: sia init, sia refresh, and the controller itself. Build a docker image using the Dockerfile and publish to a docker registry. Make sure to replace the docker images inside of this spec to the ones which are published in your organization. Also, replace the zms url with your instance. Run the following command in order to deploy:
kubectl apply -f k8s/deployment.yaml
K8s-athenz-syncer has a variety of parameters that can be configured, they are given below.
| Parameters | Description | Default |
|---|---|---|
| admin-domain | Admin domain that can be specified in order to fetch admin domains from Athenz | |
| athenz-contact-time-cm-key | Key of ConfigMap to record the latest time that the Update Cron contacted Athenz | latest_contact |
| athenz-contact-time-cm-name | Name of ConfigMap to record the latest time that the Update Cron contacted Athenz | athenzcall-config |
| athenz-contact-time-cm-ns | Namespace of ConfigMap to record the latest time that the Update Cron contacted Athenz | kube-yahoo |
| auth-header | Authentication header field | |
| cacert | Path to X.509 ca certificate file to use for zms authentication | |
| cert | Path to X.509 certificate file to use for zms authentication | /var/run/athenz/service.cert.pem |
| disable-keep-alives | Disable keep alive for zms client | true |
| identity-key | Directory containing private keys for service identity | /var/run/keys/identity |
| inClusterConfig | Set to true to use in cluster config | true |
| key | Path to private key file for zms authentication | /var/run/athenz/service.key.pem |
| kubeconfig | Absolute path to the kubeconfig file | /root/.kube/config |
| log-location | Log location | /var/log/k8s-athenz-syncer/k8s-athenz-syncer.log |
| log-mode | Logger mode | INFO |
| ntoken-expiry | Custom nToken expiration duration | 1h0m0s |
| queue-delay-interval | Delay interval time for workqueue | 250ms |
| resync-cron | Sleep interval for controller full resync cron | 1h0m0s |
| secret-name | Secret name that contains private key | k8s-athenz-syncer |
| service-domain | Athenz domain that contains k8s-athenz-syncer | |
| service-name | Service name | k8s-athenz-syncer |
| system-namespaces | A list of cluster system namespaces that you hope the controller to fetch from Athenz | |
| update-cron | Sleep interval for controller update cron | 1m0s |
| zms-url | Athenz full zms url including api path |
Once the controller is up and running, the controller will create Kubernetes AthenzDomains Custom Resources in the cluster accordingly. Users and Applications can consume those AthenzDomains CR to get security policy information for access control checks.
- To see all the AthenzDomains CR created, run
kubectl get athenzdomains - In order to use AthenzDomains CR in applications, create AthenzDomains clientset and informers to retrieve the resources.
Please refer to the contributing file for information about how to get involved. We welcome issues, questions, and pull requests.
Core Team : omega-core@verizonmedia.com
Copyright 2019 Oath Inc. Licensed under the Apache License, Version 2.0 (the "License")