diff --git a/README.md b/README.md index 9585a46f..f34f28b6 100644 --- a/README.md +++ b/README.md @@ -37,7 +37,7 @@ Ever wondered how to [deploy your WebRTC infrastructure into the cloud](https://webrtchacks.com/webrtc-media-servers-in-the-cloud)? Frightened away by the complexities of Kubernetes container networking, and the surprising ways in which it may interact -with your UDP/RTP media? Tried to read through the endless stream of [Stack +with your UDP/RTP media? Read through the endless stream of [Stack Overflow](https://stackoverflow.com/search?q=kubernetes+webrtc) [questions](https://stackoverflow.com/questions/61140228/kubernetes-loadbalancer-open-a-wide-range-thousands-of-port) [asking](https://stackoverflow.com/questions/64232853/how-to-use-webrtc-with-rtcpeerconnection-on-kubernetes) @@ -88,15 +88,12 @@ raise security concerns, and come with a non-trivial price tag. The main goal of STUNner is to allow *anyone* to deploy their own WebRTC infrastructure into Kubernetes, without relying on any external service other than the cloud-provider's standard hosted -Kubernetes offering. This is achieved by STUNner acting as a gateway for ingesting WebRTC media -traffic into the Kubernetes cluster, exposing a public-facing STUN/TURN server that WebRTC clients -can connect to. - -STUNner can act as a STUN/TURN server that WebRTC clients and media servers can use as a scalable -NAT traversal facility (headless model), or it can serve as a fully-fledged ingress gateway for -clients to reach a media server deployed behind STUNner (media-plane model). This makes it possible -to deploy WebRTC application servers and media servers into ordinary Kubernetes pods, taking -advantage of Kubernetes's excellent tooling to manage, scale, monitor and troubleshoot the WebRTC +Kubernetes offering. STUNner can act as a standalone STUN/TURN server that WebRTC clients and media +servers can use as a scalable NAT traversal facility (headless model), or it can act as a gateway +for ingesting WebRTC media traffic into the Kubernetes cluster by exposing a public-facing +STUN/TURN server that WebRTC clients can connect to (media-plane model). This makes it possible to +deploy WebRTC application servers and media servers into ordinary Kubernetes pods, taking advantage +of Kubernetes's excellent tooling to manage, scale, monitor and troubleshoot the WebRTC infrastructure like any other cloud-bound workload. ![STUNner media-plane deployment architecture](./docs/img/stunner_arch.svg) @@ -130,7 +127,7 @@ way. [hacks](https://kubernetes.io/docs/concepts/configuration/overview), like privileged pods and `hostNetwork`/`hostPort` services, typically recommended as a prerequisite to containerizing your WebRTC media plane. Using STUNner a WebRTC deployment needs only two public-facing ports, one - HTTPS port for the application server and a *single* UDP port for *all* your media. + HTTPS port for signaling and a *single* UDP port for *all* your media. * **No reliance on external services for NAT traversal.** Can't afford a [hosted TURN service](https://bloggeek.me/webrtc-turn) for client-side NAT traversal? Can't get decent @@ -142,7 +139,7 @@ way. * **Easily scale your WebRTC infrastructure.** Tired of manually provisioning your WebRTC media servers? STUNner lets you deploy the entire WebRTC infrastructure into ordinary Kubernetes pods, thus [scaling the media plane](docs/SCALING.md) is as easy as issuing a `kubectl scale` - command. Even better, use the built in Kubernetes horizontal autoscaler to *automatically* resize + command. Or you can use the built in Kubernetes horizontal autoscaler to *automatically* resize your workload based on demand. * **Secure perimeter defense.** No need to open thousands of UDP/TCP ports on your media server for @@ -155,7 +152,7 @@ way. * **Simple code and extremely small size.** Written in pure Go using the battle-tested [pion/webrtc](https://github.com/pion/webrtc) framework, STUNner is just a couple of hundred lines of fully open-source code. The server is extremely lightweight: the typical STUNner - container image size is only about 5 Mbytes. + container image size is only 15 Mbytes. ## Getting Started @@ -196,16 +193,14 @@ Find out more about the charts in the [STUNner-helm repository](https://github.c ### Configuration The standard way to interact with STUNner is via the standard Kubernetes [Gateway - API](https://gateway-api.sigs.k8s.io) version - [v1alpha2](https://gateway-api.sigs.k8s.io/v1alpha2/references/spec). This is much akin to the - way you configure *all* Kubernetes workloads: specify your intents in YAML files and issue a - `kubectl apply`, and the [STUNner gateway - operator](https://github.com/l7mp/stunner-gateway-operator) will automatically reconcile the - STUNner dataplane for the new configuration. + API](https://gateway-api.sigs.k8s.io). This is much akin to the way you configure *all* + Kubernetes workloads: specify your intents in YAML files and issue a `kubectl apply`, and the + [STUNner gateway operator](https://github.com/l7mp/stunner-gateway-operator) will automatically + reconcile the STUNner dataplane for the new configuration. 1. Given a fresh STUNner install, the first step is to register STUNner with the Kubernetes Gateway API. This amounts to creating a - [GatewayClass](https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io/v1alpha2.GatewayClass), + [GatewayClass](https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io/v1beta1.GatewayClass), which serves as the [root level configuration](/docs/GATEWAY.md#gatewayclass) for your STUNner deployment. @@ -217,7 +212,7 @@ The standard way to interact with STUNner is via the standard Kubernetes [Gatewa ``` console kubectl apply -f - < Considering the above example: even if the `default/media-plane` Service was created for the TCP:80 port, STUNner will allow connections via any protocol-port pair, say, via UDP:10000 or any other UDP port for that matter. This hack remains our only viable way to support WebRTC workloads in Kubernetes until [support for port ranges is implemented in Kubernetes services](https://github.com/kubernetes/kubernetes/issues/23864). Note that this affects only the *internal* backend services: STUNner is still exposed *externally* via a *single* protocol-port, but it can demultiplex incoming client media connections to any *internal* backend ports via a single UDPRoute. + And that's all. You don't need to worry about client-side NAT traversal and WebRTC media routing because STUNner has you covered! Even better, every time you change a Gateway API resource in @@ -372,7 +369,7 @@ STUN/TURN authentication type: plaintext STUN/TURN username: user-1 STUN/TURN password: pass-1 Listener: udp-listener -Protocol: UDP +Protocol: TURN-UDP Public address: 34.118.36.108 Public port: 3478 ``` @@ -395,10 +392,9 @@ a heartwarming welcome message. 1. Fire up the UDP greeter service. The below manifest spawns the service in the `default` namespace and wraps it in a Kubernetes - service called `media-plane`. Recall, this is the target service STUNner will route connections - to. Note that the type of the `media-plane` service is `ClusterIP`, which means that Kubernetes - will *not* expose it to the Internet: the only way for clients to obtain a response is via - STUNner. + service called `media-plane`. Recall, this is the target service in our UDPRoute. Note that the + type of the `media-plane` service is `ClusterIP`, which means that Kubernetes will *not* expose + it to the Internet: the only way for clients to obtain a response is via STUNner. ```console kubectl apply -f deploy/manifests/udp-greeter.yaml @@ -454,11 +450,11 @@ greeter) by STUNner. ``` 1. Add the new TLS Gateway. Notice how the `tls-listener` now contains a `tls` object that refers - the above Secret, this way assigning the TLS certificate to use with our TLS listener. + the above Secret, this way assigning the TLS certificate to use with our TURN-TLS listener. ```console kubectl apply -f - < var pc = new RTCPeerConnection(iceConfig); ``` @@ -96,9 +92,10 @@ The intended authentication workflow in STUNner is as follows. ## Static authentication In STUNner, `static` authentication is the simplest and least secure authentication mode, basically -corresponding to a traditional "log-in" username and password pair given to users. STUNner accepts -(and sometimes reports) the alias `plaintext` to mean the `static` authentication mode; the use of -`plaintext` is deprecated and will be removed in a later release. +corresponding to a traditional "log-in" username and password pair given to users. + +> **Note** +STUNner accepts (and sometimes reports) the alias `plaintext` to mean the `static` authentication mode; the use of `plaintext` is deprecated and will be removed in a later release. When STUNner is configured to use `static` authentication only a single username/password pair is used for *all* clients. This makes configuration easy; e.g., the ICE server configuration can be @@ -109,7 +106,7 @@ credentials, see below). The first step of configuring STUNner for the `static` authentication mode is to create a Kubernetes Secret to hold the username/password pair. The below will set the username to `my-user` -and the password to `my-password`. Note that if no `type` is set then STUNner defaults to `static` +and the password to `my-password`. If no `type` is set then STUNner defaults to `static` authentication. ```console @@ -146,9 +143,8 @@ kubectl -n stunner edit secret stunner-auth-secret > **Warning** Modifying STUNner's credentials goes *without* restarting the TURN server but may affect existing -sessions, in that existing sessions will not be able to refresh the active TURN allocation with the -old credentials. The application server may also need to be restarted to learn the new TURN -credentials. +sessions, in that active sessions will not be able to refresh the TURN allocation established with +the old credentials. ## Ephemeral authentication @@ -158,10 +154,12 @@ with a pre-configured lifetime and, once the lifetime expires, the credential ca authenticate (or refresh) with STUNner any more. This authentication mode is more secure since credentials are not shared between clients and come with a limited lifetime. Configuring `ephemeral` authentication may be more complex though, since credentials must be dynamically -generated for each session and properly returned to clients. STUNner accepts (and sometimes -reports) the alias `longterm` to mean the `ephemeral` authentication mode; the use of `longterm` is -deprecated and will be removed in a later release. Note also that the alias `timewindowed` is also -accepted. +generated for each session and properly returned to clients. + +> **Note** +STUNner accepts (and sometimes reports) the alias `longterm` to mean the `ephemeral` authentication +mode; the use of `longterm` is deprecated and will be removed in a later release. The alias +`timewindowed` is also accepted. To implement this mode, STUNner adopts the [quasi-standard time-windowed TURN authentication credential format](https://datatracker.ietf.org/doc/html/draft-uberti-behave-turn-rest-00). In this @@ -178,8 +176,8 @@ The advantage of this mechanism is that it is enough to know the shared secret f able to check the validity of a credential. > **Warning** -The user-id is used only for the integrity check but STUNner in no way checks whether it identifies -a valid user-id in the system. +The user-id is to ensure that the password generated per user-id is unique, but STUNner in no way +checks whether it identifies a valid user-id in the system. In order to switch from `static` mode to `ephemeral` authentication, it is enough to update the Secret that holds the credentials. The below will set the shared secret `my-shared-secret` for the diff --git a/docs/CONCEPTS.md b/docs/CONCEPTS.md index f09412fa..7fb4f26f 100644 --- a/docs/CONCEPTS.md +++ b/docs/CONCEPTS.md @@ -4,7 +4,7 @@ In this guide we describe STUNner's architecture and the most important componen ## Architecture -A STUNner installation consists of two parts, a *control plane* and a *dataplane*. The control plane consists of declarative policies specifying the way STUNner should route WebRTC media traffic to the media servers, plus a gateway operator that renders the high-level policies into an actual dataplane configuration. The dataplane in turn comprises one or more `stunnerd` pods, responsible for actually ingesting media traffic into the cluster through a STUN/TURN server. Since the TURN service underlying STUNner is agnostic to NATs, STUNner can inject clients' media traffic into the private Kubernetes pod network, addressing all NAT traversal steps (client-side and server-side) in a single go. +A STUNner installation consists of two parts, a *control plane* and a *dataplane*. The control plane consists of declarative policies specifying the way STUNner should route WebRTC media traffic to the media servers, plus a gateway operator that renders the high-level policies into an actual dataplane configuration. The dataplane in turn comprises one or more `stunnerd` pods, responsible for actually ingesting media traffic into the cluster through a STUN/TURN server. ![STUNner architecture](img/stunner_arch_big.svg) diff --git a/docs/DEPLOYMENT.md b/docs/DEPLOYMENT.md index 66b9b808..3e0dd317 100644 --- a/docs/DEPLOYMENT.md +++ b/docs/DEPLOYMENT.md @@ -1,14 +1,14 @@ # Deployment models -STUNner can be deployed in many combinations to support a wide range of operational +STUNner can be deployed in many different ways, supporting a wide range of operational requirements. First, it supports multiple [architectural models](#architectural-models) where it can act either as a simple headless STUN/TURN server or a fully fledged ingress gateway in front of an entire Kubernetes-based media server pool. Second, when STUNner is configured as an ingress gateway then there are multiple [ICE models](#ice-models), based on whether only the client connects via STUNner or both clients and media servers use STUNner to set up the media-plane -connection. Third, STUNner can run in one of two [control plane models](#control-plane-models), +connection. Third, STUNner can run in one of several [control plane models](#control-plane-models), based on whether the user manually supplies STUNner configuration or there is a separate STUNner -control plane that automatically reconciles the dataplane state based on a high-level [declarative +control plane that automatically reconciles the dataplane based on a high-level [declarative API](https://gateway-api.sigs.k8s.io). ## Architectural models @@ -84,9 +84,8 @@ are as below: > - on the client, set STUNner as the *only* TURN server and configure *no* STUN servers, whereas > - on the server do *not* configure *any* STUN or TURN servers whatsoever. -Most users will want to deploy STUNner using the asymmetric ICE mode. In the rest of the docs, -unless noted otherwise we will assume the asymmetric ICE mode with the media plane deployment -model. +Most users will want to deploy STUNner using the asymmetric ICE mode. In the rest of the docs we +assume the asymmetric ICE mode with the media plane deployment model, unless noted otherwise. > **Warning** Deviating from the above rules *might* work in certain cases, but may have uncanny and @@ -130,13 +129,18 @@ you from adopting STUNner. ## Control plane models -STUNner can run in one of two modes: in the default mode STUNner configuration is controlled by a -*gateway-operator* component based on high-level intent encoded in [Kubernetes Gateway API -resources](https://gateway-api.sigs.k8s.io), while in the *standalone model* the user configures -STUNner manually. The standalone mode provides perfect control over the way STUNner ingests media, -but at the same time it requires users to deal with the subtleties of internal STUNner APIs that -are subject to change between subsequent releases. As of v0.14, STUNner's operator-ful mode is -feature complete and the standalone model is considered obsolete. If still interested, -comprehensive documentation for the standalone can be found [here](OBSOLETE.md), but this mode -is no longer supported. - +STUNner can run in one of several modes. + +In the default mode STUNner configuration is controlled by a *gateway-operator* component based on +high-level intent encoded in [Kubernetes Gateway API resources](https://gateway-api.sigs.k8s.io), +while in the *standalone model* the user configures STUNner manually. The standalone mode provides +perfect control over the way STUNner ingests media, but at the same time it requires users to deal +with the subtleties of internal STUNner APIs that are subject to change between subsequent +releases. As of v0.16, STUNner's operator-ful mode is feature complete and the standalone model is +considered obsolete. If still interested, comprehensive documentation for the standalone can be +found [here](OBSOLETE.md), but this mode is no longer supported. + +In addition, STUNner supports two dataplane provisioning modes. In the *legacy* mode the dataplane +is supposed to be deployed by the user manually (by installing the `stunner/stunner` Helm chart +into the target namespaces) while in the *managed* mode the dataplane pods are provisioned by the +gateway operator automatically. As of STUNner v0.16.0, the default is the *legacy* dataplane mode. diff --git a/docs/GATEWAY.md b/docs/GATEWAY.md index c2779fa0..16742336 100644 --- a/docs/GATEWAY.md +++ b/docs/GATEWAY.md @@ -8,7 +8,7 @@ The main unit of the control plane configuration is the *gateway hierarchy*. Her ![Gateway hierarchy](img/gateway_api.svg) -In general, the scope of a gateway hierarchy is a single namespace, but this is not strictly enforced: e.g., the GatewayClass is [cluster-scoped](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions) so it is outside the namespace, GatewayClasses can refer to GatewayConfigs across namespaces, Routes can attach to Gateways across a namespace boundary (if the Gateway allows this), etc. Still, it is a good practice to keep all control plane configuration, plus the actual dataplane pods, in a single namespace as much as possible. +In general, the scope of a gateway hierarchy is a single namespace, but this is not strictly enforced: e.g., the GatewayClass is [cluster-scoped](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions) so it is outside the namespace, GatewayClasses can refer to GatewayConfigs across namespaces, Routes can attach to Gateways across a namespace boundary (if the Gateway [allows](https://gateway-api.sigs.k8s.io/guides/multiple-ns) this), etc. Still, it is a good practice to keep all control plane configuration, plus the actual dataplane pods, in a single namespace as much as possible. ## GatewayClass @@ -17,7 +17,7 @@ The GatewayClass resource provides the root of the gateway hierarchy. GatewayCla Below is a sample GatewayClass resource. Each GatewayClass must specify a controller that will manage the Gateway objects created under the hierarchy; this must be set to `stunner.l7mp.io/gateway-operator` for the STUNner gateway operator to pick up the GatewayClass. In addition, a GatewayClass can refer to further implementation-specific configuration via a `parametersRef`; in the case of STUNner this will always be a GatewayConfig object (see [below](#gatewayconfig)). ```yaml -apiVersion: gateway.networking.k8s.io/v1alpha2 +apiVersion: gateway.networking.k8s.io/v1beta1 kind: GatewayClass metadata: name: stunner-gatewayclass @@ -31,7 +31,7 @@ spec: description: "STUNner is a WebRTC ingress gateway for Kubernetes" ``` -Below is a quick reference of the most important fields of the GatewayClass [`spec`](https://kubernetes.io/docs/concepts/overview/working-with-objects/kubernetes-objects) +Below is a quick reference of the most important fields of the GatewayClass [`spec`](https://kubernetes.io/docs/concepts/overview/working-with-objects/kubernetes-objects). | Field | Type | Description | Required | | :--- | :---: | :--- | :---: | @@ -66,7 +66,7 @@ Below is a quick reference of the most important fields of the GatewayConfig [`s | :--- | :---: | :--- | :---: | | `stunnerConfig` | `string` | The name of the ConfigMap into which the operator renders the `stunnerd` running configuration. Default: `stunnerd-config`. | No | | `logLevel` | `string` | Logging level for the dataplane daemon pods (`stunnerd`). Default: `all:INFO`. | No | -| `realm` | `string` | The STUN/TURN authentication realm to be used for clients to authenticate with STUNner. The realm must consist of lower case alphanumeric characters or `-` and `-`, and must start and end with an alphanumeric character. Default: `stunner.l7mp.io`. | No | +| `realm` | `string` | The STUN/TURN authentication realm to be used for clients to authenticate with STUNner. The realm must consist of lower case alphanumeric characters or `-` and must start and end with an alphanumeric character. Default: `stunner.l7mp.io`. | No | | `authRef` | `reference` | Reference to a Secret (`namespace` and `name`) that defines the STUN/TURN authentication mechanism and the credentials. | No | | `authType` | `string` | Type of the STUN/TURN authentication mechanism. Valid only if `authRef` is not set. Default: `static`. | No | | `username` | `string` | The username for [`static` authentication](AUTH.md). Valid only if `authRef` is not set. | No | @@ -86,10 +86,10 @@ Except the TURN authentication realm, all GatewayConfig resources are safe for m Gateways describe the STUN/TURN server listeners exposed to clients. -The below Gateway will configure STUNner to open a STUN/TURN listener on the UDP port 3478 and automatically expose it on a public IP address and port by creating a [LoadBalancer service](https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer). The name and namespace of the automatically provisioned service are the same as those of the Gateway, and the service is automatically updated if the Gateway changes (e.g., a port changes). +The below Gateway will configure STUNner to open a STUN/TURN listener over the UDP port 3478 and automatically expose it on a public IP address and port by creating a [LoadBalancer service](https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer). The name and namespace of the automatically provisioned service are the same as those of the Gateway, and the service is automatically updated if the Gateway changes (e.g., a port changes). ```yaml -apiVersion: gateway.networking.k8s.io/v1alpha2 +apiVersion: gateway.networking.k8s.io/v1beta1 kind: Gateway metadata: name: udp-gateway @@ -99,13 +99,13 @@ spec: listeners: - name: udp-listener port: 3478 - protocol: UDP + protocol: TURN-UDP ``` -The below more complex example defines two TURN listeners: a UDP listener at port 3478 that accepts routes from any namespace, and a TLS/TCP listener at port 443 that accepts routes from all namespaces labeled as `app:dev`. +The below more complex example defines two TURN listeners: a TURN listener at the UDP:3478 port that accepts routes from any namespace, and a TURN listener at port TLS/TCP:443 that accepts routes from all namespaces labeled as `app:dev`. ```yaml -apiVersion: gateway.networking.k8s.io/v1alpha2 +apiVersion: gateway.networking.k8s.io/v1beta1 kind: Gateway metadata: name: complex-gateway @@ -121,13 +121,13 @@ spec: listeners: - name: udp-listener port: 3478 - protocol: UDP + protocol: TURN-UDP allowedRoutes: namespaces: from: All - name: tls-listener port: 443 - protocol: TLS + protocol: TURN-TLS tls: mode: Terminate certificateRefs: @@ -151,37 +151,34 @@ Below is a quick reference of the most important fields of the Gateway [`spec`]( | `addresses` | `list` | The list of manually hinted external IP addresses for the rendered service (only the first one is used). | No | Each TURN `listener` is defined by a unique name, a transport protocol and a port. In addition, a -`tls` configuration is required for TLS and DTLS listeners. +`tls` configuration is required for TURN-TLS and TURN-DTLS listeners. | Field | Type | Description | Required | | :--- | :---: | :--- | :---: | | `name` | `string` | Name of the TURN listener. | Yes | | `port` | `int` | Network port for the TURN listener. | Yes | -| `protocol` | `string` | Transport protocol for the TURN listener. Either UDP, TCP, TLS or DTLS. | Yes | -| `tls` | `object` | [TLS configuration](https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io%2fv1beta1.GatewayTLSConfig).| Yes (for TLS/DTLS) | +| `protocol` | `string` | Transport protocol for the TURN listener. Either TURN-UDP, TURN-TCP, TURN-TLS or TURN-DTLS. | Yes | +| `tls` | `object` | [TLS configuration](https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io%2fv1beta1.GatewayTLSConfig).| Yes (for TURN-TLS/TURN-DTLS) | | `allowedRoutes.from` | `object` | [Route attachment policy](https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io/v1beta1.AllowedRoutes), either `All`, `Selector`, or `Same` (default is `Same`) | No | -For TLS/DTLS listeners, `tls.mode` must be set to `Terminate` or omitted (`Passthrough` does not make sense for TURN), and `tls.certificateRefs` must be a [reference to a Kubernetes Secret](https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io%2fv1beta1.GatewayTLSConfig) of type `tls` or `opaque` with exactly two keys: `tls.crt` must hold the TLS PEM certificate and `tls.key` must hold the TLS PEM key. +For TURN-TLS/TURN-DTLS listeners, `tls.mode` must be set to `Terminate` or omitted (`Passthrough` does not make sense for TURN), and `tls.certificateRefs` must be a [reference to a Kubernetes Secret](https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io%2fv1beta1.GatewayTLSConfig) of type `tls` or `opaque` with exactly two keys: `tls.crt` must hold the TLS PEM certificate and `tls.key` must hold the TLS PEM key. -STUNner will automatically generate a Kubernetes LoadBalancer service to expose each Gateway to -clients. All TURN listeners specified in the Gateway are wrapped by a single Service and will be -assigned a single externally reachable IP address. If you want multiple TURN listeners on different -public IPs, create multiple Gateways. TURN listeners on UDP and DTLS protocols are exposed as UDP -services, TCP and TLS listeners are exposed as TCP. +STUNner will automatically generate a Kubernetes LoadBalancer service to expose each Gateway to clients. All TURN listeners specified in the Gateway are wrapped by a single Service and will be assigned a single externally reachable IP address. If you want multiple TURN listeners on different public IPs, create multiple Gateways. TURN over UDP and TURN over DTLS listeners are exposed as UDP services, TURN-TCP and TURN-TLS listeners are exposed as TCP. -Manually hinted external address describes an address that can be bound to a Gateway. It is defined by an address type and an address value. Note that only the first address is used. Setting the `spec.addresses` field in the Gateway, will result in the rendered Service's [loadBalancerIP](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#service-v1-core:~:text=non%20%27LoadBalancer%27%20type.-,loadBalancerIP,-string) and [externalIPs](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#service-v1-core:~:text=and%2Dservice%2Dproxies-,externalIPs,-string%20array) fields to be set. -> **Warning** -Since Kubernetes v1.24 the `loadBalancerIP` field is deprecated, thus will be ignored if the cloud-provider or your Kubernetes install does not support the feature. Also the `externalIPs` field is denied by some cloud-providers and will fail the resource creation. Be thorough when using this feature. +Manually hinted external address describes an address that can be bound to a Gateway. It is defined by an address type and an address value. Note that only the first address is used. Setting the `spec.addresses` field in the Gateway will result in the rendered Service's [loadBalancerIP](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#service-v1-core:~:text=non%20%27LoadBalancer%27%20type.-,loadBalancerIP,-string) and [externalIPs](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#service-v1-core:~:text=and%2Dservice%2Dproxies-,externalIPs,-string%20array) fields to be set. -| Field | Type | Description | Required | -| :--- | :---: | :--- | :---: | -| `type` | `string` | Type of the address. Currently we only support IPAddress. | Yes | -| `value` | `string` | Address that should be bound to the Gateway's service. | Yes | +| Field | Type | Description | Required | +|:--------|:--------:|:--------------------------------------------------------------|:--------:| +| `type` | `string` | Type of the address. Currently only `IPAddress` is supported. | Yes | +| `value` | `string` | Address that should be bound to the Gateway's service. | Yes | + +> **Warning** +Be careful when using this feature. Since Kubernetes v1.24 the `loadBalancerIP` field is deprecated and it will be ignored if the cloud-provider or your Kubernetes install do not support the feature. In addition, the `externalIPs` field is denied by some cloud-providers. -Mixed multi-protocol Gateways are supported: this means if you want to expose a UDP and a TCP port on the same LoadBalancer service you can do it with a single Gateway. By default, the STUNner gateway-operator disables the use of mixed-protocol LBs for compatibility reasons. However, it can be enabled by annotating a Gateway with the `stunner.l7mp.io/enable-mixed-protocol-lb: true` key-value pair. The below Gateway will expose both ports with their respective protocols. +Mixed multi-protocol Gateways are supported: this means if you want to expose a UDP and a TCP port on the same LoadBalancer service you can do it with a single Gateway. The below Gateway will expose both ports with their respective protocols. ```yaml -apiVersion: gateway.networking.k8s.io/v1alpha2 +apiVersion: gateway.networking.k8s.io/v1beta1 kind: Gateway metadata: name: mixed-protocol-gateway @@ -192,20 +189,20 @@ spec: listeners: - name: udp-listener port: 3478 - protocol: UDP + protocol: TURN-UDP - name: tcp-listener port: 3479 - protocol: TCP + protocol: TURN-TCP ``` > **Warning** -> Note that the mixed-protocol LB feature might not be supported in your Kubernetes version. +> Since mixed-protocol LB support is not supported in many popular Kubernetes offerings, STUNner currently defaults to disabling this feature for compatibility reasons. You can re-enable mixed-protocol LBs by annotating your Gateway with the `stunner.l7mp.io/enable-mixed-protocol-lb: true` key-value pair. -STUNner implements two ways to customize the automatically created Service, both involving setting certain [annotations](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations) to the Service. First, if any annotation is set in the GatewayConfig `loadBalancerServiceAnnotations` object then those will be copied verbatim into the Service. Note that `loadBalancerServiceAnnotations` affect *all* LoadBalancer Services created by STUNner. Second, Service annotations can be customized on a per-Gateway basis as well by adding the annotations to Gateway resources. STUNner then copies all annotations from the Gateway verbatim into the Service, overwriting the annotations specified in the GatewayConfig on conflict. This is useful to, e.g., specify health-check settings for the Kubernetes load-balancer controller. The special annotation `stunner.l7mp.io/service-type` can be used to customize the type of the Service created by STUNner. Value can be either `ClusterIP`, `NodePort`, or `LoadBalancer` (this is the default); for instance, setting `stunner.l7mp.io/service-type: ClusterIP` will prevent STUNner from exposing a Gateway publicly (useful for testing). +STUNner implements two ways to customize the automatically created Service, both involving adding certain [annotations](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations) to the Service. First, if any annotation is set in the GatewayConfig `loadBalancerServiceAnnotations` field then those will be copied verbatim into the Service. Note that `loadBalancerServiceAnnotations` affect *all* LoadBalancer Services created by STUNner under the current Gateway hierarchy. Second, Service annotations can be customized on a per-Gateway basis as well by adding the annotations to Gateway resources. STUNner then copies all annotations from the Gateway verbatim into the Service, overwriting the annotations specified in the GatewayConfig on conflict. This is useful to, e.g., specify health-check settings for the Kubernetes load-balancer controller. The special annotation `stunner.l7mp.io/service-type` can be used to customize the type of the Service created by STUNner. The value can be either `ClusterIP`, `NodePort`, or `LoadBalancer` (this is the default); for instance, setting `stunner.l7mp.io/service-type: ClusterIP` will prevent STUNner from exposing a Gateway publicly (useful for testing). > **Warning** Gateway resources are *not* safe for modification. This means that certain changes to a Gateway will restart the underlying TURN server listener, causing all active client sessions to terminate. The particular rules are as follows: -> - adding or removing a listener will start/stop *only* the TURN server to be started/stopped, without affecting the rest of the listeners; +> - adding or removing a listener will start/stop *only* the TURN listener to be started/stopped, without affecting the rest of the listeners on the same Gateway; > - changing the transport protocol, port or TLS keys/certs of an *existing* listener will restart the TURN listener but leave the rest of the listeners intact; > - changing the TURN authentication realm will restart *all* TURN listeners. diff --git a/docs/MONITORING.md b/docs/MONITORING.md index 73ff4740..9608d94d 100644 --- a/docs/MONITORING.md +++ b/docs/MONITORING.md @@ -57,33 +57,32 @@ STUNner provides deep visibility into the amount of traffic sent and received on | :--- | :--- | :--- | :--- | | `stunner_listener_connections` | Number of *active* downstream connections at a listener. | gauge | `name=` | | `stunner_listener_connections_total` | Number of downstream connections at a listener. | counter | `name=` | -| `stunner_listener_packets_total` | Number of datagrams sent or received at a listener. Unreliable for listeners running on a connection-oriented a protocol (TCP/TLS). | counter | `direction=`, `name=`| +| `stunner_listener_packets_total` | Number of datagrams sent or received at a listener. Unreliable for listeners running on a connection-oriented transport protocol (TCP/TLS). | counter | `direction=`, `name=`| | `stunner_listener_bytes_total` | Number of bytes sent or received at a listener. | counter | `direction=`, `name=` | | `stunner_cluster_connections` | Number of *active* upstream connections on behalf of a listener. | gauge | `name=` | | `stunner_cluster_connections_total` | Number of upstream connections on behalf of a listener. | counter | `name=` | -| `stunner_cluster_packets_total` | Number of datagrams sent to backends or received from backends on behalf of a listener. Unreliable for clusters running on a connection-oriented a protocol (TCP/TLS).| counter | `direction=`, `name=` | +| `stunner_cluster_packets_total` | Number of datagrams sent to backends or received from backends on behalf of a listener. Unreliable for clusters running on a connection-oriented transport protocol (TCP/TLS).| counter | `direction=`, `name=` | | `stunner_cluster_bytes_total` | Number of bytes sent to backends or received from backends on behalf of a listener. | counter | `direction=`, `name=` | ## Integration with Prometheus and Grafana -Collection and visualization of STUNner relies on Prometheus and Grafana services. The STUNer helm repository provides a ready-to-use Prometheus and Grafana stack. See [Installation](#installation) for installation steps. Metrics visualization requires user input on configuring the plots. Refer to [Configuration and Usage](#configuration-and-usage) for details. +Collection and visualization of STUNner relies on Prometheus and Grafana services. The STUNer helm repository provides a way to [install](#installation) a ready-to-use Prometheus and Grafana stack. In addition, metrics visualization requires [user input](#configuration-and-usage) on configuring the plots; see below. ### Installation A full-fledged Prometheus+Grafana helm chart is available in the STUNner helm repo. To use this chart, the installation steps involve enabling monitoring in STUNner, and installing the Prometheus+Grafana stack with helm. -1. **Configure STUNner to expose the metrics** +1. Install STUNner with Prometheus support: -- Deploy STUNner with monitoring enabled to enable the monitoring port of STUNner pods -```console -helm install stunner stunner/stunner --create-namespace --namespace=stunner --set stunner.deployment.monitoring.enabled=true -``` + ```console + helm install stunner stunner/stunner --create-namespace --namespace=stunner --set stunner.deployment.monitoring.enabled=true + ``` -- [Expose the STUNner metrics-collection server in the GatewayConfig](#configuration) +2. Configure STUNner to expose the metrics by [exposing the STUNner metrics-collection server in the GatewayConfig](#configuration). -2. **Install the Prometheus+Grafana stack with a helm chart** +3. Install the Prometheus+Grafana stack with a helm chart. -This helm chart creates a ready-to-use Prometheus+Grafana stack in the `monitoring` namespace: installs Prometheus along with the prometheus-operator, and Grafana; configures PodMonitor for monitoring STUNner pods, and sets up Prometheus as a datasource for Grafana. + The below creates a ready-to-use Prometheus+Grafana stack in the `monitoring` namespace: Prometheus, along with the prometheus-operator, is installed for metrics scarping, Grafana is set up for visualization, and the Prometheus is configured as a datasource for Grafana. ```console helm repo add stunner https://l7mp.io/stunner @@ -92,7 +91,6 @@ helm repo update helm install prometheus stunner/stunner-prometheus ``` - ### Configuration and Usage The helm chart deploys a ready-to-use Prometheus and Grafana stack, but leaves the Grafana dashboard empty to let the user pick metrics and configure their visualization. An interactive way to visualize STUNner metrics is to use the Grafana dashboard. @@ -115,11 +113,11 @@ Click on *Add panel* (1), then *Add a new panel* (2): The *Add a new panel* will open the panel configuration. The configuration steps are the following. -1. Set the datasource: **prometheus** +1. Set the datasource: **prometheus**. 2. Choose a metric. In this example, this is the `stunner_listener_connections`. -3. Click on *Run queries* (this will update the figure) +3. Click on *Run queries* (this will update the figure). 4. Fine-tune plot parameters. For example, set the title. -5. Click *Apply* +5. Click *Apply*. ![Grafana Panel Configuration](img/grafana-add-panel-config_0.png) @@ -129,7 +127,6 @@ Below is an example dashboard with data collected from the [simple-tunnel](examp ![Grafana Dashboard with the New Panel](img/grafana-add-panel-dashboard_1.png) - ### Troubleshooting Prometheus and Grafana both provide a dashboard to troubleshoot a running system, and to check the flow of metrics from STUNner to Prometheus, and from Prometheus to Grafana. @@ -141,8 +138,8 @@ The dashboard enables checking running Prometheus configuration and testing the For example, to observe the `stunner_listener_connections` metric on the Prometheus dashboard: -1. Write `stunner_listener_connections` to the marked field (next to the looking glass icon) -2. Click on the `Execute` button +1. Write `stunner_listener_connections` to the marked field (next to the looking glass icon). +2. Click on the `Execute` button. 3. Switch to `Graph` view tab. ![Prometheus Dashboard](img/prometheus-dashboard.png) diff --git a/docs/SCALING.md b/docs/SCALING.md index c591f42f..b615c501 100644 --- a/docs/SCALING.md +++ b/docs/SCALING.md @@ -65,7 +65,7 @@ connections. As usual, however, some caveats apply: even if active allocations would last longer. You can always set this by adjusting the `terminationGracePeriod` on your `stunnerd` pods. 2. STUNner pods may remain alive well after the last client connection goes away. This occurs when - an TURN/UDP allocation is left open by a client (spontaneous UDP client-side connection closure + an TURN-UDP allocation is left open by a client (spontaneous UDP client-side connection closure cannot be reliably detected by the server). As the default TURN refresh lifetime is [10 minutes](https://www.rfc-editor.org/rfc/rfc8656#section-3.2-3), it may take 10 minutes until all allocations time out, letting `stunnerd` to finally terminate. diff --git a/docs/SECURITY.md b/docs/SECURITY.md index 776f3e97..49ba8fb0 100644 --- a/docs/SECURITY.md +++ b/docs/SECURITY.md @@ -71,7 +71,7 @@ prerequisites for this: 2. the target service *must* be wrapped with a UDPRoute 3. the attacker *must* know at least one pod address or the ClusterIP for the targeted service. -Should any of these prerequisites miss, STUNner will block access to the target service. +Should any of these prerequisites fail, STUNner will block access to the target service. Now rewrite the backend service in the UDPRoute to an arbitrary non-existent service. @@ -90,7 +90,7 @@ spec: ``` Repeat the above `dig` command to query the Kubernetes DNS service again and observe how the query -times out. This demonstrates that a properly locked down STUNner installation blocks all accesses +times out. This demonstrates that a properly locked down STUNner installation blocks all access outside of the backend services explicitly opened up via a UDPRoute. ## Locking down STUNner @@ -102,7 +102,7 @@ services exposed via STUNner. STUNner's basic security model is as follows: > In a properly configured deployment, STUNner provides the same level of security as a media -server pool exposed to the Internet over public IP addresses, protected by a firewall that admits +server pool exposed to the Internet over a public IP address, protected by a firewall that admits only UDP access. A malicious attacker, even possessing a valid TURN credential, can reach only the media servers deployed behind STUNner, but no other services. @@ -196,7 +196,4 @@ though: remember, none of these private IP addresses can be reached externally. can be further reduced to the STUNner pods' private IP addresses by using the [symmetric ICE mode](DEPLOYMENT.md#symmetric-ice-mode). -Nevertheless, if worried about information exposure then STUNner may not be the best option at the -moment. In later releases, we plan to implement a feature to obscure the transport relay connection -addresses returned by STUNner, which would lock down external scanning attempts. Feel free to open -an issue if you think this limitation is a blocker for you. +Nevertheless, if worried about information exposure then STUNner may not be the best option at the moment. In later releases, we plan to implement a feature to obscure the relay transport addresses returned by STUNner. Please file an issue if you think this limitation is a blocker for your use case. diff --git a/docs/WHY.md b/docs/WHY.md index 348a560a..ce632d1b 100644 --- a/docs/WHY.md +++ b/docs/WHY.md @@ -5,8 +5,7 @@ STUNner is a *WebRTC media gateway for Kubernetes*. All words matter here: indee encapsulations, it is a *media gateway* so its job is to ingest WebRTC audio/video streams into a virtualized media plane, and it is *opinionated towards Kubernetes*, so everything around STUNner is designed and built to fit into the Kubernetes ecosystem. That being said, STUNner can easily be -used outside of this context (e.g., as a regular STUN/TURN server), but these deployment options -are not the main focus. +used outside of this context (e.g., as a regular STUN/TURN server), but this is not the main focus. ## The problem @@ -20,7 +19,7 @@ private IP address and the network dataplane applies several rounds of Network A (NAT) steps to ingest media traffic into this private pod network. Most cloud load-balancers apply a DNAT step to route packets to a node and then an SNAT step to put the packet to the private pod network, so that by the time a media packet reaches a pod essentially all header fields in the [IP -5-tuple](https://www.techopedia.com/definition/28190/5-tuple) are modified, except the destination +5-tuple](https://www.techopedia.com/definition/28190/5-tuple) are modified except the destination port. Then, if any pod sends the packet over to another pod via a Kubernetes service load-balancer then the packet will again undergo a DNAT step, and so on. @@ -68,7 +67,7 @@ There are *lots* of reasons why this deployment model is less than ideal: - **It is a security nightmare.** Given today's operational reality, exposing a fleet of media servers to the Internet over a public IP address, and opening up all UDP ports for potentially - malicious access, is an adventurous undertaking, to say the least. Wouldn't it be nice to hide + malicious access, is an adventurous undertaking to say the least. Wouldn't it be nice to hide your media servers behind a secure perimeter defense mechanism and lock down *all* uncontrolled access and nefarious business by running it over a private IP? diff --git a/docs/examples/direct-one2one-call/README.md b/docs/examples/direct-one2one-call/README.md index a9db3f1b..a0caa56b 100644 --- a/docs/examples/direct-one2one-call/README.md +++ b/docs/examples/direct-one2one-call/README.md @@ -220,7 +220,7 @@ The most important component in the STUNner configuration is the TURN Gateway: t public TURN server on the UDP port 3478 through which clients will connect to each other. ```yaml -apiVersion: gateway.networking.k8s.io/v1alpha2 +apiVersion: gateway.networking.k8s.io/v1beta1 kind: Gateway metadata: name: udp-gateway diff --git a/docs/examples/direct-one2one-call/direct-one2one-call-stunner.yaml b/docs/examples/direct-one2one-call/direct-one2one-call-stunner.yaml index c0df1a8e..ce7aa1cb 100644 --- a/docs/examples/direct-one2one-call/direct-one2one-call-stunner.yaml +++ b/docs/examples/direct-one2one-call/direct-one2one-call-stunner.yaml @@ -1,4 +1,4 @@ -apiVersion: gateway.networking.k8s.io/v1alpha2 +apiVersion: gateway.networking.k8s.io/v1beta1 kind: GatewayClass metadata: name: stunner-gatewayclass @@ -24,7 +24,7 @@ spec: password: "pass-1" --- -apiVersion: gateway.networking.k8s.io/v1alpha2 +apiVersion: gateway.networking.k8s.io/v1beta1 kind: Gateway metadata: name: udp-gateway @@ -37,7 +37,7 @@ spec: protocol: UDP --- -apiVersion: gateway.networking.k8s.io/v1alpha2 +apiVersion: gateway.networking.k8s.io/v1beta1 kind: Gateway metadata: name: tcp-gateway diff --git a/docs/examples/kurento-one2one-call/README.md b/docs/examples/kurento-one2one-call/README.md index b90f6c49..05c67c6c 100644 --- a/docs/examples/kurento-one2one-call/README.md +++ b/docs/examples/kurento-one2one-call/README.md @@ -345,7 +345,7 @@ STUNner. Learn the external IP address Kubernetes assigned to the LoadBalancer s application server. ``` console -export WEBRTC_SERVER_IP=$(kubectl get service -n stunner webrtc-server -o jsonpath='{.status.loadBalancer.ingress[0].ip}') +export WEBRTC_SERVER_IP=$(kubectl get service webrtc-server -o jsonpath='{.status.loadBalancer.ingress[0].ip}') ``` Then, open `https://${WEBRTC_SERVER_IP}:8443` in your browser, accept the self-signed TLS certificate, diff --git a/docs/examples/kurento-one2one-call/kurento-one2one-call-stunner.yaml b/docs/examples/kurento-one2one-call/kurento-one2one-call-stunner.yaml index 91f4c8ef..9930fd99 100644 --- a/docs/examples/kurento-one2one-call/kurento-one2one-call-stunner.yaml +++ b/docs/examples/kurento-one2one-call/kurento-one2one-call-stunner.yaml @@ -1,4 +1,4 @@ -apiVersion: gateway.networking.k8s.io/v1alpha2 +apiVersion: gateway.networking.k8s.io/v1beta1 kind: GatewayClass metadata: name: stunner-gatewayclass @@ -23,7 +23,7 @@ spec: password: "pass-1" --- -apiVersion: gateway.networking.k8s.io/v1alpha2 +apiVersion: gateway.networking.k8s.io/v1beta1 kind: Gateway metadata: name: udp-gateway @@ -33,7 +33,7 @@ spec: listeners: - name: udp-listener port: 3478 - protocol: UDP + protocol: TURN-UDP --- apiVersion: gateway.networking.k8s.io/v1alpha2 @@ -48,4 +48,3 @@ spec: - backendRefs: - name: kms namespace: default - diff --git a/docs/examples/simple-tunnel/README.md b/docs/examples/simple-tunnel/README.md index 1f87a266..25337d15 100644 --- a/docs/examples/simple-tunnel/README.md +++ b/docs/examples/simple-tunnel/README.md @@ -65,7 +65,7 @@ that the UDPRoute specifies the `iperf-server` service as the `backendRef`, whic STUNner will forward the client connections received in any of the Gateways to the iperf server. ```yaml -apiVersion: gateway.networking.k8s.io/v1alpha2 +apiVersion: gateway.networking.k8s.io/v1beta1 kind: Gateway metadata: name: udp-gateway @@ -75,10 +75,10 @@ spec: listeners: - name: udp-listener port: 3478 - protocol: UDP + protocol: TURN-UDP --- -apiVersion: gateway.networking.k8s.io/v1alpha2 +apiVersion: gateway.networking.k8s.io/v1beta1 kind: Gateway metadata: name: tcp-gateway @@ -88,7 +88,7 @@ spec: listeners: - name: tcp-listener port: 3478 - protocol: TCP + protocol: TURN-TCP --- apiVersion: gateway.networking.k8s.io/v1alpha2 diff --git a/docs/examples/simple-tunnel/iperf-stunner.yaml b/docs/examples/simple-tunnel/iperf-stunner.yaml index f1a295cf..d30ff989 100644 --- a/docs/examples/simple-tunnel/iperf-stunner.yaml +++ b/docs/examples/simple-tunnel/iperf-stunner.yaml @@ -1,4 +1,4 @@ -apiVersion: gateway.networking.k8s.io/v1alpha2 +apiVersion: gateway.networking.k8s.io/v1beta1 kind: GatewayClass metadata: name: stunner-gatewayclass @@ -24,7 +24,7 @@ spec: password: "pass-1" --- -apiVersion: gateway.networking.k8s.io/v1alpha2 +apiVersion: gateway.networking.k8s.io/v1beta1 kind: Gateway metadata: name: udp-gateway @@ -34,10 +34,10 @@ spec: listeners: - name: udp-listener port: 3478 - protocol: UDP + protocol: TURN-UDP --- -apiVersion: gateway.networking.k8s.io/v1alpha2 +apiVersion: gateway.networking.k8s.io/v1beta1 kind: Gateway metadata: name: tcp-gateway @@ -47,7 +47,7 @@ spec: listeners: - name: tcp-listener port: 3478 - protocol: TCP + protocol: TURN-TCP --- apiVersion: gateway.networking.k8s.io/v1alpha2