Janus container

Introduction

This project creates the Janus Gateway Docker image and provides the procedure to set up the container using the default bridge network driver. There are multiple advantages to support this configuration such as it avoids having to reserve dedicated IP address per container, configuring/parameterizing the image to use different sets of ports internally and makes automatic scaling much easier. The default bridge configuration has the most constraints hence images supporting it will support most of other configurations.

The strategy followed in this project is to create a build Docker image (build image for short) first. The build image runs the Docker tools as well as the Janus build environment. It compiles and creates the target Janus gateway image (target image for short). This allows to create a substantially smaller target image than if a single image combining the build and execution was built (~300MB vs ~1.6GB). We provide two ways of building the images, the first one is manual that requires a Docker host that purpose is to build, store and run the target images. This build process is orchestrated by the container.sh script. The second build method is directly using GitLab Continous Integration (CI) scheme orchestrated by the .gitlab-ci.yml script. This method requires a GitLab setup that includes Kubernetes executors and has access to a registry (e.g. the GitLab internal container registry) for storing the created images. The second method also requires a Docker host for executing the target image.
We also provide a very simple procedure for deploying the target Janus Gteway image on Azure Kubernetes Service (AKS) using Helm charts.
Finally, at the bottom of this page in the Experimentation and observations section, we have added a discussion about some limitations that need to be considered when deploying the target image.

Notes:

• Please visit Meetecho Janus project for a detailed description of the Janus gateway.
• Out-of-the-box this project provides the simplest single host Docker configuration which may be tailored to any other more complex configuration. The procedure below allows to setup a single host running the Janus Gateway in a Docker container accessed using HTTPS only and relying on the host for certificate management. This procedure may be greately simplified by modifying the Janus Gateway configuraiton to avoid mounting multiple host folders, avoiding the installation of Certbot and the HTTP server (Nginx) etc. and, allowing instead, to simply run the Janus Gateway image.
• Only the video room plug-in (and echo test plug-in) with HTTP transport have been tried. Possibly, other plug-ins and transports will require adjustments in the content of the target image (e.g. included Ubuntu packages).
• The author welcomes questions, comments and suggestions!

Execution host setup

The figure below depicts the host configuration.

The host contains the following components:

• Docker engine for executing the build and target images.
• Nginx HTTP server for allowing Certbot automatic Letsencrypt certificates update and for serving the Janus HTML samples.
• Cetbot certificate renewal service.

The Janus target image mounts the following host volumes:

• /var/www/html/container (to container /html): Upon startup the target image copies the content of the folder containing the Janus HTML samples. This folder is accessible through HTTPS. Please note that the /var/www/html folder contains the Nginx default index.html page which is accessible through HTTP. Its purpose is to allow Letsencrypt host validation.
• /var/janus/recordings (to container /janus/bin/janus-recordings): This folder is used by the target image to store the video room recordings (when enabled).
• /etc/letsencrypt/live/ (to container /etc/certs) and /etc/letsecrypt/archive (to container /archive): These folders contain the links and Letsencrypt certificates required for TLS and DTLS shared by both Nginx and Janus gateway
• <Janus config host folder> (to container /janus/etc/janus_host: Optionally (when the RUN_WITH_HOST_CONFIGURATION_DIR environment variable is set) the target image may mount a configuration folder from the host, this configuration will override the built-in configuration.

The Janus build image mounts the following host volume:

• /var/run/docker.sock (to container /var/run/docker.sock) enables the build image to use the Docker service from the host.
• <clone directory>/janus_config, when the BUILD_WITH_HOST_CONFIG_DIR build parameter is set to 'true' the host janus configuration directory will be mounted and used in the target image creation process instead of using the default configuration that has been embedded into the build image during the build image creation.

Process

The figure below depicts the target image creation process.

The process consists in the following steps:

1. Preparation: The project is cloned from the Github repository. The default Janus gateway server configuration in <clone directory>/janus_config sub-folder is reviewed and modified according to the requirements of the target image. It is possible to add a bash script that will create the configuration Janus Gateway configuration files (see Target image execution step). When building using the GitLab CI this step has to be performed before the commit that triggers the build process.
2. Build image creation: Triggered by manually invoking the container.sh or automatically (CI) invoking .gitlab-ci.yml. The build relies on Dockerfile.build and setup.sh scripts along with some environment variables (see below) to install the necessary components of the build image. The Janus gateway configuration along with the start.sh script are stored in the build image making it self contained during the Target Image creation step (i.e. when the BUILD_WITH_HOST_CONFIG_DIR is set to false).
3. Target Image creation: Once the build image is created the container.sh or .gitlab-ci.yml scripts trigger the target image build process that relies on Dockerfile.exec and build.sh scripts, stored in the build image (/image directory) in the previous step. In this step, the required version of the Janus software is cloned and checked out as specified by the JANUS_REPO and JANUS_VERSION environment variables. Binary and source dependencies are fetched. The whole package is compiled and the target image is created. When building using the manual procedure (container.sh), instead of using the embedded Janus gateway configuration it is possible, by defining the BUILD_WITH_HOST_CONFIG_DIR variable, to mount the <clone directory>/janus_config, containing Janus gateway configuration. In that case configuration from the mounted directory will be copied into the target image instead of using the configuration embedded into the build image. Please note that if you would like to update files that are used by the target image or target image build porcess (so far only the start.sh and Dockerfile.exec scripts fall into this category) you must recreate the build image.
4. Target image execution: The created target image contains a start.sh script that is configured as the entry point. This scripts copies the Janus HTML samples, if the environment variable COPY_JANUS_SAMPLES is set to "true", and invokes the Janus gateway application. If RUN_WITH_HOST_CONFIGURATION_DIR is set to "true" the start.sh script will use the Janus configuration host folder mounted inside the container at /janus/etc/janus_host instead of using the embedded configuration located in /janus/etc/janus directory. Also CONFIG_GEN_SCRIPT may specify the name of a bash script that creates the configuraiton files. This variable conains the name of the script that may either be embedded in the Target image (/janus/etc/janus directory) or provided by the host (/janus/etc/janus_host directory) when the RUN_WITH_HOST_CONFIGURATION_DIR is set to "true". The configuration creation script will be triggered by the start.sh script at container startup. The script for generating the configuration may rely on additional environment vaiable parameters.

Build/execution host installation

This section provides the default installation procedure. The default configuration allows to access the Janus Gateway server only through HTTPs using the host's obtanied Letsencrypt certificates. Please note that this project is using Ubuntu 18.04-LTS Linux distribution. Although it has been tried only on that specific version, a priori, there are no reasons for it not to work on any other recent version of the Ubuntu distribution.

1. Provision Ubuntu 18.04 physical or virtual host with the default packages and using the default parameters. Make sure that you have access to a sudo capable user. We assume that the host is directly connected to the Internet through a 1-to-1 NAT.

   1. Make sure that the 1-to-1 NAT redirects the following ports: 80 (http), 443 (https), 8089 (janus-api), 7889 (janus-admin) to the Janus host.
   2. Reserve a name for your host in your domain (e.g. <host>.<domain>) and update the /etc/hosts file accordingly, for example:

      127.0.0.1 localhost <host>.<domain>
      [...]

2. Install Docker following these instructions then follow these steps for some additional convenience settings. Please note that the build process includes also the option to use Podman instead of Docker but Podman only allows to create the build image. It does not work yet for the target image creation. If yo wish to experiment with Podman you may use these installation instructions. Both Podman and Docker may be installed on the same host.

3. Install Nginx HTTP server. We need NGINX to automate the Letsencrypt certificate updates using the Certbot and for serving the janus HTML examples (from the /var/www/html/container host directory)

   sudo apt install nginx
   sudo apt update

4. Install the TLS certificates and the automatic certificate update service

   1. Add the Certbot PPA to your list of repositories

      sudo apt install software-properties-common
      sudo add-apt-repository universe
      sudo add-apt-repository ppa:certbot/certbot
      sudo apt update

   2. Install Certbot

      sudo apt install certbot python-certbot-nginx

   3. Get the certificates

      sudo certbot certonly --nginx

      Saving debug log to /var/log/letsencrypt/letsencrypt.log Plugins selected: Authenticator nginx, Installer nginx Enter email address (used for urgent renewal and security notices) (Enter 'c' to cancel): <your e-mail address>

      ---

      Please read the Terms of Service at https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf. You must agree in order to register with the ACME server at https://acme-v02.api.letsencrypt.org/directory

      ---

      (A)gree/(C)ancel: A

      ---

      Would you be willing to share your email address with the Electronic Frontier Foundation, a founding partner of the Let's Encrypt project and the non-profit organization that develops Certbot? We'd like to send you email about our work encrypting the web, EFF news, campaigns, and ways to support digital freedom.

      ---

      (Y)es/(N)o: N No names were found in your configuration files. Please enter in your domain name(s) (comma and/or space separated) (Enter 'c' to cancel): <host>.<domain>

      Obtaining a new certificate Performing the following challenges: http-01 challenge for bart-test-access.eastus.cloudapp.azure.com Waiting for verification... Cleaning up challenges

      IMPORTANT NOTES:

      • Congratulations! Your certificate and chain have been saved at: /etc/letsencrypt/live/bart-test-access.eastus.cloudapp.azure.com/fullchain.pem Your key file has been saved at: /etc/letsencrypt/live/bart-test-access.eastus.cloudapp.azure.com/privkey.pem Your cert will expire on 2020-05-04. To obtain a new or tweaked version of this certificate in the future, simply run certbot again. To non-interactively renew all of your certificates, run "certbot renew"
      • If you like Certbot, please consider supporting our work by:

      Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate Donating to EFF: https://eff.org/donate-le

   4. As specified in the output above the certificates may be found here:

      /etc/letsencrypt/live/<host>.<domain>/fullchain.pem
      /etc/letsencrypt/live/<host>.<domain>/privkey.pem

   These files are links from the /etc/letsencrypt/archive directory.

   !!VERY IMPORTANT!! Make sure that NON root users have read access to the links and the certificates.

   chmod -R a+r+x /etc/letsencrypt/live
   chmod -R a+r+x /etc/letsencrypt/archive

   1. You may test the Certbot certificate renewal by issuing the following command:

      sudo certbot renew --dry-run --allow-subset-of-names

5. Clone the project repo

   git clone https://github.com/bartbalaz/janus-container.git <clone directory>
   cd <clone directory>

6. Create a http server configuration

   1. Copy the configuration file

      sudo mkdir /var/www/html/container
      cd <clone directory>
      sudo cp ./scripts/nginx.conf /etc/nginx/sites-available/<host>.<domain>.conf
      sudo ln -s /etc/nginx/sites-available/<host>.<domain>.conf /etc/nginx/sites-enabled/

      Note that the /var/www/html/container directory will be used to store the Janus HTML samples.
   2. Edit the configuration file /etc/nginx/sites-available/<host>.<domain>.conf and replace the <host>.<domain> place holder with your host and domain name.
   3. Restart the Nginx server

      sudo systemctl restart nginx

7. Create a recording folder

   sudo mkdir -p /var/janus/recordings

Manual build procedure

This procedure allows to create build and target images on a simple Docker host.

1. Set the build parameters environment variables by issuing the export command for each parameter or by editing the <clone directory>/scripts/config file and issuing the source command. All the available parameters are sumarized in the table below.

   # Set each parmeter individually 
   export SOME_PARAMETER=some_value
   
   # Or set all the parameters saved in the config file
   cd <clone directory>
   source scripts/config

Parameter	Mandatory (Y/N/C)	Default	Process step	Description
IMAGE_REGISTRY	N	not set	2, 3	Registry for storing both the build and target images, including the project/user folder if necessary (i.e. docker.io/some_project).
IMAGE_REGISTRY_USER	N	not set	2, 3	Registry user name
IMAGE_REGISTRY_PASSWORD	N	not set	2, 3	Registry user password
BUILD_IMAGE_NAME	N	janus_build	2, 3	Name of the build image
BUILD_IMAGE_TAG	N	latest	2, 3	The version to tag the build image with
IMAGE_TOOL	N	docker	2, 3	Tool for creating and managing the images, either "podman", "docker" or "external" when image building is handled outside of the project scripts (e.g. by Gitlab CI )
HOST_NAME	N	<host>.<domain>	3	Name of the host in full fqdn format. This value is only used in displaying the execution command at the end of an successful build
JANUS_REPO	N	https://github.com/meetecho/janus-gateway.git	3	Repository to fetch Janus gatway sources from
JANUS_VERSION	N	master	3	Version of the Janus gateway sources to checkout (e.g. v0.10.0). If none is specified the master branch latest available version will be used
TARGET_IMAGE_NAME	N	janus	3	Target image name
TARGET_IMAGE_TAG	N	latest	3	The version to tag the target image with
SKIP_BUILD_IMAGE	N	false	3	When set to "true" the build image will not be build
SKIP_TARGET_IMAGE	N	false	3	When set to "true" the target image will not be build
BUILD_WITH_HOST_CONFIG_DIR	N	false	3	When set to "true" the build image will mount the host Janus gateway configuration directory (i.e. /janus-config) instead of using the one that was copied during the build image creation
RUN_WITH_HOST_CONFIGURATION_DIR	N	false	4	When set to "true" the image execution command displayed at the end of the successful build will add an option to use host Janus server configuration directory (i.e. /janus-config) instead of the embedded configuration during the target image creation process
COPY_JANUS_SAMPLES	N	false	4	When set to "true" the image execution command displayed at the end of the successful build will add an option to trigger the image to copy the Janus HTML samples to a mounted folder
CONFIG_GEN_SCRIPT	N	empty	4	When set to the name of the script that generates the Janus Gateway configuration upon the container startup, the image execution command displayed at the end of the successful build will add an option to trigger that process. Note the referred script may be placed in the mounted configuration folder (when using RUN_WITH_HOST_CONFIGURATION_DIR) or embeddd in the /janus_config folder (created during the Target image creation)

2. Review the Janus gateway configuration files stored in /janus_config directory these files will be integrated into the build image and into the target image.
3. Launch the build process, this process performs two steps: creates the build image (unless the SKIP_BUILD_IMAGE is set to "true"), then creates the target image (unless SKIP_TARGET_IMAGE is set to "true"). Both images will appear in the local image Docker registry (issue "docker images" to verify). To perform either step set the above mentioned "SKIP_" parameters to the appropriate values.

   cd <clone directory>
   ./container.sh

Gitlab CI build procedure

This procedure is integrated into GitLab and provides a full automation pipeline of the build and target images creation. The procedure relies on the Kaniko tool for creating the container images. The main advantage of Kaniko is that it is self contained and does not require proviledged access to any host resources. The automation pipeline defined in the .gitlab-ci.yml is divided into three steps that are triggered by committing two different types of tags:

1. Create the build image, triggered by committing a tag that has the form build-x.y.z. The resulting build image will be tagged with build-x.y.z and latest tags.
2. Create the target image content, triggered by committing a tag that has the form x.y.z (i.e. release) or a branch that starts with dev- (i.e. development branch).
3. Create the target image, triggered by the same conditions as the previous step. The resulting target image will be tagged with x.y.z and latest tags.

As stated earlier, the automation relies on GitLab Kubernetes executor. Although, we did not try, the GitLab Docker executor perhaps may also work.
The following parameters have to be defined in your environment. Please note that the current CI configuration pushes the images to two registries (ACR and NCR) if you would like to use a single registry instead simply remove the lines referring either to ACR or NCR from the .gitlab-ci.yml file and ignore the related parameters below.

Parameter	Description
ACR_AUTH	Base64 encoded value of ACR image registry credentials ":" values, see section "Define an image from a private Container Registry" on this page
NCR_AUTH	Base64 encoded value of NCR image registry credentials ":" values, see section "Define an image from a private Container Registry" on this page
ACR_REGISTRY	ACR registry location (e.g. "gcr.io")
NCR_REGISTRY	NCR registry location (e.g. "gcr.io")
ACR_PROJECT	The project in the ACR registry where the images will be stored (e.g. "some_project_name"), may be left empty
NCR_PROJECT	The project in the NCR registry where the images will be stored (e.g. "some_project_name"), may be left empty
DOCKER_AUTH_CONFIG	See section "Define an image from a private Container Registry" on this page
JANUS_BUILD_IMAGE	Name of the Janus build image (e.g. "janus_build")
JANUS_TARGET_IMAGE	Name of the Janus Gateway target image (e.g. "janus")

The following two parameters are defined in the .gitlab-ci.yml file in the create_target_image_content stage to ensure that they are version controlled.

Parameter	Description
JANUS_REPO	The repository to fetch the Janus Gateway source code (e.g. https://github.com/meetecho/janus-gateway.git)
JANUS_VERSION	The Janus Gateway source code version to checkout (e.g. "v0.10.0")

Please note that further tuning of the .gitlab-ci.yml is required to fit into your setup. For example, you must set the right location and version of the build image and you may need to tag the jobs with different tags so they get picked up by the appropriate runner, set the right version of the janus buld image etc.

Running the target image on the build/execution host

1. Launch the target image by invoking either of the commands on the build/execution host that are displayed at the end of a successful manual target image build (if SKIP_TARGET_IMAGE was set to "false" or not exported). For example:

   docker run --rm -d -p 8089:8089 -p 7889:7889 \
   	-v /etc/letsencrypt/live/<host>.<domain>:/etc/certs \
   	-v /etc/letsencrypt/archive:/archive \
   	-v /var/janus/recordings:/janus/bin/janus-recordings \
   	-v <clone folder>/janus_config:/janus/etc/janus_host -e "RUN_WITH_HOST_CONFIGURATION_DIR=true" \
   	-v /var/www/html/container:/html -e "COPY_JANUS_SAMPLES=true" \
   -e "CONFIG_GEN_SCRIPT=<configuration_generation_script>"
   	some.container.registry.com/janus:some_tag

   Notes:
   • If the RUN_WITH_HOST_CONFIGURATION_DIR parameter is set to "false" or not specified (see above) it is not necessary to mount the <clone folder>/janus_config folder.
   • If the COPY_JANUS_SAMPLES parameter is set to "false" or not specified (see above)it is not necessary to mount the /var/www/html/container folder.
2. Try the image by browsing to https://<host>.<domain>/container Please note that:
   • By default the video room plugin configuration (configuration file: <clone directory>/janus_config/janus.plugin.videoroom.jcfg) is set to require string video room names which is not the Janus gateway default configuraiton.
   • The default configuration allows only HTTPS transport through secure ports 8089 - janus-api and 7889 - janus-admin.

Quick Docker tips

1. List all the images available locally

   docker images

2. List all the containers that are stopped but have not been removed

   docker ps -a

3. Remove a stopped container

   docker rm <first few chars of the container id as displayed by "ps" command>

4. Remove an image

   docker rmi <first few chars of the image id as displayed by "images" command>

5. Stop a container

   docker stop <first few chars of the container id as displayed by "ps" command>

6. Start a container in interactive mode, that will be removed when stopped, overriding the defined entrypoint, exposes a port, mounts a volume and sets an environment variable

   docker run --rm -it -p <host port>:<container port> -v <host volume/directory>:<container directory> -e "VARIABLE_NAME=VARIABLE_VALUE" --entrypoint <new entrypoint command (e.g. "/bin/bash"> <image name>:<image tag>

7. Execute an interactive command in a running container

   docker exec -it <first few chars of the container id as displayed by "ps" command> <command to execute (e.g. "/bin/bash")>

Deploying the target image on Azure Kubernetes Service (AKS) using Helm charts

This is an example of procedure for deploying and running the Janus target image on AKS. Plese note that we have only tried this deployment with the video room sample application. The other applications will require some adjustments in the procedure.

Prerequisites

The following prerequisites must be satisfied

1. MS Azure subscription and a user able to create Azure resources groups because each cluster requires one additional resource group that gets created during cluster creation. As explained here the purpose of this resource group is to contain the resources that are solely dedicated to a cluster. For example, the cluster networking resources are part of this dedicated resource group.
2. A configurable domain that will allow to assign <host>.<domain> to the static IP address assigned to the AKS cluster
3. TLS certificate and the associated key file for <host>.<domain>

Add AKS/Kubernetes tools to the build/execution host

The build/execution host will be used to interact with the Azure Kubernetes cluster.

1. Follow these instructions to setup the Azure CLI.
2. After installing the Azure CLI issue sudo bash az aks install-cli to install kubectl the Kubernetes CLI and then bash az login --use-device-code to login to Azure.
3. For greater convenience we also suggest to install the kubectl shell autocompletion as presented on this page.
4. Finally, you need to install Helm following these instructions.

Create the Kubernetes cluster

For creating a Kubernetes cluster the user has to be able to create Azure resources groups because each cluster requires one additional resource group that gets created during cluster creation. As explained here the purpose of this resource group is to contain the resources that are solely dedicated to a cluster. For example, the cluster networking resources are part of this dedicated resource group. To create the Kubernetes cluster follow the following steps:

1. Create a new resource group, note this step may be skipped if a suitable resource group already exists

   az group create --name <resource group> --location eastus

2. Create the Kubernetes cluster. To run the Janus compact deployment for testing and demonstration purposes a single node cluster is sufficient. As explained above, the following command will also create a resource group with the default name MC_<resource group>_<cluster name>_eastus

   az aks create --resource-group <resource group> --name <cluster name> --node-count <desired nuber of nodes> --generate-ssh-keys

3. Login into the cluster

   az aks get-credentials --resource-group <resource group> --name <cluster name>

4. It should be possible now to see the cluster nodes by issuing

   kubectl get nodes

Create a static IP address

Create the cluster IP address, it has to belong to the resource group dedicated to the cluster, namely MC_<resource group>_<cluster name>_eastus

```bash
az network public-ip create --resource-group MC_<resource group>_<cluster name>_eastus --name <IP address name> --sku Standard --allocation-method static
```

After creating the static IP address it is a good time to configure your DNS to point the <host>.<domain> to that address

Storage account and file share

A file share is required in a storage account for saving the conference room recordings.

1. Create a azure storage account

   az storage account create --name <storage account> --resource-group <ressource group> --location eastus --sku Standard_RaGRS --kind StorageV2

2. Get the connection string from the storage account. The account name/user name follows AccountName= while the key follows AccountKey= in the command output

   az storage account show-connection-string --name <storage account> -g <resource group> -o tsv
   DefaultEndpointsProtocol=https;EndpointSuffix=core.windows.net;AccountName=<storage account>;AccountKey=<account key>

3. Create the file share

   az storage share create -n <file share> --account-name <storage account> --account-key <key obtained in the previous step>

4. Get the storage account keys

   az storage account keys list --resource-group <resource group> --account-name <storage account>

5. Mount the file share on a Linux host. Please note that the host has to have CIFS installed. This directory will contain the raw recordings.

   mkdir <mount point>
   sudo mount -t cifs //<storage account>.file.core.windows.net/<file share> <mount point> -o vers=3.0,username=<storage account>,password=<account key>,dir_mode=0777,file_mode=0777,serverino

Secrets

Once all the resources are created we need to create several Kubernetes secrets that will store the TLS certificates, the required information for accessing the file share and the registry credentials (if necessary) for fetching the target image.

1. Create the secret containing the TLS keys

   cd <home directory>
   kubectl create secret tls certs  \
   --cert=tls.crt\
   --key=tls.key

2. Create the secret allowing to mount the file share in the pod

   kubectl create secret generic file-share --from-literal=azurestorageaccountname=<storage account> --from-literal=azurestorageaccountkey=<storage account key>

3. Create the secret(s) allowing to fetch the images from the registries.

   docker login <container registry>
   User: <container registry user name>
   Password: <container registry password>
   kubectl create secret generic regcred \
   --from-file=.dockerconfigjson=<home directory>/.docker/config.json> \
   --type=kubernetes.io/dockerconfigjson

Configure the deployment

Edit the <clone directory>/janus_helm/values.yaml file. The table below sumarizes the required values.

Parameter	Description
env.clusterIp	The IP address that was allocated to the AKS cluster
env.clusterName	<host>.<domain> of the AKS cluster
env.shareName	The name of the share for storing the recordings
env.secrets.tlsCertificates	The name of the secret that contains the TLS key and certificate (e.g. "certs")
env.secrets.fileShare	The name of the secret that contains the file share parameters (e.g. "file-share")
env.secrets.registriesCredentials	The name of the secret that contains the registry credentials (e.g. "regcred")
janus.containerRegistry	The registry to fetch the target image
janus.imageName	The name of the image to fetch
janus.imageTag	The tag of the image to fetch
janus.sessionPort	The session management port number
janus.adminPort	The admin port numer, optional if not set the admin port will be desactivated
janus.tokenSecret	Token for authenticating the session management messages, optional if not set no token will be required
janus.adminSecret	Password for accessing the admin port, optional if not present the admin port will be desactivated
janus.recordFolder	Folder where the file share will be mounted in the target image and the recordings will be stored
janus.eventUser	User name for accessing the event collector
janus.eventPassword	Password for accessing the event collector
janus.eventBackendUrl	The URL of the event collector (e.g. https://some.host/some/path)
janus.nat.stunServer	Stun server name (e.g. "stun.l.google.com")
janus.nat.stunPort	Stun port 19302
janus.nat.niceDebug	Enable (true) or disable (false) nice debugging.
janus.nat.fullTrickle	Enable (true) or disable (false) the full tricke optimization
nginx.httpsPort	The HTTPS port to expose (e.g. 443)

Launch the deployment

• After configuring and before starting the deployment it is a good practice to verify if the configuration by issuing:

   cd <clone directory>
   helm install --debug --dry-run janus ./janus_helm

• If the command does not report any errors and a valid Kubernetes manifest is displayed the deployment may be launched by issuing:

   cd <clone directory>
   helm install janus ./janus_helm

• To verify the status of the deployment issue:

   helm status janus

• To stop the deployment issue:

   helm uninstall janus

kubectl utility may be used to query and manipulate the deployment. Once the deployment is running the Janus HTML samples may be accessed at https://<host>.<domain>/

Experimentation and observations

The figure below shows the network configuraiton when running Janus Gateway in a Docker container configured with the default bridge network. The Docker host is a data center virtual or physical machine accessible through a 1-to-1 NAT firewall through subnet Y. The Janus client is located in a private "home" subnet X that offers a typical "home" router/firewall. Optionally, the home network may belong to an ISP that provides private subnet W and a NAT enabled firewall. Such ISP configuraiton is frequent with mobile opertors. The default Docker bridge configuration provides a private subnet for the containers. The conainers may access the public network thanks to the netfilter MASQUERADE target NAT functionlity applied to any packets leaving the private subnet Z. The container is configured to expose the Janus gateway control (e.g. 8089 for Janus API and 7889 for Janus admin) and initially media ports (e.g. 10000-12000). As you will see below one of our conclusions consists in not exposing the media ports. Janus gateway server is configured to run in tricke and full ICE mode. The solution relies on a STUN server that allows to discover the public addresses of the client and the server. In normal circumstances the depicted TURN server should not be required as the ICE protocol with the help of the STUN server allows to establish the end-to-end communication.

Issue #1 - Issue caused by Janus Gateway server running in a Docker container using the default bridge configuration

The figure below shows a simplified successfull sequence where the ICE suceeds to establish bidirectional media streams between the client and the gateway.

1. The offer is issued by the client.
2. Based on the offer and/or trickled candidates the gateway sends connectivity checks that cannot reach the client.
3. Eventually the gateway sends an aswer message that allows the client to start sending STUN probles.
4. Thanks to the gateway earlier connectivity checks the client STUN probles reach the server (the firewall port is open).
5. Thanks to the client connectivity checks (the firewall port is open) the gateway connectivity checks are reaching the client.

The next figure shows the unsucessful sequence.

1. This time the offer is sent by the gateway.
2. Based on the offer and/or trickled candidates the client sends connectivity checks that cannot reach the gateway. These probes are rejected by the MASQUERADE netfilter target because the 1-to-1 NAT firewall is configured to forward any media traffic to the gatway. An ICMP error message is generated for each rejected probe.
3. The client generates an answer.
4. Based on the answer and/or trickled candidates the gateway generates connectivity checks that for some reason never make it to the client.
5. The client connectivity checks never make it to the gateway neither.

Therefore our initial analysis has lead us to the same concusion as presented in this slide pack by Alessandro Amirante from Meetecho. Now, going a bit more into details the next figure below shows an excerpt of the packet capture at the virtual machine network interface.

1. connectivity check sent by the client before the gateway had a chance to open the port. As presented in step 2 on the previous figure above.
2. An ICMP "destination unreachable" error is generated.
3. The gateway sends a STUN request to a STUN server to retrieve its server reflexive address and port.
4. The STUN server replies indicating the reflexive port is 20422
5. The gateway issues connectivity checks from port 20422 to the client local addresses (local subnet 192.x.y.z and some VPN 10.x.y.z) which are unreachable because the client is on a private subnet.
6. The connectivity check destined to the client server reflexive (i.e. "reachable") address and port gets its source port reassigned to 1599 (instead of 20422). This happens because the earlier connectivity check from the client destined to the gateway address and port 20422 has altered the state of the MASQUERADE netfilter target. Please note we were not able to identify the reason for this behavior (e.g. security vunerability protection, standard specification, DOS attack protection etc.).

Therefore the client connectivity checks are lost because of the race condition between the gateway opening firewall ports and the client sending STUN probles and because the MASQUERADE netfilter target does not allow a host from the private subnet to send packets to a remote host using the same quintuple (source address, destination address, source port, destination port, protocol) as the one recently rejected from the remote host. On the other hand the server STUN probles are most probably rejected by the client side firewall because its NAT configuraiton is port restricted.

Solutions

In an attempt to eliminate the root cause and to delay the STUN client probes we have configred the gateway to trickle the candidates, which was unsuficient. Therefore we have also added an addional 1s delay in the client (janus.js file) when processing the received trickle candidates from the gateway. While this is not an acceptable solution the problem appeares to be solved.

1. The gateway sends the offer.
2. The gateway starts trickling the candidates. But the processing of the received tricked candidates is delayed by 1s at the client.
3. The client issues an answer.
4. The gatway sends a connectivity check that does not reach the client because the firewall port is still closed because of the delay.
5. After the delay has expired the client sends a connectivity check that opens the firewall port and reaches the gatway.
6. Finally the server resends a connectivity check that this time reaches the client.

The second solution consists in reconfiguring the 1-to-1 NAT firewall to disallow exposing any ports besides 80 (http), 443 (https), 8089 (janus-api) and 7889 (janus-admin). All the other ports require the gateway to send an initial "opening" request.

1. The gateway sends the offer.
2. Based on the offer and/or tricked candidates the client sends connectivity checks to the gatway. All these probes get filtered out by the 1-to-1 NAT and hence don't cause the above mentioned issue any more.
3. The client sends an answer to the gatway.
4. Based on the answer and/or trickled candidates the gatweay sends connectivity checks to the client.
5. The client resends the connectivity checks that this time reach the server.

Issue #2 - Issue caused by the presence of the ISP firewall

In some rare cases, the ISP firewall behaves the same way as the Netfilter MASQUERADE target and in the situation a connectivity check is received from the Janus Gateway server before the Janus client had a chance to issue a connectivity check towards the server the ISP firewall will replace the discovered port nuber with a new one. This issue occurs when the Janus client sends an offer message to the Janus Gateway server along with the candidates before receiving the Janus Gatway server candidates. As a matter of fact this issue is a mirror of the previous issue.

Solutions

To solve this issue we have to either delay/prevent the Janus Gatway to issue connectivity checks or mitigate the (STUN) server reflexive candidates failure. There are no easy ways to delay the connectivty checks but they may be disabled by changing the Janus Gateway server configuration from bridge to host network and activating the ICE Lite mode. According to RFC 8445, in ICE Lite mode no connectivity checks should be made. Unfortunately after reconfiguring the Janus Gateway to ICE Lite mode the connectivity checks are still emitted, this may be a bug in the implementation of the libnice library or its integration within the Janus Gateway. Eventually, it was possible to supress the conectivity checks by temporarely modifying the Janus Gatway code which has resolved the issue. We have also tried to enable the TURN server which, as expected, solves the issue by providing additional relayed candidates. The main drawback of this solution is the need for an addional server that relays all the traffic and which creates a bottleneck that needs to be managed. Hopefully only a minority of Janus clients will be using ISPs having such firewall configuration.

Conclusion

It is possible to use the default Docker bridged network driver but some conditions have to be met by the infrastructure specifically the firwall leading to the Janus Gateway server. The firewall has to be able to block the client requests without triggering a port change as it happens with the MASQUERADE netfilter target. Additnally to avoid any potential firewall issues with some ISPs that provide private IP addresses to their customers a TURN server must be added to the deployment.