AWS IoT Greengrass V2 Community Component - The Things Stack LoRaWAN

The Things Stack is an open-source LoRaWAN stack. Users can deploy the stack as Open Source (free) or Enterprise (requires a license) from images available on Docker Hub.

This repository facilitates packaging The Things Stack into an AWS IoT Greengrass V2 component named aws.greengrass.labs.TheThingsStackLoRaWAN. This component enables customer use cases that require a private LoRaWAN Network Server (LNS) at the edge. The Things Stack is also offered as an AWS Marketplace AMI. This Greengrass component is a logical equivalent, but pitching at the edge.

For Enterprise deployments, ingestion into AWS IoT Core can be achieved by deploying The Things Stack AWS IoT default integration into your AWS account. For Open Source deployments, The Things Stack Pub/Sub integration is a good option for achieving ingestion into AWS IoT Core.

The Pub/Sub integration can also be used to publish and subscribe to topics on the Greengrass local Moquette MQTT broker. This allows integration with the cloud, other Greengrass components on the same device and/or with other devices on the local network. The Things Stack can therefore leverage AWS-managed Greengrass components, custom Greengrass components, community Greengrass components and AWS services to deliver powerful edge solutions that extend The Things Stack's capabilities.

Architecture

An overview of the system architecture is presented below.

The aws.greengrass.labs.TheThingsStackLoRaWAN component is a thin wrapper around a conventional The Things Stack deployment. The Things Stack is composed of multiple Docker containers, using Docker Compose. Subject to configuration, there may be a Redis container and either a PostreSQL or CockroachDB container, in addition to The Things Stack container.

The Things Stack is delivered as Docker images on Docker Hub. This component downloads Docker images from Docker Hub with the help of the Docker application manager managed component.

The Things Stack configuration includes a considerable amount of sensitive information. Accordingly, the configuration is stored as a secret in AWS Secrets Manager. This component gets its configuration from Secrets Manager with the help of the Secret manager managed component.

The Things Stack offers an API and various integrations that can be used for remote ingestion or control, or can be used by other Greengrass components to achieve local data processing and actions.

For Enterprise deployments, ingestion into AWS IoT Core can be best achieved by deploying The Things Stack AWS IoT default integration into your AWS account. This is option is shown in green on the architecture diagram.

As shown in blue, The Things Stack can use its Pub/Sub integration to connect directly to AWS IoT Core. This option would typically only be used for deployments of the Open Source edition of The Things Stack. Alternatively, and more powerfully, the Pub/Sub integration can instead connect to the local Greengrass Moquette MQTT broker. This is option is shown in red. If The Things Stack and other TCP/IP devices on the local network are registered as Local Client Devices with Greengrass, this architecture allows The Things Stack to communicate with those other devices via the broker and with other Greengrass components via Greengrass Interprocess Communication. Furthermore, the MQTT bridge connects AWS IoT Core as an additional publisher and subscriber, facilitating cloud integration as well.

Repository Contents

Item	Description
/artifacts	Greengrass V2 component artifacts that run on the Greengrass edge runtime.
/cicd	CDK Typescript app for a CodePipeline CI/CD pipeline.
/images	Images for README files.
/libs	Python libraries shared by Python scripts.
/robot	Robot Framework integration tests.
/tests	Pytest unit tests.
/tts-config	The Things Stack configuration files.
create_certs.sh	Creates self-signed TLS certificates for a given domain name or IP address.
create_config_minimal.py	Creates a minimal configuration needed to deploy the component for a given domain name or IP address.
create_config_secret.py	Creates or updates The Things Stack configuration secret in Secrets Manager.
deploy_component_version.py	Deploys a component version to the Greengrass core device target.
gdk_build.py	Custom build script for the Greengrass Development Kit (GDK) - Command Line Interface.
gdk-config.json	Configuration for the Greengrass Development Kit (GDK) - Command Line Interface.
quickstart.sh	Creates and deploys a component with default configuration for a given domain name or IP address.
recipe.json	Greengrass V2 component recipe template.

Requirements and Prerequisites

Greengrass Core Device

Platform

This component requires that the Greengrass device be running a Linux operating system. It supports all architectures supported by Greengrass itself.

Edge Runtime

The Greengrass edge runtime needs to be deployed to a suitable machine, virtual machine or EC2 instance. Please see The Things Stack stated prerequisites for guidance on the resources required. For a small number of devices and gateways, the processing and memory requirements are small.

The Greengrass machine or instance should not be running any process that uses any ports that may be needed by The Things Stack. The specific ports used depends on your particular configuration of The Things Stack. The configuration generated by quickstart.sh requires ports 443, 1700, 6379, 8881, 8882, 8883, 8884, 8885, 8887, 26256 and 26257 to be available.

Docker Requirements

The Things Stack is generally composed as an application with multiple Docker containers, using Docker Compose. Images are obtained from Docker Hub.

Therefore your core device must meet the requirements to run Docker containers using Docker Compose and Docker Hub.

Not all releases of these third-party container images support all Greengrass architectures. Care must be taken to select appropriate images and releases to suit your target architecture.

Python Requirements

This component requires both python3 and pip3 to be installed on the core device.

Greengrass Cloud Services

Core Device Role

Assuming the bucket name in gdk-config.json is left unchanged, this component downloads artifacts from an S3 bucket named greengrass-tts-lorawan-REGION-ACCOUNT. Therefore your Greengrass core device role must allow the s3:GetObject permission for this bucket. For more information: https://docs.aws.amazon.com/greengrass/v2/developerguide/device-service-role.html#device-service-role-access-s3-bucket

Additionally, this component downloads sensitive The Things Stack configuration from Secrets Manager. Therefore your Greengrass core device role must also allow the secretsmanager:GetSecretValue permission for the greengrass-tts-lorawan-ID secret.

Policy template to add to your device role (substituting correct values for ACCOUNT, REGION and ID):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject"
      ],
      "Resource": "arn:aws:s3:::greengrass-tts-lorawan-REGION-ACCOUNT/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue"
      ],
      "Resource": "arn:aws:secretsmanager:REGION:ACCOUNT:secret:greengrass-tts-lorawan-ID"
    }
  ]
}

Developer Machine

AWS CLI

It may be necessary to upgrade your AWS CLI if you wish to use any greengrassv2 commands, as these are relatively recent additions to the CLI.

cfssl

If you wish to generate your own self-signed TLS certificates for your domain, or use quickstart.sh, then cfssl is required. Example installation on an Ubuntu or Debian machine:

sudo apt update
sudo apt install -y golang-cfssl

Or using Homebrew (Mac or Linux):

brew install cfssl

Alternatively consider installing from released binaries available in the cfssl GitHub repository. Windows installation executables are available.

Python

Most of the scripts in this repository are Python scripts. They are Python 3 scripts and hence python3 and pip3 are required.

Package dependencies can be resolved as follows:

pip3 install -r requirements.txt
pip3 install -r robot/requirements.txt

Please consider to use a virtual environment.

Boto3 is included in the package dependencies and therefore your machine requires appropriate credentials.

GDK CLI

This component makes use of the Greengrass Development Kit (GDK) - Command Line Interface (CLI). This can be installed as follows:

pip3 install git+https://github.com/aws-greengrass/aws-greengrass-gdk-cli.git

Bash

The quickstart.sh and create_certs.sh scripts are Bash scripts. If using a Windows machine, you will need a Bash environment.

jq

The jq utility is used by quickstart.sh. Release packages for Linux, OS X and Windows are available on the jq site.

Alternatively, Ubuntu includes a jq package:

sudo apt update
sudo apt install jq

Getting Started

Here we define two ways to get started: Quickstart or Slowstart.

All scripts are compatible with Linux, Mac or Windows operating systems, provided a Bash environment is available.

Quickstart

The quickstart.sh bash script is supplied to help you get going fast. For experimentation in non-production settings.

Before running the script, users must:

Deploy Greengrass V2 to a Linux machine, virtual machine or EC2 instance.
Initialize the Greengrass core device to satisfy the the requirements to run Docker containers using Docker Compose and Docker Hub.
Install cfssl on the developer machine.
Set the AWS region in gdk-config.json.

The Quickstart script will:

Install required Python packages.
Create self-signed certificates for a given domain name or IP address.
Create a minimal The Things Stack configuration for the given domain name or IP address.
Upload the secure configuration to a configuration secret in Secrets Manager.
Use GDK to build the component.
Use GDK to publish a new component version to Greengrass cloud services and upload artifacts to an S3 bucket.
Prompt you to add permissions for the configuration secret and artifacts bucket to the Greengrass core device role.
Deploy the new component version to the Greengrass core (creating The Things Stack admin user in the process).
Run Robot Framework integration tests to confirm The Things Stack is running under Greengrass.

The script accepts 4 arguments:

Domain name or IP address (this can be an EC2 default domain name).
The password of The Things Stack admin user that will be created.
The email of The Things Stack admin user that will be created.
Greengrass Core device name.

Example execution:

bash quickstart.sh example.com mypassword user@example.com GGTheThingsStackLoRaWAN

Slowstart

For any serious use of the component, Quickstart shall not be appropriate.

Manual Deployment

If not using Quickstart, you must perform the following steps:

Deploy the Greengrass runtime to your machine, virtual machine or EC2 instance. This should be a Linux machine, but can be of any architecture.
Configure the LoRaWAN stack as per https://www.thethingsindustries.com/docs/getting-started/installation/configuration/. Take care to select Docker images with the correct architecture.
Configure certificates as per https://www.thethingsindustries.com/docs/getting-started/installation/certificates/.
If BYO/generating custom TLS certificates (rather than using Lets Encrypt), then add them to the /tts-config directory.
If using an Enterprise license file, add it to the /tts-config directory.
Set the AWS region in gdk-config.json.
Run create_config_secret.py to create the configuration secret in Secrets Manager.
Run gdk component build to build the component.
Run gdk component publish to create a component version in Greengrass cloud service, and upload artifacts to S3.
Add permissions for the configuration secret and artifacts bucket to the Greengrass core device role.
Run deploy_component_version.py to deploy the new component version to your Greengrass device.

For iterative configuration changes, repeat steps 2, 7, 8, 9 and 11.

Example Execution

Example of steps 7, 8, 9 and 11:

python3 create_config_secret.py mypassword user@example.com
gdk component build
gdk component publish
python3 deploy_component_version.py 1.0.0 MyCoreDeviceThingName

This example:

Creates a Secrets Manager secret in your account in the region specified in gdk-config.json.
Sets the admin user for The Things Stack to have password mypassword and email user@example.com.
Builds the component and publishes it to your account in the region specified in gdk-config.json.
Deploys the new component version to Greengrass core device MyCoreDeviceThingName.

CI/CD Pipeline

This repository offers a CodePipeline CI/CD pipeline as a CDK application. This can be optionally deployed to the same account as the Greengrass core.

This CI/CD pipeline automates steps 8,9 and 11. Following deployment, it performs automated smoke tests to ensure that The Things Stack has started correctly. With the pipeline deployed, users can make iterative configuration changes, update the configuration secret using create_config_secret.py, and then trigger the CI/CD pipeline to handle the rest.

Automated Testing

This repository includes an automated test suite built on top of Robot Framework. This can be run on demand from the command-line but it is also included as part of the CI/CD pipeline.

The Things Stack Configuration Tips

Configuration of the LoRaWAN stack is mainly an exercise in editing tts-config/docker-compose.yaml and tts-config/config/stack/ttn-lw-stack-docker.yml. Please consult The Things Stack configuration documentation for details.

Defaults

The tts-config/docker-compose.yaml and tts-config/config/stack/ttn-lw-stack-docker.yml files in this repository are The Things Stack Enterprise example configuration files, with the following divergence:

All Enterprise configuration elements are commented out so that it defaults to Open Source edition.
Acme Let's Encrypt TLS certificates are disabled and custom TLS certificates are enabled. This allows deployment to a server that only has an IP address, a local network DNS name or an EC2 default domain name.
All non-TLS ports and endpoints are disabled.
Postgres is selected as the database, instead of Cockroach, so that this component can be easily deployed to many different architectures. Cockroach only supports amd64 architecture.
Versioned image tags are used instead of just latest tags. These versioned tags are the most recently tested combination. The latest tags do not always support as many architectures as formal releases do.

These default settings likely facilitate minimal effort in deploying this component to your Greengrass core device.

Enterprise versus Open Source

As compared to Open Source, an Enterprise license adds:

Device Claim Server
Multi-tenant support
Storage Integration
AWS IoT Default Integration support
Support plan

Production Deployments

Per The Things Stack recommendations, production deployments should reference specific Docker image tags (not just latest) and consideration should also be given to using external managed instances of the databases.

Image Architecture

CockroachDB is not available for Arm architecture. You must use PostgreSQL instead.

The Things Stack images tagged latest on Docker Hub are not available for all architectures. Only formal releases are.

Pub/Sub Integration

AWS IoT Core

The Pub/Sub integration can be used to achieve ingestion into AWS IoT Core. This is particularly useful for the Open Source edition of the stack since the The Things Stack AWS IoT default integration is only available for Enterprise deployments.

Firstly create a Thing representing The Things Stack, and obtain the device certificate, private key and CA certificate.

Obtain the AWS IoT Core endpoint:

aws iot describe-endpoint --endpoint-type iot:Data-ATS
{
    "endpointAddress": "ENDPOINTID-ats.iot.REGION.amazonaws.com"
}

Add a Pub/Sub integration in your Application in The Things Stack with the following settings:

Item	Value
Provider	MQTT
MQTT Configuration: Use secure connection	Enabled
MQTT Configuration: Root CA certificate	Root CA saved when creating the Thing in AWS IoT Core
MQTT Configuration: Client certificate	Device certificate saved when creating the Thing in AWS IoT Core
MQTT Configuration: Client private key	Private key saved when creating the Thing in AWS IoT Core
MQTT Configuration: Server URL	mqtts://ENDPOINTID-ats.iot.ap-REGION.amazonaws.com:8883

Greengrass Moquette MQTT Broker

Greengrass V2 includes an AWS-managed Moquette MQTT broker component. This can be deployed to Greengrass to allow Greengrass components and devices on your local network to communicate with each other, without relying on an internet connection to AWS IoT Core.

Using the Pub/Sub integration, The Things Stack can be a "local IoT device" that connects to the Moquette broker. Additionally, Greengrass V2 includes an AWS-managed MQTT Bridge component. When this is also deployed, The Things Stack can use its Pub/Sub Integration to communicate with local devices, Greengrass components and AWS IoT Core; the best of all worlds.

By default, The Things Stack deploys an MQTT Server on port 8883. Prior to deploying the Moquette MQTT broker you must:

Remove the MQTT Server from The Things Stack configuration (and re-publish the configuration secret) OR
Move the MQTT Server to a port other than 8883 in The Things Stack configuration (and re-publish the configuration secret) OR
Decide to modify the Moquette MQTT broker component default configuration to run it on a port other than 8883

Next, update the Greengrass deployment to add the necessary components to your Greengrass core device:

As with connecting to AWS IoT Core, a Thing should be created in AWS IoT Core and the device certificate and private key should be saved. However, instead of the Amazon Root CA certificate being used for authenticating AWS IoT Core, you instead need the CA generated by the component installations; this can be found at /greengrass/v2/work/aws.greengrass.clientdevices.Auth/ca.pem on your Greengrass device.

Instead of connecting to the AWS IoT Core endpoint, the Pub/Sub integration shall connect to the local Moquette broker using its IP address (not localhost).

Add a Pub/Sub integration in your Application in The Things Stack with the following settings:

Item	Value
Provider	MQTT
MQTT Configuration: Use secure connection	Enabled
MQTT Configuration: Root CA certificate	ca.pem copied from your Greengrasss core device
MQTT Configuration: Client certificate	Device certificate saved when creating the Thing in AWS IoT Core
MQTT Configuration: Client private key	Private key saved when creating the Thing in AWS IoT Core
MQTT Configuration: Server URL	mqtts://xxx.xxx.xxx.xxx:8883 (assuming Moquette runs on 8883)

Operations

Clean Uninstall

Removing this component from your deployment will not remove all vestiges from your Greengrass core device. Additional steps:

Remove any Docker images that have persisted.
Remove the working directory: /greengrass/v2/work/aws.greengrass.labs.TheThingsStackLoRaWAN. This also deletes the databases.

Data Backup

If this component is deployed with default settings, the databases are located in /greengrass/v2/work/aws.greengrass.labs.TheThingsStackLoRaWAN/.env/data.

Troubleshooting

Tips for investigating failed deployments, or deployments that succeed but the Things Stack is still not running.

Troubleshooting Tools

Core Device Log Files

Detailed component logs can be found on the Core Device in /greengrass/v2/logs/aws.greengrass.labs.TheThingsStackLoRaWAN.log.

The Greengrass Core log file can be found at /greengrass/v2/logs/greengrass.log.

For more information please refer to the Greengrass V2 documentation: https://docs.aws.amazon.com/greengrass/v2/developerguide/monitor-logs.html

Greengrass CLI

Consider to install the Greengrass Command Line Interface component to obtain greater visibility into the state of your core device.

Docker Container Logs

The logs within the Docker containers can be inspected as follows:

docker logs awsgreengrasslabsthethingsstacklorawan_redis_1
docker logs awsgreengrasslabsthethingsstacklorawan_postgres_1
docker logs awsgreengrasslabsthethingsstacklorawan_cockroach_1
docker logs awsgreengrasslabsthethingsstacklorawan_stack

Common Failures

The Things Stack can fail to start for a variety of reasons.

Wrong Docker Image Architecture

If a Docker image of the wrong architecture is deployed, it will fail to start. A message similar to the following indicates that the wrong architecture is being used:

standard_init_linux.go:228: exec user process caused: exec format error

In the case of The Things Stack container, this message will appear in /greengrass/v2/logs/aws.greengrass.labs.TheThingsStackLoRaWAN.log. For the database containers, this message will appear in the Docker container logs. Other consequential errors will appear in /greengrass/v2/logs/aws.greengrass.labs.TheThingsStackLoRaWAN.log. For example, if Cockroach (which only supports amd64 architecture) is deployed to an Arm system, errors similar to the following will be observed:

aws.greengrass.labs.TheThingsStackLoRaWAN: stdout. ERROR: for awsgreengrasslabsthethingsstacklorawan_cockroach_1  Cannot start service cockroach
aws.greengrass.labs.TheThingsStackLoRaWAN: stdout. ERROR: for cockroach  Cannot start service cockroach
aws.greengrass.labs.TheThingsStackLoRaWAN: stdout. dial tcp 172.21.0.4:26257: connect: no route to host

To resolve incorrect architecture, please check the available architectures for the image tag. Image tags on DockerHub do not always support all architectures. Update docker-compose.yml and update the configuration secret.

Redis Keeps Restarting

This will prevent The Things Stack from starting correctly. Please check the Redis Docker container logs for the following error:

redis-server: monotonic.c:149: monotonicInit_posix: Assertion `rc == 0' failed.

Redis 6.2 included a change that means it requires libseccomp 2.4.2 or newer. On some systems, such as Raspbian, libseccomp may not meet this requirement. To check the version under Raspbian, Ubuntu or Debian:

dpkg -l | grep libseccomp

The issue can be resolved by using Redis 6.0 or by upgrading libseccomp.

Configuration Changes Not Deployed

The Greengrass Secret Manager component needs to fetch the configuration secret from the cloud, for any changes to be seen by The Things Stack LoRaWAN component. The Secret Manager will not necessarily fetch the secret even when a new version of the component is deployed. Restart or reboot the core device to force a fetch.

The deployed configuration can be found at /greengrass/v2/work/aws.greengrass.labs.TheThingsStackLoRaWAN/docker-compose.yml and /greengrass/v2/work/aws.greengrass.labs.TheThingsStackLoRaWAN/config/stack/ttn-lw-stack-docker.yml.

Database Access Fails

The following error may appear in /greengrass/v2/logs/aws.greengrass.labs.TheThingsStackLoRaWAN.log when The Things Stack starts up:

pq: could not open file "global/pg_filenode.map": Permission denied.

Or alternatively:

pq: database "ttn_lorawan" does not exist

These messages should only appear when trying to recover from other errors. Ensure the latest configuration secret is deployed by restarting the core device. Should the problem persist, consider a Clean Uninstall.

Development

Static Analysis

Static analysis is performed using Pylint. Example execution:

pylint artifacts libs tests *.py

Unit Tests

Unit tests are performed using pytest and moto.