This project will deploy a serverless Asset Tracking infrastructure using Mapbox, AWS, and Pulumi.
This repository contains the following:
- Deployment code using Pulumi
- Node scripts to send sample events to the pipeline
- A sample web map to visualize the live location and status of events sent to the pipeline.
After deploying this solution, you will have see a sample live asset dashboard .
Before you begin, make sure you have the following tools configured:
You will need an AWS account in order to deploy the infrastructure via Pulumi. If you do not have one, please read the Getting Started with AWS.
Once you have an account, you can get more details on how to get your AWS credentials and set them for use with Pulumi by reading their setup instructions.
You do not need to have aws-cli installed, but it can help with setting up your AWS profile. Installation instructions for aws-cli can be found in its AWS documentation.
If you do not want to set up a terminal profile, or if you wish to set your credentials manually you can do so via your terminal.
export AWS_ACCESS_KEY_ID={my-aws-access-key}
export AWS_SECRET_ACCESS_KEY={my-aws-secret-key}If you already have your AWS profile configured in your terminal of choice, Pulumi will read those by default. The only warning is that your user (if using IAM) must have sufficient credentials to deploy the chosen infrastructure (listed below).
If you are using an IAM user for AWS credentials, your user must have permissions to the following services
- IoT
- Kinesis Data Streams
- Kinesis Firehose
- Lambda
- API Gateway
- DynamoDB
- S3
Pulumi is used to deploy the solution AWS infastructure. Follow the instructions below to install the Pulumi CLI, or consult the documentation for more details.
You will also need to sign up for a free account with Pulumi. Without one you will likely be prompted to create one when you run your first CLI command.
Requires homebrew to install.
brew install pulumiYou should also make sure you are on the latest version of Pulumi (>=3.5.1)
brew upgrade pulumiRun the following from cmd.exe
@"%SystemRoot%\System32\WindowsPowerShell\v1.0\powershell.exe" -NoProfile -InputFormat None -ExecutionPolicy Bypass -Command "[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12; iex ((New-Object System.Net.WebClient).DownloadString('https://get.pulumi.com/install.ps1'))" && SET "PATH=%PATH%;%USERPROFILE%\.pulumi\bin"If the above command fails, follow the manual install instructions from Pulumi's website docs.
Pulumi requires Node.js v12+. Check that your installed node version is >= v12 with node -v.
You need a valid Mapbox token to make requests to Mapbox APIs with a public (pk) token scope. Sign up for Mapbox here and find your access token here.
You can use your default token, or create one with only Public scopes.
Once you have cloned this repository, cd into the src directory and run npm ci. This will get the project bootstrapped and ready to deploy.
Once you have completed this and set your AWS credentials, run the following from inside src.
pulumi stack init testingFor more information on
stacks, see the Pulumi documentation.
Set your region (i.e. us-west-2)
pulumi config set aws:region <region>Set your Mapbox Access Token from above (i.e. pk.xyz...)
pulumi config set token <MapboxToken> --plaintextDeploy the stack
pulumi upOnce this is completed, you will be presented with a preview of your infrastructure, which you can choose to deploy.
Choose yes if you wish to deploy.
The solution code is located in src.
index.js: Core Pulumi infrastructure codeIoTHarness: Sample application that will push data into IoT Core for testing purposes.frontEnd: Sample HTML page that queries Dynamo and displays the current results on a map.
As referenced above, all deployments and updates are handled via the pulumi up command.
To remove all infratructure used in the stack, run pulumi destroy.
You can skip the preview state by passing a -y parameter to pulumi.
Your initial deployment will output the following
- An
index.jsfile inIoTHarnessto use in testing the pipeline. - Certificates for running the
IoTHarnessapplication. - An
index.htmlfile infrontEndfor viewing live data pushed into the pipeline from theIoTHarnessapplication.
Note: the
index.jsandindex.htmlfiles are written from theirinitcounterparts and will be re-written on every deploy.
This solution contains a testing Harness that will generate data and pass it to your IoT Core endpoint. Use the following commands to install and start sending data to the endpoint. This will also start a webserver to visualize the data you are sending.
cd IoTHarness
npm ci
npm startAfter starting the testing Harness, validate that the harness is sending sample data.
Alternatively, you can submit events into the assetingest topic via the IoT Core Console's Test page.
You will need to send data with the following minimum schema:
{
"id": number,
"coordinates": [float,float],
"timestamp": timestamp
...
}You can pass any other properties you like, but the listed ones are required.
The solution will deploy an API that queries Dynamo.
You can paste it directly into your browser to query Dynamo. If there isn't any data, you will see the following:
{ "message": "No assets are currently available." }If there is data in Dynamo, you will see a geoJSON Feature Collection. It will look similar to the sample below, but may have additional properties based on the data you have sent.
{
"type": "Feature",
"geometry": {
"type": "Point",
"coordinates": [125.6, 10.1]
},
"properties": {
"id": 1,
"elevation": 0.0,
"geofenceStatus":"INSIDE"
}
}If you want to visualize your data via a map you will need to starting the testing harness and then open localhost:5000 in a browser. The sample web page will query the Dynamo-backed API and display current asset positions on a Mapbox map.
This should result in the map below. The map shows the live asset location and asset metadata in a mouseover tooltip:
When an asset is inside one of the geofenced regions, the point-color will change from blue to red.
You can query the DynamoDB API endpoint to return all current asset locations and metadata. To get your stack API endpoint, run pulumi stack output url. Fetch the URL using your client of choice i.e. Chrome.
The Mapbox Stream processor also has an option to push data into an IoT topic for real-time display of data in a browser client. This functionality is enabled by default, through the frontend topic. In order to consume this topic in a browser, please consult the AWS documentation.
As part of the processing pipeline, location data is geofenced in real-time. This utilizes a sample polygon tileset whose shapes each have a name parameter. Every point is compared against that tileset via the Mapbox Tilequery API. If a point falls into one of these polygons, it is logged as INSIDE else OUTSIDE. The sample tileset is public and can be accessed with any Mapbox token.
If you would like to test with your own tilesets, you will need to follow these steps:
- Create a polygon tileset, ensuring it has a
nameattribute. - Upload it to your Mapbox account.
- Copy the tileset ID and paste it into the source code
- Re-deploy via
pulumi up. - Update your IoTHarness
route.jsonto include points that will fall into those geofences. - Run your IoTHarness with
node index.js.
Now that you have built your infrastructure - here are a few things to try next.
- Open up the AWS Console and trace the data through Cloudwatch. Watch it from ingestion, through Kinesis into Lambda, and into Dynamo and S3.
- Change the Lambda functions. The Stream Processor does the heavy lifing on the data. Make some updates by adding more API calls, perhaps add some new
npmpackages and insert that data into Dynamo. The Query Processor uses Turf to standardize the data - but there are a large number of other modules to do even more geospatial work. Try some out and send your map new data. - Create some new geofence tilesets via the Mapbox Datasets editor or upload your own. Update their properties and add that information to the stream via the stream processor.
- Experiment with adding an SNS resource and updating your Lambda to funnel geofence status into the notification service.
- Alter the scale. While most of these services are scalable by default (IoT Core, API Gateway, Dynamo), if you want to experiment with scaling further, change the number of Kinesis shards and/or adjust the Lambda batch size. This will help you with ingesting and processing more records.
- Deploy to another region. Create another stack and deploy again. Then try and find a way to deploy it to every AWS region!
- Explore Pulumi - use the stack graph to explore your resource dependencies and then try to add some more. You'll probably run ito IAM issues, so also take a glance at the AWS documentation as you build. Most of all - check out how their lambda functions are magic.
The core architecture works as follows:
- IoT Core: Data Ingestion
- Kinesis: Streaming Collection
- Lambda: Data Processing
- Mapbox
- Vector Maps: map dashboard
- Terrain RGB raster tiles: elevation
- Tilequery API: geofencing
- DynamoDB: Data storage
- Kinesis Firehose: Data archival
- S3: Storage
- API Gateway: Front-end query endpoint
- Turf: Data processing
- Pulumi: Infrastructure as Code
This solution was created by the Mapbox Solutions Architecture team.
This project is licensed under the BSD-3-Clause License - see the LICENSE file for details.
- Alex Yule for initial research and development
- Cyrus Najmabadi and Nishi Davidson from Pulumi