Osgate is an opinionated open source python implmentation of an IoT gateway; primarily aimed at data collection via sensors in an industrial context. The primary opinions are:
- Export to a local MQTT broker (bridged to a cloud broker) is the default way of getting data out. This ensures a cloud-agnostic and durable (via persistant sessions) data export strategy.
- Configuration is designed with a pull-first strategy. This ensures the burden of truth is placed on some aggregate system rather than individual gateways. Note that the gateway can be run using local configuration file as well (e.g., completely deployable without having set up a platform that includes remote configuration)..
- In a further bid to reduce complexity, no on-board database is used (the configuration is cached though to ensure durability). Similarly, persistance of data is ensured given the usage of a local MQTT broker and persistant sessions.
The project is inspired by the gateway built by Thingsboard. In short, the main pieces are:
- Device - represents a physical entity in the real world; usually a sensor of some kind.
- Connector - bridge on which device data walks to and from a device to the gateway. Facilitates all forms of communication, including commands send to the devices (e.g. mbus).
- Sink - resting place of data sourced from devices through connectors. Connectors use sinks to export and persist data.
- GatewayService - acting as controller of the gateway. Runs a jsonrpc server and delegates inbound rpc calls.
This is just to frame some form of commitment. Note that these are in no particular order.
- Inital Project setup with resource-2-class layout.
- Lock down data sourcing strategy across connectors (two kinds; listener and poller, e.g. mqtt vs. mbus).
- Durable exportService for efficent data export via MQTT cloud bridge.
- Durable Remote-first configuration with local fallback.
- Basic RPC management, including authentication by API key.
- Auth + SSL for Mqtt Brokers (both connectors and sinks)
- Allow configurable data transformation/filtering on a channel level.
- MQTT connector (cabable of multiple broker sources), including customisable topic/message matching.
- HTTP connector
- Modbus connector
- Metric-bus (Mbus) connector
- ZWAVE connector
- Docker
- Python >= 3.10
- Mosquitto - Implmentation relies on a locally running MQTT broker (for bridging with a cloud twin).
This part of guide assumes you the above dependencies are installed and in your path. Note that it is possible to run the gateway without having set up any cloud bridging (this should be done before any deployment). To get started:
-
Download via Git
git clone git@github.com:expan75/osgate && cd osgate
-
Create virutal environment and install dependencies
python3 -m venv env && source env/bin/activate && pip install -r requirements.txt
-
Start local MQTT broker (mosquitto)
# Option 1: Start via docker docker pull eclipse-mosquitto docker run -it --name mosquitto -p 1883:1883 eclipse-mosquitto docker stop mosquitto docker start mosquitto # use a volume if you want to use a configuration file AND/OR verify persisted messages! # Option 2: Start via local install (your paths might be different) bash /usr/local/opt/mosquitto/sbin/mosquitto -c /usr/local/etc/mosquitto/mosquitto.conf
-
Copy example configuration
cp ./examples/osgate.json osgate.json
-
Run the server in debug mode
python3 ./osgate/main -d
-
Alternatively, run tests via pytest
python -m test pytest
-
Alternatively, listen to the export topic(s) using mosquitto_sub to see exported data:
mosquitto_sub -h localhost -p 1883 -t telemetry/# -v
To contribute, you must first aquaint yourself with this gitbranching model. In short, develop contains the latest code changes that have gone through inital code review. The develop branch is what you as a developer will branch off of for new features etc.
For releases, a release branch is used. This release branch is effectively made up by tagged checkouts of develop at various points in time. For a new release, a checkout of develop with a certain tag is used, e.g. "1.0". This release branch is then tested thourougly, lastly merged into master, i.e. the deployment of the release.
For your commits, please try to write in imperative form and keep it under 50 characters. If more is needed, e.g. for a briefer explanation, write the first 50 characters and then proceed to add a new line and as many description lines (< 70 characters a line) as you please. For a brief rundown on why this matter, check out this excellent piece by Chris Beams.
As a developer, you do not need to worry to much about the release strategy when developing. What you should worry about is readability and traceability. In order to understand the changes made and features added, it is important to be clear and concise when defining your new branches.
Bad: git checkout -b login
Better: git checkout -b feature/authentication
Bad: git checkout -b bugfix
Better: git checkout -b bugfix/authentication-timeout
Once you have defined a new branch and committed changes to it that you're happy with, it is time to create a pull request. To do so, use the web UI on github.com for your branch. Remember, you want to merge into DEVELOP. See guide.
When you have passed code review, you are advised to squash your commits. This way every new feature will have a single commit which is easy to navigate, and revert if need be!
Regardless if you are getting reviewed or are reviewing, you should comply with the Google Python Style Guide. Note that any programming paradigm is allowed as long as you comply with the above style guide and your code is understandable. In addition to the style guide, this project relies on Black for automatic linting and code style.
pip install black
black ./osgate
Black does not yet cover all of python3.10 syntax, so if something is not detected correctly, you can always flag it:
# fmt: off
match protocol:
case "default":
return DefaultConnector(name, uuid, devices, sinks, connector_metadata)
case "mqtt":
return MqttConnector(name, uuid, devices, sinks, connector_metadata)
case _:
raise NotImplementedError(
f"No connector of protocol {protocol} exists! Perhaps there's a typo in the config."
)
# fmt: on
Insert brief explanation of CI/CD pipeline here.
<insert ci/cd here>