A concept project for monitoring and injesting data from MQTT sensors. This concept was created to test the viability of combining MQTT, react, and the InfluxDB database to persist and display sensor data with streaming support. Due to time constraints, only a prototype was built, missing key features, such as a timeline with playback, customisable timeframes, a dashboard builder, alerts, configuration storage etc. However, if you so desire, with minimal modifications you could create your own custom dashboard and display your own sensor data (even though the builder hasn't been created).
In the heart of this project lies the Eclipse Mosquitto broker.
The user can connect multiple MQTT sensors to that broker by supplying them
its credentials.
Then, provided the sensors use a specific publishing format (sensors/$group/$client/$sensor/$unit)
their data will be timestamped and saved automatically to a InfluxDB instance.
This requires no configuration by the user, sensors are completely plug and play.
A lightweight python api connects to both InfluxDB and the Broker and provides the frontend
with access to historical data and a websocket subscription for streaming directly from the MQTT
broker.
With a special /discovery endpoint, the frontend can ask the backend for sensors which
have been recorded lately.
Then, in the (future) builder the user can pick a sensor without knowing its exact publishing
endpoint.
Also, since with each point we store its group, client, sensor, and unit type we can average
the data for any of these tags.
For example, if the group refers to a room in a factory, and we pin sensor to temperature, by averaging the client, we could find the average temperature of the room.
Storing separate tags for the data doesn't increase cardinality or storage consumption since
they are only stored once for each time series by the InfluxDB.
The frontend is a React app with a redux model for its configuration. For websockets the socket.io protocol is used. To ingest and persist data from the broker, InfluxData's telegraf is used. This could be changed in the future. For user convenience, the docker-compose file contains a Cronograf instance to test out queries. To build the frontend files, webpack is used with a node container which will build and watch the files on startup. To host the files and provide reverse proxying for the api an NGINX instance is used.
The API providing data to the frontend is built with Python and Flask.
The file db.py contains an abstraction for communicating with the InfluxDB database,
using their new driver and Flux, so as to be futureproofed.
The file entry.py contains the API endpoints and the websocket code.
To enable streaming, it connects to the broker and subscribes to sensors/#.
Then, whenever it receives data from a sensor, it forwards it to the Socket.io room
sensors/$group/$client/$sensor/$unit, which sends it to the clients in that room.
When a dashboard wants to receive streaming data from a sensor it requests to be
added to that room.
That way, Socket.io completely handles client subscriptions and there are no memory leaks.
If a client disconnects, Socket.io will remove it from all of its rooms automatically.
The frontend is built using standard practices for a React app and with Typescript.
It is split into four folders: components, pages, styles, and store.
componentscontains the custom react components (for now the implemented Graphs)pagescontains the app Pages (Dashboard and Overview are implemented for now)stylescontains the SASS Styles fro the project which are split into partials (for components) and pages (for pages). The rest is standard SASS.storecontains the model of the app, the types, and the api abstraction.
In the redux store only the configuration of the app is stored (Dashboard, preferences).
It would be too slow to attempt to store the graph data in it.
The folders actions, reducers, and selectors contain the app's redux
functions and types the app's types.
The store/api folder contains the simple, for now, api abstraction that manages the
connection to the API.
It is built using the new react context mechanic so the Dashboard subcomponents can
access an instance of it.
They can then submit a callback with their graph information and start receiving data.
The data skips the redux store and is saved directly to their state using hooks
for performance.
On startup, as the Graphs register themselves to the API provider, the provider will make a Get request to fill them with data. Then, depending on the graph timespan, it will either register them for polling or for streaming. It is impractical to stream graphs that span hours and it would be too slow to refresh them at that speed, so they are always polled. Data that span minutes can either be streamed, or if disabled, polled as well.
For now, only two types of graphs are built. Line graphs and table "graphs". They both support multiple sensors and will adjust if more than one is added, adding a legend (line graph) or a series column (table). The line graphs also adjust the tick marks according to the width, and can set a title, min, max, adjustable timespans, and measurement unit.
A Graph is a component that displays the data of one or more sensors. It could be a graph, a table, or a guage. To define one, we can use the following data: