Kirby is an extremely strong circle, and that's all the explanation I need.
Kirby Transform is middleware used to define a common data format and transformation logic to make it easier for me to build IOT pipelines. This transformation logic is split into:
The preprocessor is responsible for:
- Input validation
- Generating a set of data points, and metadata that is useful for tracking/debugging.
- Generating a common output format to be used down the line to send to specific data sources.
Outputs should be (mostly) self contained ways of generating data in the required format and sending data to storage solutions.
- InfluxDB2
- AWS Timestream
I'm finding that I'm spending a lot of time creating and maintaining IOT Device -> Data Sink pipelines compared to creating the devices themselves.
By ensuring that things will just work if I put the data in a common input format, I should be able to build the pipeline once and have something reasonably low maintenance.
Messages are expected to be valid JSON dictionaries
collector: String [Required]
Name of the collection source
data: List of dictonaries [Required]
The actual data to send. See below for what this looks like.
data_tags: Dict [Optional]
Dictionary of tags to associate with every data point
destination: String [Required]
Initial Destination (EG AWS IOT)
language: String [Required]
Language the code is written in
messages: List[dict] [Optional]
List of dictionaries with formatted as dict(fields,tags,timestamp). See the *data* section for more details.
meta_tags: Dict [Optional]
Dictonary of tags to associate with meta fields
platform: String [Required]
Platform the collector is running on (arduino, router etc)
timestamp: Unix Epoch [Optional]
If missing then this will be treated as when the data was recieved (Which is a really bad workaround).
uptime: Float [Optional]
Total uptime if this is something continously sending data
version: String [Required]
Version of the thing sending data
The actual datapoints come in as a list of dictionaries. The naming scheme is basically defined because InfluxDB is the first thing that I created this for.
tags: Dictionary [Optional]
Dictonary of tags to add to individual datapoints
fields: Dictonary [Required]
Datapoint fields.
timestamp: Unix Epoch [Optional]
Timestamp for the data (Resolution in seconds). If this is missing then the report timestamp will be usedThe best place to look for examples is in the tests folder. There are examples of data and how to use,
wrapped up as test cases.
As this is middleware for very inter related projects, this how I'm expecting to structure the project:
- This package is uploaded to PiPi
- Things that depend on this package (Eg Lambdas) should be put in their own repo, being submoduled to this one to make testing everything significantly easier.
- Dependent packages should be pinned to a specific Kirby Transform version, and should be updated as needed.