This is the data ingestion service for the DtLab system. It unpacks incoming data from its incoming package.
Deconstructed data is closer to the domain- unbiased- and hence makes minimal assumptions on usage - easy to integrate/exploit/build services/self service on top of it
The Ingest Service accepts HTTP posts of JSON documents. The JSON will tend to be observations of the state of some thing in the world -
- a machine's engine temperature
- a retail sale transaction completion
- the new availability of funds from a bank account transaction
- a door opening
- a motion detector triggered
- an approaching asteroid's current speed
- etc...
The raw data announcing these events tends to be verbose and usually combines multiple observations into a single JSON document. The DTLab Ingest Service decomposes this raw JSON data into telemetry data - name, datetime, numerical value and forwards the telemetry data in a universal DTLab format to DTLab actors.
The steps for defining a new ingest process are:
- Define your types in the DTLab API - each value extracted from the incoming JSON must have a property defined in a DtType.
- Define the object extractor if your incoming data contains arrays of objects - loading batches of time-series data.
- Define the telemetry extractor
- use the forward name specified in the object extractor
- define groups of actor paths to the actors that will receive, remember, and act on the extracted telemetry
- define groups of properties to extract and send to the above paths
- Post your JSON
- possibly from a CRON cURL job polling NASA API hourly
- possibly from a Kafka consumer
- possibly from a MQTT consumer
widdershins --environment reference/env.json reference/dtlab-ingest-scala-alligator.v1.yaml reference/README.md