Buz is a system for multi-protocol event collection, validation, annotation, and delivery.
It ships as a single lightweight binary for deployment flexibility.
Toss Buz into an AWS lambda function, GCP Cloud Run service, or K8s pod and start collecting events in minutes.
Buz supports multiple input protocols including:
It even hosts a pixel for use in constrained tracking environments.
SDK's are supported out of the box so you can point existing Snowplow Analytics or Cloudevents tracking directly to Buz and it will just work™.
Or point your in-house SDK to it using self-describing JSON.
Buz supports two dozen different destinations including:
- Redpanda
- Postgres
- Kinesis Firehose
- Google Pub/Sub
- Splunk
- AWS EventBridge
- TimescaleDB
- NATS
- ...and many more.
You can send events to one or more destinations, so fanning them out to where they need to Bee is simple. As is using Buz to migrate from one destination system to another.
Every incoming payload is validated in microseconds using JSON Schema.
If a payload doesn't conform to the associated schema, it is marked as such.
If a payload doesn't have an explicitly-associated schema (such as the case with arbitrary webhooks and pixels), payload contents are not validated. It is enveloped as arbitrary for downstream processing.
Buz ships with a lightweight schema registry that supports multiple schema backends including:
Schemas are cached locally once sourced from the configured backend. Cache ttl and maximum size are configurable bu have sane defaults.
Schemas are available via HTTP at /s/$PATH_TO_SCHEMA or /s/$SCHEMA_NAME, depending on the backend.
The onboard schema cache can be purged via a GET or POST to the /c/purge route.
Each incoming payload is wrapped in a lightweight envelope.
This envelope appends a bit of metadata such as isValid, buzTimestamp, schema vendor, namespace, version, an event uuid, the associated protocol, etc. Metadata is then used to power payload routing and sharding as well as namespace-level statistics.
As an example of an arbitrary pixel event, wrapped in said envelope:
{
"uuid": "1f9a7a20-8fa7-4179-a0c2-35a80783854a",
"timestamp": "2023-05-03T02:50:59.464042Z",
"buzTimestamp": "2023-05-03T02:50:59.464042Z",
"buzVersion": "x.x.dev",
"buzName": "buz-bootstrap",
"buzEnv": "development",
"protocol": "pixel",
"schema": "io.silverton/buz/pixel/arbitrary/v1.0.json",
"vendor": "io.silverton",
"namespace": "buz.pixel.arbitrary",
"version": "1.0",
"isValid": true,
"payload": {
"msg": "hello",
"subject": "zander"
}
}
It minimizes the typical infrastructure footprint of collecting events from many different sources and allows for low-infrastructure, highly-flexible rollouts.
Buz doesn't care what your existing systems look like or what you want them to look like in the future.
It helps with the "now", and helps get your infrastructure to where you'd like it to be (without another migration).
Buz aims to improve the lives of pipeline maintainers and drastically reduce long-term maintenance of event collection systems.
Roll it out fast, keep it going without much thought, and shut it off when it isn't doing anything.
(No, you don't need to talk to anyone. Though we're relatively friendly and there's a Discord if you want to...)
You'll need go on your machine. But don't need to be a gopher.
Clone:
$ git clone git@github.com:silverton-io/buz.git && cd buz
Bootstrap:
$ make bootstrap
Events will be sent to two sinks by default - colorized envelopes will be sent to stdout and sent to buz_events.json or buz_invalid_events.json files.
curl -X POST localhost:8080/cloudevents -H 'Content-Type:application/cloudevents+json' -d '{"dataschema":"io.silverton/buz/example/gettingStarted/v1.0.json", "data": {"userId": 10, "name": "you", "action": "didSomething"}}'
curl -X POST "localhost:8080/webhook" -H 'Content-Type:application/json' -d '{"arbitrary": "thing"}'
curl -X POST "localhost:8080/webhook/io.silverton/buz/example/generic/sample/v1.0" -H 'Content-Type:application/json' -d '{"id": "10"}'
curl -X POST "localhost:8080/self-describing" -H 'Content-Type:application/json' -d '{"payload": {"schema": "io.silverton/buz/example/generic/sample/v1.0.json", "data": {"id": "10"}}}'
localhost:8080/pixel?msg=hello&subject=world
localhost:8080/pixel/io.silverton/buz/example/generic/sample/v1.0?id=10
Quickstart documentation for setting up a lightweight streaming stack with Buz, a sample ui, nginx, Redpanda, and Kowl can be found here.
Buz can be deployed in a variety of ways. We've included end-to-end (terraformed) samples for AWS and GCP:
Full documentation can be found here.
🍻🐝