From 177bbdbe922ce1f837ed5b2582d7baa84cfbcb01 Mon Sep 17 00:00:00 2001 From: Santiago Carmuega Date: Thu, 14 Jul 2022 17:21:36 -0300 Subject: [PATCH] docs: Expand Redis sink section --- book/src/advanced/custom_network.md | 2 +- book/src/sinks/README.md | 2 -- book/src/sinks/redis_streams.md | 32 +++++++++++++++++++---------- 3 files changed, 22 insertions(+), 14 deletions(-) diff --git a/book/src/advanced/custom_network.md b/book/src/advanced/custom_network.md index abf5d753..5bfe8280 100644 --- a/book/src/advanced/custom_network.md +++ b/book/src/advanced/custom_network.md @@ -1,6 +1,6 @@ # Custom networks -Required configuration connect Oura to a custom network (aka: not mainnet/testnet). +Instructions on how to configure Oura for connecting to a custom network (aka: other than mainnet / testnet). ## Context diff --git a/book/src/sinks/README.md b/book/src/sinks/README.md index a0ecaae4..e4f00fa9 100644 --- a/book/src/sinks/README.md +++ b/book/src/sinks/README.md @@ -14,8 +14,6 @@ These are the existing sinks that are included as part the main _Oura_ codebase: - [AWS SQS](aws_sqs.md): a sink that sends each event as message to an AWS SQS queue. - [AWS Lamda](aws_lambda.md): a sink that invokes an AWS Lambda function for each event. - [AWS S3](aws_s3.md): a sink that saves the CBOR content of the blocks as an AWS S3 object. -- [GCP PubSub](gcp_pubsub.md): a sink that sends each event as a message to a google cloud PubSub topic. -- [GCP CloudFunction](gcp_cloudfunction.md): a sink that sends each event as JSON to a Cloud Function via HTTP. - [Redis Streams](redis_streams.md): a sink that sends each event into a Redis stream. New sinks are being developed, information will be added in this documentation to reflect the updated list. Contributions and feature request are welcome in our [Github Repo](https://github.com/txpipe/oura). \ No newline at end of file diff --git a/book/src/sinks/redis_streams.md b/book/src/sinks/redis_streams.md index 163d1825..68ddf363 100644 --- a/book/src/sinks/redis_streams.md +++ b/book/src/sinks/redis_streams.md @@ -1,35 +1,45 @@ # Redis Streams -A sink that implements a _Redis Stream_ producer. The sink allows different stream strategies. +A sink that outputs events into _Redis Stream_. -It is possible to send all Event to a single stream or create multiple streams, one for each event type. -Both modes use `-` as unique entry ID (redis stream standard). -With StreamStrategy `None` a single redis-stream is used for all events, a stream name can be defined by `stream_name`, the default stream name is `oura`. -StreamStrategy `ByEventType` creates its own redis-stream for each event type. By appling filters it is possible to define the streams which should be created. +_Redis Streams_ works as an append-only log where multiple consumers can read from the same queue while keeping independent offsets (as opposed to a PubSub topic where one subscriber affect the other). You can learn more about the _Streams_ feature in the official [Redis Documentation](https://redis.io/docs/manual/data-types/streams). -The sink will use fingerprints as keys, if fingerprints are active otherwise the event type name in lowercase is used. +This sink will process incoming events and send a JSON-encoded message of the payload for each one using the `XADD` command. The Redis instance can be local or remote. ## Configuration -_Single Stream Mode:_ +Example configuration that sends all events into a single stream named `mystream` of a Redis instance running in port 6379 of the localhost. ```toml [sink] type = "Redis" -redis_server = "redis://default:@127.0.0.1:6379/0" +redis_server = "redis://localhost:6379" stream_name = "mystream" -stream_strategy = "None" ``` -_Multi Stream Mode:_ +Example configuration that sends events into different streams (named by the type of event) of a Redis instance running in port 6379 of the localhost. + ```toml [sink] type = "Redis" -redis_server = "redis://default:@127.0.0.1:6379/0" +redis_server = "redis://localhost:6379" stream_strategy = "ByEventType" ``` +### Section: `sink` + - `type`: the literal value `Redis`. - `redis_server`: the redis server in the format `redis://[][:]@[:port][/]` - `stream_name` : the name of the redis stream for StreamStrategy `None`, default is "oura" if not specified - `stream_strategy` : `None` or `ByEventType` + +## Conventions + +It is possible to send all Event to a single stream or create multiple streams, one for each event type. By appling the [selection](/filters/selection) filter it is possible to define the streams which should be created. + +The sink uses the default Redis convention to define the unique entry ID for each message sent to the stream ( `-`). + +Messages in Redis Streams are required to be `hashes` (maps between the string fields and the string values). This sink will serialize the event into a single-entry map with the following parts: + +- `key`: the [fingerprint](/filters/fingerprint) value if available, or the event type name. +- `value`: the json-encoded payload of the event.