Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: Expand Redis sink section #366

Merged
merged 1 commit into from
Jul 14, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion book/src/advanced/custom_network.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Custom networks

Required configuration connect Oura to a custom network (aka: not mainnet/testnet).
Instructions on how to configure Oura for connecting to a custom network (aka: other than mainnet / testnet).

## Context

Expand Down
2 changes: 0 additions & 2 deletions book/src/sinks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,6 @@ These are the existing sinks that are included as part the main _Oura_ codebase:
- [AWS SQS](aws_sqs.md): a sink that sends each event as message to an AWS SQS queue.
- [AWS Lamda](aws_lambda.md): a sink that invokes an AWS Lambda function for each event.
- [AWS S3](aws_s3.md): a sink that saves the CBOR content of the blocks as an AWS S3 object.
- [GCP PubSub](gcp_pubsub.md): a sink that sends each event as a message to a google cloud PubSub topic.
- [GCP CloudFunction](gcp_cloudfunction.md): a sink that sends each event as JSON to a Cloud Function via HTTP.
- [Redis Streams](redis_streams.md): a sink that sends each event into a Redis stream.

New sinks are being developed, information will be added in this documentation to reflect the updated list. Contributions and feature request are welcome in our [Github Repo](https://github.com/txpipe/oura).
32 changes: 21 additions & 11 deletions book/src/sinks/redis_streams.md
Original file line number Diff line number Diff line change
@@ -1,35 +1,45 @@
# Redis Streams

A sink that implements a _Redis Stream_ producer. The sink allows different stream strategies.
A sink that outputs events into _Redis Stream_.

It is possible to send all Event to a single stream or create multiple streams, one for each event type.
Both modes use `<millisecondsTime>-<sequenceNumber>` as unique entry ID (redis stream standard).
With StreamStrategy `None` a single redis-stream is used for all events, a stream name can be defined by `stream_name`, the default stream name is `oura`.
StreamStrategy `ByEventType` creates its own redis-stream for each event type. By appling filters it is possible to define the streams which should be created.
_Redis Streams_ works as an append-only log where multiple consumers can read from the same queue while keeping independent offsets (as opposed to a PubSub topic where one subscriber affect the other). You can learn more about the _Streams_ feature in the official [Redis Documentation](https://redis.io/docs/manual/data-types/streams).

The sink will use fingerprints as keys, if fingerprints are active otherwise the event type name in lowercase is used.
This sink will process incoming events and send a JSON-encoded message of the payload for each one using the `XADD` command. The Redis instance can be local or remote.

## Configuration

_Single Stream Mode:_
Example configuration that sends all events into a single stream named `mystream` of a Redis instance running in port 6379 of the localhost.

```toml
[sink]
type = "Redis"
redis_server = "redis://default:@127.0.0.1:6379/0"
redis_server = "redis://localhost:6379"
stream_name = "mystream"
stream_strategy = "None"
```

_Multi Stream Mode:_
Example configuration that sends events into different streams (named by the type of event) of a Redis instance running in port 6379 of the localhost.

```toml
[sink]
type = "Redis"
redis_server = "redis://default:@127.0.0.1:6379/0"
redis_server = "redis://localhost:6379"
stream_strategy = "ByEventType"
```

### Section: `sink`

- `type`: the literal value `Redis`.
- `redis_server`: the redis server in the format `redis://[<username>][:<password>]@<hostname>[:port][/<db>]`
- `stream_name` : the name of the redis stream for StreamStrategy `None`, default is "oura" if not specified
- `stream_strategy` : `None` or `ByEventType`

## Conventions

It is possible to send all Event to a single stream or create multiple streams, one for each event type. By appling the [selection](/filters/selection) filter it is possible to define the streams which should be created.

The sink uses the default Redis convention to define the unique entry ID for each message sent to the stream ( `<millisecondsTime>-<sequenceNumber>`).

Messages in Redis Streams are required to be `hashes` (maps between the string fields and the string values). This sink will serialize the event into a single-entry map with the following parts:

- `key`: the [fingerprint](/filters/fingerprint) value if available, or the event type name.
- `value`: the json-encoded payload of the event.