docs: Describe 'retry policy' mechanism

book/src/SUMMARY.md

@@ -36,6 +36,7 @@
     - [Mapper Options](./advanced/mapper_options.md)
     - [Intersect Options](./advanced/intersect_options.md)
     - [Custom Network](./advanced/custom_network.md)
+    - [Retry Policy](./advanced/retry_policy.md)
 - [Guides](./guides/README.md)
     - [Cardano => Kafka](./guides/cardano_2_kafka.md)
 

book/src/advanced/README.md

@@ -7,4 +7,5 @@ This section provides detailed information on the some of the advanced features
 - [Pipeline Metrics](./pipeline_metrics.md): allows operators to track the progress and performance of long-running Oura sessions.
 - [Mapper Options](./mapper_options.md): A set of "expensive" event mapping procedures that require an explicit opt-in to be activated.
 - [Intersect Options](./intersect_options.md): Advanced options for instructing Oura from which point in the chain to start reading from.
-- [Custom Network](./custom_network.md): Instructions on how to configure Oura for connecting to a custom network.
+- [Custom Network](./custom_network.md): Instructions on how to configure Oura for connecting to a custom network.
+- [Retry Policy](./retry_policy.md): Instructions on how to configure retry policies for different operations

book/src/advanced/retry_policy.md

@@ -7,6 +7,7 @@ Advanced options for instructing Oura how to deal with failed attempts in certai
 - [GCP CloudFunction](../sinks/gcp_cloudfunction.md)
 - [GCP PubSub](../sinks/gcp_pubsub.md)
 - [Webhook](../sinks/webhook.md)
+- [Elasticsearch](../sinks/elastic.md)
 
 ## Configuration
 

book/src/sinks/elastic.md

@@ -21,6 +21,8 @@ password = "my very secret stuff"
 - `type`: the literal value `Elastic`.
 - `url`: the location of the Elasticsearch's API
 - `index`: the name of the index (or data stream) to store the event documents
+- `idempotency` (optional): flag that if enabled will force idempotent calls to ES (to avoid duplicates)
+- `retry_policy` (optional): controls the policy to retry failed requests (see [retry policy](../advanced/retry_policy.md))
 
 ### Section: `sink.credentials`
 
@@ -31,3 +33,11 @@ We currently only implement _basic_ auth, other mechanisms will be implemented a
 - `type`: the mechanism to use for authentication, only `Basic` is currently implemented
 - `username`: username of the user with access to Elasticsearch
 - `password`: password of the user with access to Elasticsearch
+
+## Idempotency
+
+In services and API calls, _idempotency_ refers to a property of the system where the execution of multiple "equivalent" requests have the same effect as a single request. In other words, "idempotent" calls can be triggered multiple times without problem.
+
+In our Elasticsearch sink, when the `idempotency` flag is enabled, each document sent to the index will specify a particular content-based ID: the [fingerprint](../filters/fingerprint.md) of the event. If Oura restarts without having a cursor or if the same block is processed for any reason, repeated events will present the same ID and Elasticsearch will reject them and Oura will continue with the following event. This mechanism provides a strong guarantee that our index won't contain duplicate data.
+
+If the flag is disabled, each document will be generated using a random ID, ensuring that it will be indexed regardless.

book/src/sinks/webhook.md

@@ -34,4 +34,4 @@ extra_header_2 = "123"
 - `allow_invalid_certs` (optional): a flag to skip TLS cert validation (usually for self-signed certs).
 - `timeout` (optional): the timeout value for the HTTP response in milliseconds. Default value is `30000`.
 - `error_policy` (optional): either `Continue` or `Exit`. Default value is `Exit`.
-- [retry_policy](../advanced/retry_policy.md)
+- `retry_policy` (optional): controls the policy to retry failed requests (see [retry policy](../advanced/retry_policy.md))