From 434c5f2a48119abbe66c1d3edc26fb179f35e528 Mon Sep 17 00:00:00 2001 From: "tomas.zdara" Date: Wed, 7 Jun 2023 11:51:19 +0200 Subject: [PATCH 1/2] DSET-3995 refactor - add mermaid graph --- README.md | 35 ++++++++++++++++++++++++++++++++++- 1 file changed, 34 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 556b0e1..473f933 100644 --- a/README.md +++ b/README.md @@ -101,4 +101,37 @@ present and future, pursuant to the license of the project. TODO - add explanation why we have this architecture - pub/sub, events, buffers, grouping, sessions, DataSet addEvents session limitations, etc. - - convert existing diagram into PlantUML (/docs/img/overview.puml) and embed generated image (/docs/img/overview.png) here +```mermaid +graph TB +subgraph "DataSet" + DataSetApi["/api/addEvents REST endpoint"] +end +subgraph Collector[OpenTelemetry collector] + collector[Processing data] + collector --"consume traces"--> convTraces[Converts OTel structure into events] + collector --"consume logs"--> convLogs[Converts OTel structure into events] + + subgraph DataSetExporter + Publisher[Publishes events into topic based on group_by] + convLogs --> Publisher --propagates errors so that new data are rejected--> convLogs + convTraces --> Publisher --propagates errors so that new data are rejected--> convTraces + + subgraph "DataSet Go library" + PubSubEvents[Pub/Sub for events - topic based on group_by] + EventsConsumer[consumes events, stores them in buffers, publishes buffer in topic based on group_by] + PubSubBuffers[Pub/Sub for buffers - topic based on group_by] + BufferConsumer[consumes buffers, calls DataSet API, retries in case of error] + BufferSweeper[Buffer sweeper - publishes old buffers] + + Publisher --> PubSubEvents + PubSubEvents --"DataSetClient.ListenAndSendBundlesForKey"--> EventsConsumer + EventsConsumer --> PubSubBuffers + EventsConsumer --> BufferSweeper + BufferSweeper --> PubSubBuffers + PubSubBuffers --"DataSetClient.SendAddEventsBuffer"--> BufferConsumer + BufferConsumer --"propagates errors"--> Publisher + BufferConsumer -."use HTTP POST".-> DataSetApi + end + end +end +``` From 1bab7984e14c20ca9ebac74b3eb8736da6e75b23 Mon Sep 17 00:00:00 2001 From: "tomas.zdara" Date: Wed, 7 Jun 2023 11:54:52 +0200 Subject: [PATCH 2/2] DSET-3995 refactor - add mermaid graph --- README.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 473f933..d492c0f 100644 --- a/README.md +++ b/README.md @@ -103,7 +103,7 @@ TODO - add explanation why we have this architecture - pub/sub, events, buffers, grouping, sessions, DataSet addEvents session limitations, etc. ```mermaid graph TB -subgraph "DataSet" +subgraph "DataSet API servers" DataSetApi["/api/addEvents REST endpoint"] end subgraph Collector[OpenTelemetry collector] @@ -113,10 +113,10 @@ subgraph Collector[OpenTelemetry collector] subgraph DataSetExporter Publisher[Publishes events into topic based on group_by] - convLogs --> Publisher --propagates errors so that new data are rejected--> convLogs - convTraces --> Publisher --propagates errors so that new data are rejected--> convTraces + convLogs --DataSetClient.AddEvents--> Publisher --propagates errors so that new data are rejected--> convLogs + convTraces --DataSetClient.AddEvents--> Publisher --propagates errors so that new data are rejected--> convTraces - subgraph "DataSet Go library" + subgraph "DataSet Go - Library" PubSubEvents[Pub/Sub for events - topic based on group_by] EventsConsumer[consumes events, stores them in buffers, publishes buffer in topic based on group_by] PubSubBuffers[Pub/Sub for buffers - topic based on group_by] @@ -130,7 +130,7 @@ subgraph Collector[OpenTelemetry collector] BufferSweeper --> PubSubBuffers PubSubBuffers --"DataSetClient.SendAddEventsBuffer"--> BufferConsumer BufferConsumer --"propagates errors"--> Publisher - BufferConsumer -."use HTTP POST".-> DataSetApi + BufferConsumer -."HTTP POST addEvents".-> DataSetApi end end end