DSET-3951: Log also amount of transferred bytes

[martin-majlis-s1]

🥅 Goal

It looks that the commands for pulling dataset-latest does not work quite well. Let's update them.

🛠️ Solution

It looks to me, that the least error prone solution is to delete that branch first before we try to pull it. Then we can check it out and pull.

🏫 Testing

1. Go to the repository - dataset-exporter-opentelemetry
2. Update it - git pull; make submodule-pull
3. Go to the directory dataset-go and switch to the current branch - git checkout DSET-3951-benchmarking-log-transferred-bytes-and-throughput
4. Go to the root directory of that folder.
5. Build docker image - make docker-build
6. Go to reliability directory - scripts/reliability
7. Run test - ./test.sh
8. You can see, that log line showing amount of transferred bytes is there:

dataset-collector-rel     | 2023-05-30T15:42:42.093Z      info    client/client.go:320    Buffers' Queue Stats:   {"kind": "exporter", "data_type": "logs", "name": "dataset/logs", "processed": 1, "enqueued": 1, "dropped": 0, "waiting": 0}
dataset-collector-rel     | 2023-05-30T15:42:42.093Z     info    client/client.go:331    Events' Queue Stats:{"kind": "exporter", "data_type": "logs", "name": "dataset/logs", "processed": 3963, "enqueued": 3963, "waiting": 0}
dataset-collector-rel     | 2023-05-30T15:42:42.093Z      info    client/client.go:345    Transfer Stats: {"kind": "exporter", "data_type": "logs", "name": "dataset/logs", "bytesSentMB": 0.47244739532470703, "bytesAcceptedMB": 0.06749248504638672, "throughputMBpS": 0.0005624237685745855, "perBufferMB": 0.06749248504638672, "successRate": 0.14285887306771525}

[codecov-commenter]

Codecov Report

Merging #27 (dea7c71) into main (eca25cb) will decrease coverage by 0.75%.
The diff coverage is 94.29%.

@@            Coverage Diff             @@
##             main      #27      +/-   ##
==========================================
- Coverage   72.03%   71.29%   -0.75%     
==========================================
  Files          10       10              
  Lines        1298     1320      +22     
==========================================
+ Hits          935      941       +6     
- Misses        303      318      +15     
- Partials       60       61       +1     

Impacted Files	Coverage Δ
pkg/client/add_events.go	80.67% <60.00%> (-2.17%)	⬇️
internal/pkg/os_util/os_util.go	100.00% <100.00%> (ø)
pkg/buffer_config/buffer_config.go	52.07% <100.00%> (ø)
pkg/client/client.go	81.07% <100.00%> (-2.44%)	⬇️
pkg/config/config.go	88.61% <100.00%> (+3.80%)	⬆️

[martin-majlis-s1]

From the demo instance:

otel-col                 | 2023-05-30T16:29:22.433Z     info    client/client.go:328    Buffers' Queue Stats:   {"kind": "exporter", "data_type": "logs", "name": "dataset/bbb", "processed": 5, "enqueued": 5, "dropped": 0, "waiting": 0}
otel-col                 | 2023-05-30T16:29:22.433Z     info    client/client.go:339    Events' Queue Stats:    {"kind": "exporter", "data_type": "logs", "name": "dataset/bbb", "processed": 1816, "enqueued": 1816, "waiting": 0}
otel-col                 | 2023-05-30T16:29:22.433Z     info    client/client.go:353    Transfer Stats: {"kind": "exporter", "data_type": "logs", "name": "dataset/bbb", "bytesSentMB": 1.7470026016235352, "bytesAcceptedMB": 1.7470026016235352, "throughputMBpS": 0.029113767424345235, "perBufferMB": 0.34940052032470703, "successRate": 1}
otel-col                 | 2023-05-30T16:29:22.433Z     info    client/client.go:328    Buffers' Queue Stats:   {"kind": "exporter", "data_type": "traces", "name": "dataset/aaa-tracing", "processed": 3, "enqueued": 3, "dropped": 0, "waiting": 0}
otel-col                 | 2023-05-30T16:29:22.433Z     info    client/client.go:339    Events' Queue Stats:    {"kind": "exporter", "data_type": "traces", "name": "dataset/aaa-tracing", "processed": 1335, "enqueued": 1335, "waiting": 0}
otel-col                 | 2023-05-30T16:29:22.434Z     info    client/client.go:353    Transfer Stats: {"kind": "exporter", "data_type": "traces", "name": "dataset/aaa-tracing", "bytesSentMB": 0.6831750869750977, "bytesAcceptedMB": 0.6831750869750977, "throughputMBpS": 0.011385420406876193, "perBufferMB": 0.22772502899169922, "successRate": 1}
otel-col                 | 2023-05-30T16:29:22.434Z     info    client/client.go:328    Buffers' Queue Stats:   {"kind": "exporter", "data_type": "traces", "name": "dataset/ccc-tracing", "processed": 3, "enqueued": 3, "dropped": 0, "waiting": 0}
otel-col                 | 2023-05-30T16:29:22.434Z     info    client/client.go:339    Events' Queue Stats:    {"kind": "exporter", "data_type": "traces", "name": "dataset/ccc-tracing", "processed": 1335, "enqueued": 1335, "waiting": 0}
otel-col                 | 2023-05-30T16:29:22.434Z     info    client/client.go:353    Transfer Stats: {"kind": "exporter", "data_type": "traces", "name": "dataset/ccc-tracing", "bytesSentMB": 0.6831750869750977, "bytesAcceptedMB": 0.6831750869750977, "throughputMBpS": 0.011385838934378157, "perBufferMB": 0.22772502899169922, "successRate": 1}
otel-col                 | 2023-05-30T16:29:22.467Z     info    client/client.go:328    Buffers' Queue Stats:   {"kind": "exporter", "data_type": "traces", "name": "dataset/bbb", "processed": 3, "enqueued": 3, "dropped": 0, "waiting": 0}
otel-col                 | 2023-05-30T16:29:22.467Z     info    client/client.go:339    Events' Queue Stats:    {"kind": "exporter", "data_type": "traces", "name": "dataset/bbb", "processed": 1335, "enqueued": 1335, "waiting": 0}
otel-col                 | 2023-05-30T16:29:22.467Z     info    client/client.go:353    Transfer Stats: {"kind": "exporter", "data_type": "traces", "name": "dataset/bbb", "bytesSentMB": 0.6831750869750977, "bytesAcceptedMB": 0.6831750869750977, "throughputMBpS": 0.011379832774586363, "perBufferMB": 0.22772502899169922, "successRate": 1}
otel-col                 | 2023-05-30T16:29:22.471Z     info    client/client.go:328    Buffers' Queue Stats:   {"kind": "exporter", "data_type": "logs", "name": "dataset/ccc-tracing", "processed": 5, "enqueued": 5, "dropped": 0, "waiting": 0}
otel-col                 | 2023-05-30T16:29:22.471Z     info    client/client.go:339    Events' Queue Stats:    {"kind": "exporter", "data_type": "logs", "name": "dataset/ccc-tracing", "processed": 1816, "enqueued": 1816, "waiting": 0}
otel-col                 | 2023-05-30T16:29:22.471Z     info    client/client.go:353    Transfer Stats: {"kind": "exporter", "data_type": "logs", "name": "dataset/ccc-tracing", "bytesSentMB": 1.7470026016235352, "bytesAcceptedMB": 1.7470026016235352, "throughputMBpS": 0.029115542837839785, "perBufferMB": 0.34940052032470703, "successRate": 1}
otel-col                 | 2023-05-30T16:29:22.476Z     info    client/client.go:328    Buffers' Queue Stats:   {"kind": "exporter", "data_type": "logs", "name": "dataset/aaa-tracing", "processed": 5, "enqueued": 5, "dropped": 0, "waiting": 0}
otel-col                 | 2023-05-30T16:29:22.476Z     info    client/client.go:339    Events' Queue Stats:    {"kind": "exporter", "data_type": "logs", "name": "dataset/aaa-tracing", "processed": 1816, "enqueued": 1816, "waiting": 0}
otel-col                 | 2023-05-30T16:29:22.476Z     info    client/client.go:353    Transfer Stats: {"kind": "exporter", "data_type": "logs", "name": "dataset/aaa-tracing", "bytesSentMB": 1.7470026016235352, "bytesAcceptedMB": 1.7470026016235352, "throughputMBpS": 0.029113305790744823, "perBufferMB": 0.34940052032470703, "successRate": 1}