Use separate exporter per worker for better throughput

[marcelbirkner]

Description:
Adding a feature: Allow writing more metrics using "telemetrygen"

When running telemetrygen with multiple worker the exporter gets used by all workers. This results in limited throughput. On my machine that means about 250 metrics per worker.

With this code change each worker gets its own exporter. This allows scaling metrics throughput almost linear with the number of workers.

How to reproduce

Step 1: Install latest version of telemetrygen

go install github.com/open-telemetry/opentelemetry-collector-contrib/cmd/telemetrygen@latest

Step 2: Create metrics using 1 worker:

telemetrygen metrics \
    --duration 60s \
    --workers 1 \
    --otlp-http=false \
    --otlp-header authorization=\"Bearer\ AUTH TOKEN\" \
    --otlp-endpoint <HOSTNAME>:4317 \
    --otlp-attributes "service.name"=\"telemetrygen\"

Output

metrics/worker.go:67	metrics generated	{"worker": 0, "metrics": 1226}

Step 3: Create metrics using 10 worker:

telemetrygen metrics \
    --duration 60s \
    --workers 1 \
    --otlp-http=false \
    --otlp-header authorization=\"Bearer\ AUTH TOKEN\" \
    --otlp-endpoint <HOSTNAME>:4317 \
    --otlp-attributes "service.name"=\"telemetrygen\"

Output

metrics/worker.go:67	metrics generated	{"worker": 1, "metrics": 126}
metrics/worker.go:67	metrics generated	{"worker": 0, "metrics": 126}
metrics/worker.go:67	metrics generated	{"worker": 2, "metrics": 128}
metrics/worker.go:67	metrics generated	{"worker": 9, "metrics": 127}
metrics/worker.go:67	metrics generated	{"worker": 3, "metrics": 127}
metrics/worker.go:67	metrics generated	{"worker": 4, "metrics": 127}
metrics/worker.go:67	metrics generated	{"worker": 5, "metrics": 127}
metrics/worker.go:67	metrics generated	{"worker": 6, "metrics": 127}
metrics/worker.go:67	metrics generated	{"worker": 7, "metrics": 127}
metrics/worker.go:67	metrics generated	{"worker": 8, "metrics": 127}

Total number of metrics is the same, no matter how many workers we use.

Test results with code change

With this code change each worker can export about 1200-1300 metrics per second.

metrics/worker.go:119	metrics generated	{"worker": 4, "metrics": 1326}
metrics/worker.go:119	metrics generated	{"worker": 3, "metrics": 1286}
metrics/worker.go:119	metrics generated	{"worker": 9, "metrics": 1265}
metrics/worker.go:119	metrics generated	{"worker": 0, "metrics": 1240}
metrics/worker.go:119	metrics generated	{"worker": 7, "metrics": 1213}
metrics/worker.go:119	metrics generated	{"worker": 6, "metrics": 1235}
metrics/worker.go:119	metrics generated	{"worker": 8, "metrics": 1377}
metrics/worker.go:119	metrics generated	{"worker": 2, "metrics": 1212}
metrics/worker.go:119	metrics generated	{"worker": 1, "metrics": 1212}
metrics/worker.go:119	metrics generated	{"worker": 5, "metrics": 1172}