Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Filelog receiver loses logs even if persistence is set up #31074

Open
OverOrion opened this issue Feb 6, 2024 · 31 comments
Open

Filelog receiver loses logs even if persistence is set up #31074

OverOrion opened this issue Feb 6, 2024 · 31 comments
Assignees
@andrzej-stencel
Labels
bug Something isn't working receiver/filelog release:required-for-ga Must be resolved before GA release

Comments

@OverOrion
Copy link
Contributor

OverOrion commented Feb 6, 2024 •
edited
Loading

Component(s)

extension/storage/filestorage, receiver/filelog

What happened?

Description

UPDATE: filelog receiver might be the culprit: #31074 (comment)
I tried the fault tolerant log collection example, but it seemingly loses logs (does not matter if the exporter is file or OTLP(GRPC)).
file_storage is enabled for receiver (and for the OTLP exporter as well), so the state should not be lost.
I tried the fileexporter and the logs are lost here as well, below are the results.

Steps to Reproduce

See gist for files: https://gist.github.com/OverOrion/2086516236e343ae2c362557302c70ce

$ cat input_gen.sh
#!/usr/bin/env bash

for i in $(seq 1 600000); do
  echo line$i
done

Steps:

  1. Generate a big enough input file for the collector to read:
./input_gen.sh > input.log
  1. Start the collector with the given configuration file (make sure the file receiver path points to input.log)
./otelcontribcol_linux_amd64 --config ./sender.yaml
  1. Watch the received logs (written by the exporter/file) with:
 watch -n 1 -d "cat sender_output.log | grep -E 'line[0-9]+' -o -w | wc -l"
  1. After some time stop the controller by pressing CTRL+C (sending a SIGINT)
  2. Restart the controller using the same configuration file
  3. Wait for it to finalize sending the logs and check the sender_output.log file:
cat sender_output.log | grep -E 'line[0-9]+' -o -w | wc -l
598887

Expected Result

All logs should be written, no logs should be lost.

Actual Result

In my case I consistently lose 1k+ logs out of 600k.

Collector version

8ae1539

Environment information

Environment

OS: Ubuntu 23.10
Compiler(if manually compiled): go1.21.6 linux/amd64

OpenTelemetry Collector configuration

extensions:
  file_storage/default:
    directory: /home/orion/work/others/open-telemetry/opentelemetry-collector-contrib/persist_dir/receivers
    fsync: true

receivers:
  filelog:
    include: ["/home/orion/work/others/open-telemetry/opentelemetry-collector-contrib/bin/input.log"]
    storage: file_storage/default
    start_at: beginning
    retry_on_failure:
      enabled: true

exporters:
  # Optional debug output
  debug:
    verbosity: detailed
  otlp/debug:
    endpoint: localhost:4317
    retry_on_failure:
      enabled: true
    sending_queue:
      queue_size: 10000
    tls:
      insecure: true

service:
  extensions: [file_storage/default]
  pipelines:
    logs:
      receivers: [filelog]
      exporters: [otlp/debug]

Log output

No response

Additional context

Can you please confirm that the given configuration should not lose any logs? EDIT: confirmed
I think this issue is related: #20552
I think it would be great to have an example config snippet that showcases

  • fault tolerance
  • and is E2E tested by the CI
@OverOrion OverOrion added bug Something isn't working needs triage New item requiring triage labels Feb 6, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Feb 6, 2024

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@mx-psi
Copy link
Member

mx-psi commented Feb 7, 2024

cc @swiatekm-sumo in case you can help here

@andrzej-stencel
Copy link
Member

andrzej-stencel commented Feb 7, 2024 •
edited
Loading

I'm not able to reproduce this

Updating my comment: I can reproduce this with both v0.92.0 and 8ae1539. I was missing the crucial step of stopping the collector and restarting it.

I'm on Ubuntu 23.10 as well.

Here's the configuration of the OTLP receiving collector for reference:

exporters:
  debug:
receivers:
  otlp:
    protocols:
      grpc:
service:
  pipelines:
    logs:
      exporters:
      - debug
      receivers:
      - otlp
  telemetry:
    metrics:
      address: localhost:8889

Will look into this in more detail. Thanks for raising this @OverOrion!

@andrzej-stencel andrzej-stencel removed the needs triage New item requiring triage label Feb 7, 2024
@OverOrion
Copy link
Contributor Author

Thanks @astencel-sumo for your fast response. As we've discussed during the SIG meeting, I collected the metrics as well.

Sender metrics

Before stopping it

# HELP otelcol_exporter_queue_capacity Fixed capacity of the retry queue (in batches)
# TYPE otelcol_exporter_queue_capacity gauge
otelcol_exporter_queue_capacity{exporter="otlp/debug",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev"} 10000
# HELP otelcol_exporter_queue_size Current size of the retry queue (in batches)
# TYPE otelcol_exporter_queue_size gauge
otelcol_exporter_queue_size{exporter="otlp/debug",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev"} 1
# HELP otelcol_exporter_send_failed_log_records Number of log records in failed attempts to send to destination.
# TYPE otelcol_exporter_send_failed_log_records counter
otelcol_exporter_send_failed_log_records{exporter="otlp/debug",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev"} 0
# HELP otelcol_exporter_sent_log_records Number of log record successfully sent to destination.
# TYPE otelcol_exporter_sent_log_records counter
otelcol_exporter_sent_log_records{exporter="otlp/debug",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev"} 72251
# HELP otelcol_process_cpu_seconds Total CPU user and system time in seconds
# TYPE otelcol_process_cpu_seconds counter
otelcol_process_cpu_seconds{service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev"} 2.66
# HELP otelcol_process_memory_rss Total physical memory (resident set size)
# TYPE otelcol_process_memory_rss gauge
otelcol_process_memory_rss{service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev"} 1.83762944e+08
# HELP otelcol_process_runtime_heap_alloc_bytes Bytes of allocated heap objects (see 'go doc runtime.MemStats.HeapAlloc')
# TYPE otelcol_process_runtime_heap_alloc_bytes gauge
otelcol_process_runtime_heap_alloc_bytes{service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev"} 5.1445392e+07
# HELP otelcol_process_runtime_total_alloc_bytes Cumulative bytes allocated for heap objects (see 'go doc runtime.MemStats.TotalAlloc')
# TYPE otelcol_process_runtime_total_alloc_bytes counter
otelcol_process_runtime_total_alloc_bytes{service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev"} 2.20852816e+08
# HELP otelcol_process_runtime_total_sys_memory_bytes Total bytes of memory obtained from the OS (see 'go doc runtime.MemStats.Sys')
# TYPE otelcol_process_runtime_total_sys_memory_bytes gauge
otelcol_process_runtime_total_sys_memory_bytes{service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev"} 1.03866648e+08
# HELP otelcol_process_uptime Uptime of the process
# TYPE otelcol_process_uptime counter
otelcol_process_uptime{service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev"} 2.938275694
# HELP otelcol_receiver_accepted_log_records Number of log records successfully pushed into the pipeline.
# TYPE otelcol_receiver_accepted_log_records counter
otelcol_receiver_accepted_log_records{receiver="filelog",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",transport=""} 72351
# HELP otelcol_receiver_refused_log_records Number of log records that could not be pushed into the pipeline.
# TYPE otelcol_receiver_refused_log_records counter
otelcol_receiver_refused_log_records{receiver="filelog",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",transport=""} 0
# HELP otelcol_rpc_client_duration Measures the duration of inbound RPC.
# TYPE otelcol_rpc_client_duration histogram
otelcol_rpc_client_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="0"} 0
otelcol_rpc_client_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="5"} 708
otelcol_rpc_client_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="10"} 723
otelcol_rpc_client_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="25"} 723
otelcol_rpc_client_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="50"} 723
otelcol_rpc_client_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="75"} 723
otelcol_rpc_client_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="100"} 723
otelcol_rpc_client_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="250"} 723
otelcol_rpc_client_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="500"} 723
otelcol_rpc_client_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="750"} 723
otelcol_rpc_client_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="1000"} 723
otelcol_rpc_client_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="2500"} 723
otelcol_rpc_client_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="5000"} 723
otelcol_rpc_client_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="7500"} 723
otelcol_rpc_client_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="10000"} 723
otelcol_rpc_client_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="+Inf"} 723
otelcol_rpc_client_duration_sum{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev"} 2310.766873
otelcol_rpc_client_duration_count{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev"} 723
# HELP otelcol_rpc_client_request_size Measures size of RPC request messages (uncompressed).
# TYPE otelcol_rpc_client_request_size histogram
otelcol_rpc_client_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="0"} 0
otelcol_rpc_client_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="5"} 723
otelcol_rpc_client_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="10"} 723
otelcol_rpc_client_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="25"} 723
otelcol_rpc_client_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="50"} 723
otelcol_rpc_client_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="75"} 723
otelcol_rpc_client_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="100"} 723
otelcol_rpc_client_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="250"} 723
otelcol_rpc_client_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="500"} 723
otelcol_rpc_client_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="750"} 723
otelcol_rpc_client_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="1000"} 723
otelcol_rpc_client_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="2500"} 723
otelcol_rpc_client_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="5000"} 723
otelcol_rpc_client_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="7500"} 723
otelcol_rpc_client_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="10000"} 723
otelcol_rpc_client_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="+Inf"} 723
otelcol_rpc_client_request_size_sum{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev"} 1446
otelcol_rpc_client_request_size_count{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev"} 723
# HELP otelcol_rpc_client_requests_per_rpc Measures the number of messages received per RPC. Should be 1 for all non-streaming RPCs.
# TYPE otelcol_rpc_client_requests_per_rpc histogram
otelcol_rpc_client_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="0"} 0
otelcol_rpc_client_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="5"} 723
otelcol_rpc_client_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="10"} 723
otelcol_rpc_client_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="25"} 723
otelcol_rpc_client_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="50"} 723
otelcol_rpc_client_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="75"} 723
otelcol_rpc_client_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="100"} 723
otelcol_rpc_client_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="250"} 723
otelcol_rpc_client_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="500"} 723
otelcol_rpc_client_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="750"} 723
otelcol_rpc_client_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="1000"} 723
otelcol_rpc_client_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="2500"} 723
otelcol_rpc_client_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="5000"} 723
otelcol_rpc_client_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="7500"} 723
otelcol_rpc_client_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="10000"} 723
otelcol_rpc_client_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="+Inf"} 723
otelcol_rpc_client_requests_per_rpc_sum{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev"} 723
otelcol_rpc_client_requests_per_rpc_count{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev"} 723
# HELP otelcol_rpc_client_response_size Measures size of RPC response messages (uncompressed).
# TYPE otelcol_rpc_client_response_size histogram
otelcol_rpc_client_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="0"} 0
otelcol_rpc_client_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="5"} 0
otelcol_rpc_client_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="10"} 0
otelcol_rpc_client_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="25"} 0
otelcol_rpc_client_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="50"} 0
otelcol_rpc_client_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="75"} 0
otelcol_rpc_client_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="100"} 0
otelcol_rpc_client_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="250"} 0
otelcol_rpc_client_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="500"} 0
otelcol_rpc_client_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="750"} 0
otelcol_rpc_client_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="1000"} 0
otelcol_rpc_client_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="2500"} 0
otelcol_rpc_client_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="5000"} 1
otelcol_rpc_client_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="7500"} 724
otelcol_rpc_client_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="10000"} 724
otelcol_rpc_client_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="+Inf"} 724
otelcol_rpc_client_response_size_sum{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev"} 4.192492e+06
otelcol_rpc_client_response_size_count{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev"} 724
# HELP otelcol_rpc_client_responses_per_rpc Measures the number of messages received per RPC. Should be 1 for all non-streaming RPCs.
# TYPE otelcol_rpc_client_responses_per_rpc histogram
otelcol_rpc_client_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="0"} 0
otelcol_rpc_client_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="5"} 723
otelcol_rpc_client_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="10"} 723
otelcol_rpc_client_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="25"} 723
otelcol_rpc_client_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="50"} 723
otelcol_rpc_client_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="75"} 723
otelcol_rpc_client_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="100"} 723
otelcol_rpc_client_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="250"} 723
otelcol_rpc_client_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="500"} 723
otelcol_rpc_client_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="750"} 723
otelcol_rpc_client_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="1000"} 723
otelcol_rpc_client_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="2500"} 723
otelcol_rpc_client_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="5000"} 723
otelcol_rpc_client_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="7500"} 723
otelcol_rpc_client_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="10000"} 723
otelcol_rpc_client_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev",le="+Inf"} 723
otelcol_rpc_client_responses_per_rpc_sum{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev"} 723
otelcol_rpc_client_responses_per_rpc_count{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev"} 723
# HELP target_info Target metadata
# TYPE target_info gauge
target_info{service_instance_id="2430ec76-4140-446e-9d3c-a1ca5e818657",service_name="otelcontribcol",service_version="0.93.0-dev"} 1

After stopping and restarting it

# HELP otelcol_exporter_queue_capacity Fixed capacity of the retry queue (in batches)
# TYPE otelcol_exporter_queue_capacity gauge
otelcol_exporter_queue_capacity{exporter="otlp/debug",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev"} 10000
# HELP otelcol_exporter_queue_size Current size of the retry queue (in batches)
# TYPE otelcol_exporter_queue_size gauge
otelcol_exporter_queue_size{exporter="otlp/debug",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev"} 1
# HELP otelcol_exporter_send_failed_log_records Number of log records in failed attempts to send to destination.
# TYPE otelcol_exporter_send_failed_log_records counter
otelcol_exporter_send_failed_log_records{exporter="otlp/debug",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev"} 0
# HELP otelcol_exporter_sent_log_records Number of log record successfully sent to destination.
# TYPE otelcol_exporter_sent_log_records counter
otelcol_exporter_sent_log_records{exporter="otlp/debug",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev"} 66860
# HELP otelcol_process_cpu_seconds Total CPU user and system time in seconds
# TYPE otelcol_process_cpu_seconds counter
otelcol_process_cpu_seconds{service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev"} 2.31
# HELP otelcol_process_memory_rss Total physical memory (resident set size)
# TYPE otelcol_process_memory_rss gauge
otelcol_process_memory_rss{service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev"} 1.79306496e+08
# HELP otelcol_process_runtime_heap_alloc_bytes Bytes of allocated heap objects (see 'go doc runtime.MemStats.HeapAlloc')
# TYPE otelcol_process_runtime_heap_alloc_bytes gauge
otelcol_process_runtime_heap_alloc_bytes{service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev"} 7.2674888e+07
# HELP otelcol_process_runtime_total_alloc_bytes Cumulative bytes allocated for heap objects (see 'go doc runtime.MemStats.TotalAlloc')
# TYPE otelcol_process_runtime_total_alloc_bytes counter
otelcol_process_runtime_total_alloc_bytes{service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev"} 2.1385068e+08
# HELP otelcol_process_runtime_total_sys_memory_bytes Total bytes of memory obtained from the OS (see 'go doc runtime.MemStats.Sys')
# TYPE otelcol_process_runtime_total_sys_memory_bytes gauge
otelcol_process_runtime_total_sys_memory_bytes{service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev"} 1.00131096e+08
# HELP otelcol_process_uptime Uptime of the process
# TYPE otelcol_process_uptime counter
otelcol_process_uptime{service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev"} 2.531517877
# HELP otelcol_receiver_accepted_log_records Number of log records successfully pushed into the pipeline.
# TYPE otelcol_receiver_accepted_log_records counter
otelcol_receiver_accepted_log_records{receiver="filelog",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",transport=""} 66960
# HELP otelcol_receiver_refused_log_records Number of log records that could not be pushed into the pipeline.
# TYPE otelcol_receiver_refused_log_records counter
otelcol_receiver_refused_log_records{receiver="filelog",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",transport=""} 0
# HELP otelcol_rpc_client_duration Measures the duration of inbound RPC.
# TYPE otelcol_rpc_client_duration histogram
otelcol_rpc_client_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="0"} 0
otelcol_rpc_client_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="5"} 657
otelcol_rpc_client_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="10"} 669
otelcol_rpc_client_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="25"} 669
otelcol_rpc_client_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="50"} 669
otelcol_rpc_client_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="75"} 669
otelcol_rpc_client_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="100"} 669
otelcol_rpc_client_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="250"} 669
otelcol_rpc_client_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="500"} 669
otelcol_rpc_client_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="750"} 669
otelcol_rpc_client_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="1000"} 669
otelcol_rpc_client_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="2500"} 669
otelcol_rpc_client_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="5000"} 669
otelcol_rpc_client_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="7500"} 669
otelcol_rpc_client_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="10000"} 669
otelcol_rpc_client_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="+Inf"} 669
otelcol_rpc_client_duration_sum{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev"} 2171.3251760000026
otelcol_rpc_client_duration_count{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev"} 669
# HELP otelcol_rpc_client_request_size Measures size of RPC request messages (uncompressed).
# TYPE otelcol_rpc_client_request_size histogram
otelcol_rpc_client_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="0"} 0
otelcol_rpc_client_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="5"} 669
otelcol_rpc_client_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="10"} 669
otelcol_rpc_client_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="25"} 669
otelcol_rpc_client_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="50"} 669
otelcol_rpc_client_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="75"} 669
otelcol_rpc_client_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="100"} 669
otelcol_rpc_client_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="250"} 669
otelcol_rpc_client_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="500"} 669
otelcol_rpc_client_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="750"} 669
otelcol_rpc_client_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="1000"} 669
otelcol_rpc_client_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="2500"} 669
otelcol_rpc_client_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="5000"} 669
otelcol_rpc_client_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="7500"} 669
otelcol_rpc_client_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="10000"} 669
otelcol_rpc_client_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="+Inf"} 669
otelcol_rpc_client_request_size_sum{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev"} 1338
otelcol_rpc_client_request_size_count{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev"} 669
# HELP otelcol_rpc_client_requests_per_rpc Measures the number of messages received per RPC. Should be 1 for all non-streaming RPCs.
# TYPE otelcol_rpc_client_requests_per_rpc histogram
otelcol_rpc_client_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="0"} 0
otelcol_rpc_client_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="5"} 669
otelcol_rpc_client_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="10"} 669
otelcol_rpc_client_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="25"} 669
otelcol_rpc_client_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="50"} 669
otelcol_rpc_client_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="75"} 669
otelcol_rpc_client_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="100"} 669
otelcol_rpc_client_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="250"} 669
otelcol_rpc_client_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="500"} 669
otelcol_rpc_client_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="750"} 669
otelcol_rpc_client_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="1000"} 669
otelcol_rpc_client_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="2500"} 669
otelcol_rpc_client_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="5000"} 669
otelcol_rpc_client_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="7500"} 669
otelcol_rpc_client_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="10000"} 669
otelcol_rpc_client_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="+Inf"} 669
otelcol_rpc_client_requests_per_rpc_sum{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev"} 669
otelcol_rpc_client_requests_per_rpc_count{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev"} 669
# HELP otelcol_rpc_client_response_size Measures size of RPC response messages (uncompressed).
# TYPE otelcol_rpc_client_response_size histogram
otelcol_rpc_client_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="0"} 0
otelcol_rpc_client_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="5"} 0
otelcol_rpc_client_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="10"} 0
otelcol_rpc_client_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="25"} 0
otelcol_rpc_client_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="50"} 0
otelcol_rpc_client_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="75"} 0
otelcol_rpc_client_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="100"} 0
otelcol_rpc_client_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="250"} 0
otelcol_rpc_client_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="500"} 0
otelcol_rpc_client_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="750"} 0
otelcol_rpc_client_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="1000"} 0
otelcol_rpc_client_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="2500"} 0
otelcol_rpc_client_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="5000"} 2
otelcol_rpc_client_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="7500"} 670
otelcol_rpc_client_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="10000"} 670
otelcol_rpc_client_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="+Inf"} 670
otelcol_rpc_client_response_size_sum{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev"} 3.95734e+06
otelcol_rpc_client_response_size_count{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev"} 670
# HELP otelcol_rpc_client_responses_per_rpc Measures the number of messages received per RPC. Should be 1 for all non-streaming RPCs.
# TYPE otelcol_rpc_client_responses_per_rpc histogram
otelcol_rpc_client_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="0"} 0
otelcol_rpc_client_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="5"} 669
otelcol_rpc_client_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="10"} 669
otelcol_rpc_client_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="25"} 669
otelcol_rpc_client_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="50"} 669
otelcol_rpc_client_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="75"} 669
otelcol_rpc_client_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="100"} 669
otelcol_rpc_client_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="250"} 669
otelcol_rpc_client_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="500"} 669
otelcol_rpc_client_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="750"} 669
otelcol_rpc_client_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="1000"} 669
otelcol_rpc_client_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="2500"} 669
otelcol_rpc_client_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="5000"} 669
otelcol_rpc_client_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="7500"} 669
otelcol_rpc_client_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="10000"} 669
otelcol_rpc_client_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev",le="+Inf"} 669
otelcol_rpc_client_responses_per_rpc_sum{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev"} 669
otelcol_rpc_client_responses_per_rpc_count{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev"} 669
# HELP target_info Target metadata
# TYPE target_info gauge
target_info{service_instance_id="c00a9386-d1a2-4e78-ba1e-c6e00788a022",service_name="otelcontribcol",service_version="0.93.0-dev"} 1

Receiver metrics

Before the stop/restart

# HELP otelcol_exporter_send_failed_log_records Number of log records in failed attempts to send to destination.
# TYPE otelcol_exporter_send_failed_log_records counter
otelcol_exporter_send_failed_log_records{exporter="file",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev"} 0
# HELP otelcol_exporter_sent_log_records Number of log record successfully sent to destination.
# TYPE otelcol_exporter_sent_log_records counter
otelcol_exporter_sent_log_records{exporter="file",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev"} 406828
# HELP otelcol_process_cpu_seconds Total CPU user and system time in seconds
# TYPE otelcol_process_cpu_seconds counter
otelcol_process_cpu_seconds{service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev"} 15.49
# HELP otelcol_process_memory_rss Total physical memory (resident set size)
# TYPE otelcol_process_memory_rss gauge
otelcol_process_memory_rss{service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev"} 1.8558976e+08
# HELP otelcol_process_runtime_heap_alloc_bytes Bytes of allocated heap objects (see 'go doc runtime.MemStats.HeapAlloc')
# TYPE otelcol_process_runtime_heap_alloc_bytes gauge
otelcol_process_runtime_heap_alloc_bytes{service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev"} 6.5002952e+07
# HELP otelcol_process_runtime_total_alloc_bytes Cumulative bytes allocated for heap objects (see 'go doc runtime.MemStats.TotalAlloc')
# TYPE otelcol_process_runtime_total_alloc_bytes counter
otelcol_process_runtime_total_alloc_bytes{service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev"} 3.233055752e+09
# HELP otelcol_process_runtime_total_sys_memory_bytes Total bytes of memory obtained from the OS (see 'go doc runtime.MemStats.Sys')
# TYPE otelcol_process_runtime_total_sys_memory_bytes gauge
otelcol_process_runtime_total_sys_memory_bytes{service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev"} 1.04784152e+08
# HELP otelcol_process_uptime Uptime of the process
# TYPE otelcol_process_uptime counter
otelcol_process_uptime{service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev"} 122.324911696
# HELP otelcol_receiver_accepted_log_records Number of log records successfully pushed into the pipeline.
# TYPE otelcol_receiver_accepted_log_records counter
otelcol_receiver_accepted_log_records{receiver="otlp",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",transport="grpc"} 406828
# HELP otelcol_receiver_refused_log_records Number of log records that could not be pushed into the pipeline.
# TYPE otelcol_receiver_refused_log_records counter
otelcol_receiver_refused_log_records{receiver="otlp",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",transport="grpc"} 0
# HELP otelcol_rpc_server_duration Measures the duration of inbound RPC.
# TYPE otelcol_rpc_server_duration histogram
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="0"} 0
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="5"} 4028
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="10"} 4074
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="25"} 4074
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="50"} 4074
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="75"} 4074
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="100"} 4074
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="250"} 4074
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="500"} 4074
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="750"} 4074
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="1000"} 4074
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="2500"} 4074
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="5000"} 4074
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="7500"} 4074
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="10000"} 4074
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="+Inf"} 4074
otelcol_rpc_server_duration_sum{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev"} 11692.281866999976
otelcol_rpc_server_duration_count{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev"} 4074
# HELP otelcol_rpc_server_request_size Measures size of RPC request messages (uncompressed).
# TYPE otelcol_rpc_server_request_size histogram
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="0"} 0
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="5"} 0
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="10"} 0
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="25"} 0
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="50"} 0
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="75"} 1
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="100"} 1
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="250"} 1
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="500"} 2
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="750"} 2
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="1000"} 2
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="2500"} 2
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="5000"} 13
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="7500"} 4076
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="10000"} 4076
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="+Inf"} 4076
otelcol_rpc_server_request_size_sum{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev"} 2.3944305e+07
otelcol_rpc_server_request_size_count{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev"} 4076
# HELP otelcol_rpc_server_requests_per_rpc Measures the number of messages received per RPC. Should be 1 for all non-streaming RPCs.
# TYPE otelcol_rpc_server_requests_per_rpc histogram
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="0"} 0
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="5"} 4074
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="10"} 4074
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="25"} 4074
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="50"} 4074
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="75"} 4074
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="100"} 4074
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="250"} 4074
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="500"} 4074
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="750"} 4074
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="1000"} 4074
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="2500"} 4074
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="5000"} 4074
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="7500"} 4074
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="10000"} 4074
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="+Inf"} 4074
otelcol_rpc_server_requests_per_rpc_sum{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev"} 4074
otelcol_rpc_server_requests_per_rpc_count{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev"} 4074
# HELP otelcol_rpc_server_response_size Measures size of RPC response messages (uncompressed).
# TYPE otelcol_rpc_server_response_size histogram
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="0"} 0
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="5"} 4074
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="10"} 4074
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="25"} 4074
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="50"} 4074
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="75"} 4074
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="100"} 4074
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="250"} 4074
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="500"} 4074
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="750"} 4074
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="1000"} 4074
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="2500"} 4074
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="5000"} 4074
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="7500"} 4074
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="10000"} 4074
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="+Inf"} 4074
otelcol_rpc_server_response_size_sum{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev"} 8148
otelcol_rpc_server_response_size_count{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev"} 4074
# HELP otelcol_rpc_server_responses_per_rpc Measures the number of messages received per RPC. Should be 1 for all non-streaming RPCs.
# TYPE otelcol_rpc_server_responses_per_rpc histogram
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="0"} 0
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="5"} 4074
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="10"} 4074
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="25"} 4074
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="50"} 4074
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="75"} 4074
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="100"} 4074
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="250"} 4074
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="500"} 4074
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="750"} 4074
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="1000"} 4074
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="2500"} 4074
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="5000"} 4074
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="7500"} 4074
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="10000"} 4074
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="+Inf"} 4074
otelcol_rpc_server_responses_per_rpc_sum{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev"} 4074
otelcol_rpc_server_responses_per_rpc_count{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev"} 4074
# HELP target_info Target metadata
# TYPE target_info gauge
target_info{service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev"} 1

After the restart (1)

# HELP otelcol_exporter_send_failed_log_records Number of log records in failed attempts to send to destination.
# TYPE otelcol_exporter_send_failed_log_records counter
otelcol_exporter_send_failed_log_records{exporter="file",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev"} 0
# HELP otelcol_exporter_sent_log_records Number of log record successfully sent to destination.
# TYPE otelcol_exporter_sent_log_records counter
otelcol_exporter_sent_log_records{exporter="file",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev"} 485785
# HELP otelcol_process_cpu_seconds Total CPU user and system time in seconds
# TYPE otelcol_process_cpu_seconds counter
otelcol_process_cpu_seconds{service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev"} 18.6
# HELP otelcol_process_memory_rss Total physical memory (resident set size)
# TYPE otelcol_process_memory_rss gauge
otelcol_process_memory_rss{service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev"} 1.86535936e+08
# HELP otelcol_process_runtime_heap_alloc_bytes Bytes of allocated heap objects (see 'go doc runtime.MemStats.HeapAlloc')
# TYPE otelcol_process_runtime_heap_alloc_bytes gauge
otelcol_process_runtime_heap_alloc_bytes{service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev"} 4.4095368e+07
# HELP otelcol_process_runtime_total_alloc_bytes Cumulative bytes allocated for heap objects (see 'go doc runtime.MemStats.TotalAlloc')
# TYPE otelcol_process_runtime_total_alloc_bytes counter
otelcol_process_runtime_total_alloc_bytes{service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev"} 3.855507632e+09
# HELP otelcol_process_runtime_total_sys_memory_bytes Total bytes of memory obtained from the OS (see 'go doc runtime.MemStats.Sys')
# TYPE otelcol_process_runtime_total_sys_memory_bytes gauge
otelcol_process_runtime_total_sys_memory_bytes{service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev"} 1.04784152e+08
# HELP otelcol_process_uptime Uptime of the process
# TYPE otelcol_process_uptime counter
otelcol_process_uptime{service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev"} 125.014252085
# HELP otelcol_receiver_accepted_log_records Number of log records successfully pushed into the pipeline.
# TYPE otelcol_receiver_accepted_log_records counter
otelcol_receiver_accepted_log_records{receiver="otlp",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",transport="grpc"} 485785
# HELP otelcol_receiver_refused_log_records Number of log records that could not be pushed into the pipeline.
# TYPE otelcol_receiver_refused_log_records counter
otelcol_receiver_refused_log_records{receiver="otlp",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",transport="grpc"} 0
# HELP otelcol_rpc_server_duration Measures the duration of inbound RPC.
# TYPE otelcol_rpc_server_duration histogram
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="0"} 0
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="5"} 4802
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="10"} 4864
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="25"} 4864
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="50"} 4864
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="75"} 4864
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="100"} 4864
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="250"} 4864
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="500"} 4864
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="750"} 4864
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="1000"} 4864
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="2500"} 4864
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="5000"} 4864
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="7500"} 4864
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="10000"} 4864
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="+Inf"} 4864
otelcol_rpc_server_duration_sum{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev"} 14079.032756999994
otelcol_rpc_server_duration_count{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev"} 4864
# HELP otelcol_rpc_server_request_size Measures size of RPC request messages (uncompressed).
# TYPE otelcol_rpc_server_request_size histogram
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="0"} 0
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="5"} 0
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="10"} 0
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="25"} 0
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="50"} 0
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="75"} 1
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="100"} 1
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="250"} 1
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="500"} 2
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="750"} 2
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="1000"} 2
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="2500"} 2
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="5000"} 14
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="7500"} 4865
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="10000"} 4865
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="+Inf"} 4865
otelcol_rpc_server_request_size_sum{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev"} 2.8604758e+07
otelcol_rpc_server_request_size_count{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev"} 4865
# HELP otelcol_rpc_server_requests_per_rpc Measures the number of messages received per RPC. Should be 1 for all non-streaming RPCs.
# TYPE otelcol_rpc_server_requests_per_rpc histogram
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="0"} 0
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="5"} 4864
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="10"} 4864
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="25"} 4864
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="50"} 4864
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="75"} 4864
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="100"} 4864
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="250"} 4864
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="500"} 4864
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="750"} 4864
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="1000"} 4864
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="2500"} 4864
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="5000"} 4864
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="7500"} 4864
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="10000"} 4864
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="+Inf"} 4864
otelcol_rpc_server_requests_per_rpc_sum{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev"} 4864
otelcol_rpc_server_requests_per_rpc_count{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev"} 4864
# HELP otelcol_rpc_server_response_size Measures size of RPC response messages (uncompressed).
# TYPE otelcol_rpc_server_response_size histogram
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="0"} 0
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="5"} 4864
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="10"} 4864
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="25"} 4864
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="50"} 4864
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="75"} 4864
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="100"} 4864
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="250"} 4864
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="500"} 4864
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="750"} 4864
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="1000"} 4864
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="2500"} 4864
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="5000"} 4864
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="7500"} 4864
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="10000"} 4864
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="+Inf"} 4864
otelcol_rpc_server_response_size_sum{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev"} 9728
otelcol_rpc_server_response_size_count{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev"} 4864
# HELP otelcol_rpc_server_responses_per_rpc Measures the number of messages received per RPC. Should be 1 for all non-streaming RPCs.
# TYPE otelcol_rpc_server_responses_per_rpc histogram
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="0"} 0
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="5"} 4864
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="10"} 4864
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="25"} 4864
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="50"} 4864
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="75"} 4864
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="100"} 4864
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="250"} 4864
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="500"} 4864
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="750"} 4864
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="1000"} 4864
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="2500"} 4864
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="5000"} 4864
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="7500"} 4864
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="10000"} 4864
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="+Inf"} 4864
otelcol_rpc_server_responses_per_rpc_sum{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev"} 4864
otelcol_rpc_server_responses_per_rpc_count{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev"} 4864
# HELP target_info Target metadata
# TYPE target_info gauge
target_info{service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev"} 1

After the restart (2)

# HELP otelcol_exporter_send_failed_log_records Number of log records in failed attempts to send to destination.
# TYPE otelcol_exporter_send_failed_log_records counter
otelcol_exporter_send_failed_log_records{exporter="file",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev"} 0
# HELP otelcol_exporter_sent_log_records Number of log record successfully sent to destination.
# TYPE otelcol_exporter_sent_log_records counter
otelcol_exporter_sent_log_records{exporter="file",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev"} 599400
# HELP otelcol_process_cpu_seconds Total CPU user and system time in seconds
# TYPE otelcol_process_cpu_seconds counter
otelcol_process_cpu_seconds{service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev"} 23.02
# HELP otelcol_process_memory_rss Total physical memory (resident set size)
# TYPE otelcol_process_memory_rss gauge
otelcol_process_memory_rss{service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev"} 1.85061376e+08
# HELP otelcol_process_runtime_heap_alloc_bytes Bytes of allocated heap objects (see 'go doc runtime.MemStats.HeapAlloc')
# TYPE otelcol_process_runtime_heap_alloc_bytes gauge
otelcol_process_runtime_heap_alloc_bytes{service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev"} 7.5246808e+07
# HELP otelcol_process_runtime_total_alloc_bytes Cumulative bytes allocated for heap objects (see 'go doc runtime.MemStats.TotalAlloc')
# TYPE otelcol_process_runtime_total_alloc_bytes counter
otelcol_process_runtime_total_alloc_bytes{service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev"} 4.75131948e+09
# HELP otelcol_process_runtime_total_sys_memory_bytes Total bytes of memory obtained from the OS (see 'go doc runtime.MemStats.Sys')
# TYPE otelcol_process_runtime_total_sys_memory_bytes gauge
otelcol_process_runtime_total_sys_memory_bytes{service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev"} 1.04784152e+08
# HELP otelcol_process_uptime Uptime of the process
# TYPE otelcol_process_uptime counter
otelcol_process_uptime{service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev"} 137.50533767
# HELP otelcol_receiver_accepted_log_records Number of log records successfully pushed into the pipeline.
# TYPE otelcol_receiver_accepted_log_records counter
otelcol_receiver_accepted_log_records{receiver="otlp",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",transport="grpc"} 599400
# HELP otelcol_receiver_refused_log_records Number of log records that could not be pushed into the pipeline.
# TYPE otelcol_receiver_refused_log_records counter
otelcol_receiver_refused_log_records{receiver="otlp",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",transport="grpc"} 0
# HELP otelcol_rpc_server_duration Measures the duration of inbound RPC.
# TYPE otelcol_rpc_server_duration histogram
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="0"} 0
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="5"} 5929
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="10"} 6003
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="25"} 6003
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="50"} 6003
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="75"} 6003
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="100"} 6003
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="250"} 6003
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="500"} 6003
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="750"} 6003
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="1000"} 6003
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="2500"} 6003
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="5000"} 6003
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="7500"} 6003
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="10000"} 6003
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="+Inf"} 6003
otelcol_rpc_server_duration_sum{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev"} 17375.22947799999
otelcol_rpc_server_duration_count{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev"} 6003
# HELP otelcol_rpc_server_request_size Measures size of RPC request messages (uncompressed).
# TYPE otelcol_rpc_server_request_size histogram
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="0"} 0
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="5"} 0
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="10"} 0
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="25"} 0
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="50"} 0
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="75"} 2
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="100"} 2
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="250"} 3
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="500"} 4
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="750"} 4
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="1000"} 4
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="2500"} 4
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="5000"} 18
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="7500"} 6003
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="10000"} 6003
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="+Inf"} 6003
otelcol_rpc_server_request_size_sum{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev"} 3.5313521e+07
otelcol_rpc_server_request_size_count{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev"} 6003
# HELP otelcol_rpc_server_requests_per_rpc Measures the number of messages received per RPC. Should be 1 for all non-streaming RPCs.
# TYPE otelcol_rpc_server_requests_per_rpc histogram
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="0"} 0
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="5"} 6003
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="10"} 6003
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="25"} 6003
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="50"} 6003
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="75"} 6003
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="100"} 6003
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="250"} 6003
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="500"} 6003
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="750"} 6003
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="1000"} 6003
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="2500"} 6003
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="5000"} 6003
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="7500"} 6003
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="10000"} 6003
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="+Inf"} 6003
otelcol_rpc_server_requests_per_rpc_sum{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev"} 6003
otelcol_rpc_server_requests_per_rpc_count{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev"} 6003
# HELP otelcol_rpc_server_response_size Measures size of RPC response messages (uncompressed).
# TYPE otelcol_rpc_server_response_size histogram
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="0"} 0
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="5"} 6003
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="10"} 6003
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="25"} 6003
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="50"} 6003
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="75"} 6003
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="100"} 6003
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="250"} 6003
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="500"} 6003
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="750"} 6003
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="1000"} 6003
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="2500"} 6003
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="5000"} 6003
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="7500"} 6003
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="10000"} 6003
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="+Inf"} 6003
otelcol_rpc_server_response_size_sum{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev"} 12006
otelcol_rpc_server_response_size_count{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev"} 6003
# HELP otelcol_rpc_server_responses_per_rpc Measures the number of messages received per RPC. Should be 1 for all non-streaming RPCs.
# TYPE otelcol_rpc_server_responses_per_rpc histogram
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="0"} 0
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="5"} 6003
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="10"} 6003
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="25"} 6003
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="50"} 6003
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="75"} 6003
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="100"} 6003
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="250"} 6003
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="500"} 6003
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="750"} 6003
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="1000"} 6003
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="2500"} 6003
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="5000"} 6003
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="7500"} 6003
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="10000"} 6003
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev",le="+Inf"} 6003
otelcol_rpc_server_responses_per_rpc_sum{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev"} 6003
otelcol_rpc_server_responses_per_rpc_count{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev"} 6003
# HELP target_info Target metadata
# TYPE target_info gauge
target_info{service_instance_id="35b0c0a7-266d-40e7-9d6b-743dadd946af",service_name="otelcontribcol",service_version="0.93.0-dev"} 1

This time the receiver received
599400 out of the 600 000

cat receiver_output.log | grep -E 'line[0-9]+' -o -w | wc -l
599400

I am also attaching my receiver config for reference/easier reproduction:

receivers:
  otlp:
    protocols:
      grpc:

exporters:
  debug:
    verbosity: detailed
  file:
    path: ./receiver_output.log
service:
  telemetry:
    metrics:
      level: detailed
      address: 0.0.0.0:9999
  pipelines:
    logs: 
      receivers: [otlp]
      exporters: [file]

@swiatekm
Copy link
Contributor

swiatekm commented Feb 7, 2024 •
edited
Loading

My suspicion is that this is caused by shutdown ordering in the filelog receiver rather than the persistent queue. The queue shutdown behaviour is actually very straightforward, and fully synchronous - we only delete stuff from the storage after having successfully exported it. Data which actually makes it into the queue should be safe against a SIGKILL, much less a SIGINT.

On the other hand, filelog receiver has some asynchronous components taking care of converting data from stanza format to otel format, and I'm much less confident those actually take care of flushing all the data during shutdown. Briefly looking through that code, it doesn't look like we ensure the respective channels are empty before exiting. @djaglowski am I wrong here?

@djaglowski
Copy link
Member

@swiatekm-sumo, that seems plausible.

@swiatekm
Copy link
Contributor

swiatekm commented Feb 7, 2024

We could verify if that's the case by simply disabling the persistent queue. If I'm right, we'll see similar behaviour without it.

@OverOrion
Copy link
Contributor Author

Thanks for the suggestion @swiatekm-sumo!
I disabled the persistent queue and logs are lost as well in this case:
599838 have made it to the other side successfully.

Should I update the issue to reflect these findings?

Here are the receiver's metrics without the sender's persistent queue

bash
# HELP otelcol_exporter_send_failed_log_records Number of log records in failed attempts to send to destination.
# TYPE otelcol_exporter_send_failed_log_records counter
otelcol_exporter_send_failed_log_records{exporter="file",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev"} 0
# HELP otelcol_exporter_sent_log_records Number of log record successfully sent to destination.
# TYPE otelcol_exporter_sent_log_records counter
otelcol_exporter_sent_log_records{exporter="file",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev"} 599838
# HELP otelcol_process_cpu_seconds Total CPU user and system time in seconds
# TYPE otelcol_process_cpu_seconds counter
otelcol_process_cpu_seconds{service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev"} 33.55
# HELP otelcol_process_memory_rss Total physical memory (resident set size)
# TYPE otelcol_process_memory_rss gauge
otelcol_process_memory_rss{service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev"} 2.06901248e+08
# HELP otelcol_process_runtime_heap_alloc_bytes Bytes of allocated heap objects (see 'go doc runtime.MemStats.HeapAlloc')
# TYPE otelcol_process_runtime_heap_alloc_bytes gauge
otelcol_process_runtime_heap_alloc_bytes{service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev"} 8.2304856e+07
# HELP otelcol_process_runtime_total_alloc_bytes Cumulative bytes allocated for heap objects (see 'go doc runtime.MemStats.TotalAlloc')
# TYPE otelcol_process_runtime_total_alloc_bytes counter
otelcol_process_runtime_total_alloc_bytes{service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev"} 4.982638576e+09
# HELP otelcol_process_runtime_total_sys_memory_bytes Total bytes of memory obtained from the OS (see 'go doc runtime.MemStats.Sys')
# TYPE otelcol_process_runtime_total_sys_memory_bytes gauge
otelcol_process_runtime_total_sys_memory_bytes{service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev"} 1.47677464e+08
# HELP otelcol_process_uptime Uptime of the process
# TYPE otelcol_process_uptime counter
otelcol_process_uptime{service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev"} 84.542572386
# HELP otelcol_receiver_accepted_log_records Number of log records successfully pushed into the pipeline.
# TYPE otelcol_receiver_accepted_log_records counter
otelcol_receiver_accepted_log_records{receiver="otlp",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",transport="grpc"} 599838
# HELP otelcol_receiver_refused_log_records Number of log records that could not be pushed into the pipeline.
# TYPE otelcol_receiver_refused_log_records counter
otelcol_receiver_refused_log_records{receiver="otlp",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",transport="grpc"} 0
# HELP otelcol_rpc_server_duration Measures the duration of inbound RPC.
# TYPE otelcol_rpc_server_duration histogram
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="0"} 0
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="5"} 3160
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="10"} 5925
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="25"} 6021
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="50"} 6021
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="75"} 6021
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="100"} 6021
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="250"} 6021
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="500"} 6021
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="750"} 6021
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="1000"} 6021
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="2500"} 6021
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="5000"} 6021
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="7500"} 6021
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="10000"} 6021
otelcol_rpc_server_duration_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="+Inf"} 6021
otelcol_rpc_server_duration_sum{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev"} 31278.951832999963
otelcol_rpc_server_duration_count{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev"} 6021
# HELP otelcol_rpc_server_request_size Measures size of RPC request messages (uncompressed).
# TYPE otelcol_rpc_server_request_size histogram
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="0"} 0
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="5"} 0
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="10"} 0
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="25"} 0
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="50"} 0
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="75"} 0
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="100"} 0
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="250"} 2
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="500"} 3
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="750"} 5
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="1000"} 10
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="2500"} 19
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="5000"} 39
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="7500"} 6021
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="10000"} 6021
otelcol_rpc_server_request_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="+Inf"} 6021
otelcol_rpc_server_request_size_sum{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev"} 3.5339626e+07
otelcol_rpc_server_request_size_count{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev"} 6021
# HELP otelcol_rpc_server_requests_per_rpc Measures the number of messages received per RPC. Should be 1 for all non-streaming RPCs.
# TYPE otelcol_rpc_server_requests_per_rpc histogram
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="0"} 0
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="5"} 6021
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="10"} 6021
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="25"} 6021
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="50"} 6021
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="75"} 6021
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="100"} 6021
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="250"} 6021
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="500"} 6021
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="750"} 6021
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="1000"} 6021
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="2500"} 6021
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="5000"} 6021
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="7500"} 6021
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="10000"} 6021
otelcol_rpc_server_requests_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="+Inf"} 6021
otelcol_rpc_server_requests_per_rpc_sum{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev"} 6021
otelcol_rpc_server_requests_per_rpc_count{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev"} 6021
# HELP otelcol_rpc_server_response_size Measures size of RPC response messages (uncompressed).
# TYPE otelcol_rpc_server_response_size histogram
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="0"} 0
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="5"} 6021
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="10"} 6021
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="25"} 6021
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="50"} 6021
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="75"} 6021
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="100"} 6021
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="250"} 6021
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="500"} 6021
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="750"} 6021
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="1000"} 6021
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="2500"} 6021
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="5000"} 6021
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="7500"} 6021
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="10000"} 6021
otelcol_rpc_server_response_size_bucket{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="+Inf"} 6021
otelcol_rpc_server_response_size_sum{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev"} 12042
otelcol_rpc_server_response_size_count{rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev"} 6021
# HELP otelcol_rpc_server_responses_per_rpc Measures the number of messages received per RPC. Should be 1 for all non-streaming RPCs.
# TYPE otelcol_rpc_server_responses_per_rpc histogram
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="0"} 0
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="5"} 6021
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="10"} 6021
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="25"} 6021
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="50"} 6021
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="75"} 6021
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="100"} 6021
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="250"} 6021
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="500"} 6021
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="750"} 6021
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="1000"} 6021
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="2500"} 6021
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="5000"} 6021
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="7500"} 6021
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="10000"} 6021
otelcol_rpc_server_responses_per_rpc_bucket{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev",le="+Inf"} 6021
otelcol_rpc_server_responses_per_rpc_sum{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev"} 6021
otelcol_rpc_server_responses_per_rpc_count{rpc_grpc_status_code="0",rpc_method="Export",rpc_service="opentelemetry.proto.collector.logs.v1.LogsService",rpc_system="grpc",service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev"} 6021
# HELP target_info Target metadata
# TYPE target_info gauge
target_info{service_instance_id="65955ec7-dd74-40c5-885b-6c84a39d9602",service_name="otelcontribcol",service_version="0.93.0-dev"} 1

@swiatekm
Copy link
Contributor

swiatekm commented Feb 7, 2024

Should I update the issue to reflect these findings?

It would be great if you could.

@OverOrion OverOrion changed the title OpenTelemetry Collector - Persistent Queue loses logs OpenTelemetry Collector - filelog receiver loses logs even if persistence is set up Feb 7, 2024
@swiatekm
Copy link
Contributor

swiatekm commented Feb 8, 2024

I've added a unit test which fails on main and some changes that make it pass, and also cause the original reproduction to fail. See: swiatekm@9747d92. This is very far from an actual fix, and probably misses some edge cases, but it does prove the problem lies in the stanza receiver adapter.

@BinaryFissionGames BinaryFissionGames mentioned this issue Feb 9, 2024
Open
@djaglowski djaglowski mentioned this issue Feb 12, 2024
Closed
@djaglowski djaglowski added receiver/filelog release:required-for-ga Must be resolved before GA release labels Feb 12, 2024
@github-actions GitHub Actions
Copy link
Contributor

Pinging code owners for receiver/filelog: @djaglowski. See Adding Labels via Comments if you do not have permissions to add labels yourself.

@github-actions github-actions bot mentioned this issue Feb 13, 2024
Closed
@OverOrion
Copy link
Contributor Author

Hey @swiatekm-sumo,

Just wanted to have a bit better understanding regarding the scope / fix of this:

You mentioned that

This is very far from an actual fix, and probably misses some edge cases, but it does prove the problem lies in the stanza receiver adapter.

  • What else is required for it to be considered an actual fix?
  • Could you give me an estimate regarding how long would it take for someone
    • who knows the codebase
    • who doesn't?
  • Is this issue being actively worked at? I tried looking for some board that show the currently being worked at issues, but it seems that https://github.com/orgs/open-telemetry/projects/3 is not up to date. Is there another board or something?

@swiatekm
Copy link
Contributor

You mentioned that

This is very far from an actual fix, and probably misses some edge cases, but it does prove the problem lies in the stanza receiver adapter.

* What else is required for it to be considered an actual fix?

The general idea of the change is that the asynchronous components should signal to their downstream that they're shutting down by closing the data channel, and accordingly start shutting down when their upstream does this.

There's two problems with this idea. One is that we're much more likely to create a deadlock this way if we aren't careful. So part of the proper fix would be writing some additional tests to try and detect this, and also doing some integration testing.

The second problem is that we probably don't want to wait forever. If the rest of the pipeline is synchronous, we may simply block on exporting forever. We should have some kind of timeout to deal with this.

@djaglowski please check me on this.

* Could you give me an estimate regarding how long would it take for someone
  
  * who knows the codebase
  * who doesn't?

Actually fixing the problem shouldn't take more than a day or two for someone familiar with the codebase imo. Testing it exhaustively, on the other hand, could be a while depending on how thorough we want to be.

The category of people who don't know the codebase is so broad that I can't give any reasonable estimate. I don't think it would be a big problem for someone reasonably comfortable with concurrency in Go, though. These components don't really do anything very domain-specific.

* Is this issue being actively worked at? I tried looking for some board that show the _currently being worked at_ issues, but it seems that https://github.com/orgs/open-telemetry/projects/3 is not up to date. Is there another board or something?

This issue isn't assigned to anyone, so it's not being worked on. Would you like to try your hand at it?

@OverOrion
Copy link
Contributor Author

Thanks for the detailed explanations @swiatekm-sumo, I really appreciate it!

This issue isn't assigned to anyone, so it's not being worked on. Would you like to try your hand at it?

Yes, please assign it to me. 🚀

@ChrsMark ChrsMark mentioned this issue Mar 27, 2024
Closed
@ChrsMark
Copy link
Member

ChrsMark commented Apr 2, 2024

Could you give me an estimate regarding how long would it take for someone

  • who knows the codebase
  • who doesn't?

I had a look into swiatekm@9747d92 from @swiatekm-sumo and took me some time to get the full sequence of how the components (emitter, converter, adopter etc) and their async routines are linked to each other.
So I think along with the fix we should also try to add some docstrings (like this) to explain how the ShutDown of the receiver works, which is where the current issue starts from.

I have written down a doc like this to cover swiatekm@9747d92 for my own understanding. Sharing it here and feel free to re-use it as is or part of it:

Shutdown should stop the emitterLoop, this happens through the r.pipe.Stop() since it closes the r.emitter.logChan.
Then the emitterLoop should flush down to the Converter all the items of the logChan and notify the Converter to stop.
This is triggered by the r.converter.Stop().
Then the Converter must trigger the stop of the workerLoop, which happens with a close(c.workerChan). Then wait for all
the workers to stop with c.wg.Wait().
All the workers before stopping themselves need to flush the data down for the flushLoop.
Once the workers are done, the Converter inside its Stop will notify the flushLoop to stop with the close(c.flushChan).
Then flushLoop unblocks and flushes the data to pLogsChan and calls the consumer to stop as well with the close(c.pLogsChan).
That's when the consumerLoop is triggered to flush all the data and return.
Eventually the Shutdown will resume from the r.wg.Wait() since both the emitterLoop and the consumerLoop have flushed and return.

@steelecliftonberry
Copy link

Hi - we are about to implement this file_storage extension for filelog receiver, as well as persistent queue. Would be very thankful if anyone can answer:

  • does the use of the file_storage extension for filelog receiver, even with this leak described above, still lead to fewer dropped logs than without the extension?
    Or does it basically not help at all yet till this bug is resolved?

  • to confirm: the logs are only dropped during termination, not normal operation?

  • the persistent queue extension with file_storage is working as expected and not dropping logs?

@ChrsMark
Copy link
Member

ChrsMark commented Apr 30, 2024 •
edited
Loading

Hey @steelecliftonberry. I'm sharing my own view on this but would love to hear others' opinions as well if I miss anything here :):

does the use of the file_storage extension for filelog receiver, even with this leak described above, still lead to fewer dropped logs than without the extension?
Or does it basically not help at all yet till this bug is resolved?

Without using the persistent queue file_storage extension, you lose track of the file read checkpoint (==offset) and hence upon restarts you might will lose any log lines that have been written during the restart (downtime) period. So if you are aiming for persistence during restarts the persistent queue should be your best option here.

to confirm: the logs are only dropped during termination, not normal operation?

Yeap, that bug is identified during shutdown/restart, and is basically because filelog receiver does not manage to ensure that all of its sub-routines have flushed the data they are processing before stopping them.

the persistent queue extension with file_storage is working as expected and not dropping logs?

Based on what we have seen so far, yes. I'm not personally aware of any reported issue.

For deeper understanding about the filelog receiver VS persistence queue, please also refer to the nice explanation at #31074 (comment).

@steelecliftonberry
Copy link

Thanks @ChrsMark - much appreciated.

When you wrote "persistent queue" below can I assume you meant file_storage extension for filelog? I understood persistent queue was for persisting already collected logs which had been not yet been processed, and not related to the offset tracking, both of which increase fault tolerance during restarts. But not fully sure - I notice the example config mixes them together, but I assume they can be implemented independently.

Without using the a persistent queue you lose track of the file read checkpoint (==offset) and hence upon restarts you might will lose any log lines that have been written during the restart (downtime) period. So if you are aiming for persistence during restarts the persistent queue should be your best option here.

@ChrsMark
Copy link
Member

ChrsMark commented Jun 5, 2024

The second problem is that we probably don't want to wait forever. If the rest of the pipeline is synchronous, we may simply block on exporting forever. We should have some kind of timeout to deal with this.

@swiatekm-sumo I have put some thought on your comment and specifically the above part. I'm sharing my findings to keep the ball rolling:

I think that logs which have been already read by the fileConsumer/Manager need to be "flushed" anyways no matter how much time we need to wait for it. Otherwise we still have the risk to drop logs on restarts since once the fileConsumer's Reader has read the logs the offsets are updated and stored.

I did some digging into the stopping sequence and it seems that the fileConsumer is the first to be stopped before the downstream is trigered to stop.
Once the fileConsumer is stopped there is no chance that extra logs will be read and hence we can trigger the proper sequential stopping for its downstream components (emitter and converter). Here is a more detailed walkthrough:

  1. The stanza receiver is called to stop: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.102.0/pkg/stanza/adapter/receiver.go#L136
  2. The receiver's pipeline is called to stop at https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.102.0/pkg/stanza/adapter/receiver.go#L142
    The pipeline has the input operator (which includes the fileconsumer) in the first place and the emitter at the last one.
  3. The operators of the pipeline are stopped syncronously at https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.102.0/pkg/stanza/pipeline/directed.go#L70-L82. Hence we know that the fileconsumer will be stopped and will have "flushed" all of its data before the Emitter is called to stop(last operator in the pipeline). At this point since the fileconsumer has stopped "consuming" logs the emitter can just flush and trigger its downstream without hanging.

So I think that hanging on a situation where we consume forever won't be possible here. However, that's worth covering it with unit tests.

/cc @djaglowski @OverOrion

@swiatekm
Copy link
Contributor

swiatekm commented Jun 5, 2024

@ChrsMark what I meant by downstream was the rest of the otel pipeline filelog receiver is writing to. We have no control over how quickly that succeeds. And while we can wait forever in principle, eventually the otel process will be killed by an impatient user or process manager.

A proper solution would be to update offsets in a callback after the records were successfully consumed by the downstream pipeline. This is more difficult than it sounds though, as there's no guarantee that the log lines are emitted in the same order they were read in, or that they even should be emitted at all. So we either accept some possible duplication on restart (we don't move the offset until we're certain the log was moved), or we introduce an additional on-disk database where we temporarily save the log lines, and track them individually based on id. This would definitely have a performance impact though.

@michalpristas
Copy link
Contributor

Hey @OverOrion are you working on this one?

@VihasMakwana VihasMakwana mentioned this issue Jul 29, 2024
Closed
@djaglowski
Copy link
Member

Since this was brought up in the SIG today, I'll just capture my comments here and expand on them a bit.

I'm not sure this is solvable by the filelog reciever but I'd like to hear a more detailed proposal if anyone thinks it can be.

The general idea of using callbacks and/or IDs to delay updating the offset is interesting but as @swiatekm pointed out there are problems with that as well. To articulate further:

Problem 1

Suppose log two is emitted by an exporter before log one. If the offset is updated to 16, then log one may yet be lost.

Problem 2

Suppose the receiver is used in two pipelines. What happens if one copy is emitted while the other is not?

Problem 3

Suppose the logs are passed only to a count connector. They are effectively dropped, but we now have a metric that describes them. When/how do we update the offset?

Problem 4

If all of the above problems can be solved. Can we articulate how any combination of the above is also solved?

@ChrsMark
Copy link
Member

ChrsMark commented Aug 1, 2024 •
edited
Loading

Thank's for taking this on @michalpristas!
A couple of thoughts from my side in case it can be useful (if I miss any critical information please point out).

I'm not sure this is solvable by the filelog reciever but I'd like to hear a more detailed proposal if anyone thinks it can be.

One thing that's still unclear to me is to what extend the filelogreceiver (and any other receiver) should rely on other components' progress/success/failure. Each component should do its best to ensure that the data it forwards are actually passed down to the pipeline and then update its "records", in this case the offsets in the persistent file_storage.
If then an exporter fails to export and ends up losing data upon a Collector's restart that's not a fault of the filelogreceiver.
However, that's not the case today with filelogreceiver since on its shutdown, it updates the offsets without really having forwarded the data to the rest of the pipeline by properly flushing its internal "queues" (those of the emitter, converter etc) and calling consumeLogs. That's what the patch from #31074 (comment) surfaced. It also blocks until consumeLogs has returned.

Back to how this should be handled as part of the whole Collector's pipeline, I see this as a generic issue which can affect any other receiver which needs to properly keep track of the received data. I would summarize it as "any Collector's pipeline that defines any receiver that needs to keep track of the received data should ensure that data make it sucessfully to the export stage".
To this, all arguments from @djaglowski 's comment apply here.

Combining the above assumptions, one thing that might help here could be to think of the Collector's pipeline as a set of individual components where each of these is responsible for persisting the data upon Collector's restarts.
To illustrate the point let's think of an example where the filelogreceiver forwards the data directly to an exporter.
If the filelogreceiver is properly shuted down, by flushing its data before exiting, then it's exporter's responsibility to properly handle the shutdown on its side by properly flushing its "queue" and update its progress' records.
(assuming no race-conditions occur and the exporter has actually acknowldged the incoming data). Does this mean that it leverages the storage extension to keep track of what has been exported from the incoming data along with what are the pending data? I guess that describes the exporterhelper persistent-queue.

In short, would the following make sense?

  1. each component of the pipeline should be responsible for persisting "sensitive" data upon failures, restarts etc.
  2. components should acknowledge that they have received the data from the upstream and take over the responsibility to move them forward without blocking the upstream

If we go to the direction of callbacks, things get complicated because some branches of the pipeline might succeed while others might fail to progress and in that case the offset updates can be valid for some branches of the pipeline but invalid for the rest.

From my perspective it would make sense to ensure that filelogreceiver's flush properly takes place on shutdown (with something like #31074 (comment)) to improve the situation a bit for now and keep thinking how the generic issue can best be approached. Potentially blocking for quite some time at consumeLogs is something to consider but I think it can also be a fair compromise for the short term.

@djaglowski
Copy link
Member

@ChrsMark, thanks for highlighting #31074 (comment) again. I agree this is a problem that can and should be fixed. I don't think it solves anything for abrupt termination of the collector, but if properly shut down we should be consistent.

  1. each component of the pipeline should be responsible for persisting "sensitive" data upon failures, restarts etc.

I look at this as a matter of stateful vs stateless components. Stateful components should persist their state during shutdown if a storage extension is available. There may be a case for formalizing a declaration of statefulness so that we can more easily enforce standards like this. e.g. Add a flag to consumer.Capabilities or metadata.yaml files.

  1. components should acknowledge that they have received the data from the upstream and take over the responsibility to move them forward without blocking the upstream

The benefit of blocking is that in many cases an error can be propagated back upstream which can be reacted to. For example in a simple otlp->otlp forwarding pipeline, the result can ultimately be returned from the otlp receiver to the client trying to send the data. Then the client has the ability to react as they see appropriate. However, this only works for synchronous components, and to my knowledge we could do a better job declaring components as sync vs async (again maybe in consumer.Capabilities or metadata.yaml). This would make it easier to validate expectations.

@ChrsMark
Copy link
Member

ChrsMark commented Aug 7, 2024

Thank's for the context @djaglowski! From my side the distinction of the 2 "issues" sounds reasonable and covers my previous concerns. I'm +1 for kicking off the discussion on "stateful vs stateless components" and "sync vs async components".

If we agree on this maybe we can already fix the "properly shut down case" (#31074 (comment)) (/cc @michalpristas if you are still on it:)) and branch out the discussion about sync/async and steful/stateless components in a separate issue.

@michalpristas michalpristas mentioned this issue Aug 13, 2024
Merged
@michalpristas
Copy link
Contributor

michalpristas commented Aug 13, 2024 •
edited
Loading

I took a stab at it resulting in something similar to what @swiatekm did (draft here)
I fixed some synchronization on top of that as it was still losing data and there was a risk of hitting panic
I'd appreciate some feedback there, beware that I'm missing a bit of context when it comes to broader view of this part of the code so some assumptions may be false

cc @djaglowski

@michalpristas michalpristas mentioned this issue Aug 15, 2024
Open
@michalpristas
Copy link
Contributor

michalpristas commented Aug 15, 2024 •
edited
Loading

as we agreed to address shutdown sync in this issue I created these as followups
#34702
#34703

djaglowski added a commit that referenced this issue Sep 10, 2024
@michalpristas @djaglowski @swiatekm
Synchronize shutdown in stanza adapter (#34638)
ede416f 
**Description:** 
This PR takes emitter-converter-receiver pipeline and synchronize
shutdown in a sense that lower level of the pipeline needs to be fully
finished before next one is shut down.

**Link to tracking Issue:** #31074

**Testing:** UT
run 
```
 go test -run ^TestShutdownFlush$ . -count 10000 -failfast
```

**Documentation:** in code comments

---------

Co-authored-by: Daniel Jaglowski <jaglows3@gmail.com>
Co-authored-by: Mikołaj Świątek <mail@mikolajswiatek.com>
@andrzej-stencel
Copy link
Member

Since @michalpristas is on extended leave, I'll reassign this to myself, unless anyone opposes.

I believe Michal's fix #34638 greatly improves the situation. In my understanding, it fixes the issue in the "happy path" scenarios, that is when the collector is being shut down gracefully and the telemetry can be passed on from the Filelog receiver down the collector's pipeline.

@andrzej-stencel
Copy link
Member

I've updated the labels to signify that this is an issue in the Filelog receiver and not in the File Storage extension or File exporter.

@andrzej-stencel andrzej-stencel changed the title OpenTelemetry Collector - filelog receiver loses logs even if persistence is set up Filelog receiver loses logs even if persistence is set up Sep 11, 2024
@andrzej-stencel
Copy link
Member

@djaglowski, @swiatekm and I had a discussion on the next steps for this issue. Let me describe what I got out of it and please guys add your thoughts as needed.

As mentioned in above comment, I believe currently the Filelog receiver does not lose logs when the collector is shut down gracefully and the telemetry can be pushed out of the receiver further down the pipeline.

On the other hand, when the collector is shut down ungracefully (e.g. with SIGKILL), or when during graceful shutdown the Filelog receiver is not able to send the telemetry out (e.g. because the exporter configured after the Filelog receiver cannot reach a remote endpoint to export the telemetry, and does not have its own persistent queue), then data loss can still occur. We'd like to remedy this.

The first place where data loss can occur is in the LogEmitter (batch field). The LogEmitter acts as an output operator in the Stanza pipeline. In the Stanza pipeline, every log record (an "entry" in Stanza nomenclature) is sent separately. The batching in the LogEmitter has been added (to my knowledge) to improve performance by creating Log Resources that contain more than one Log Record.

After the LogEmitter's buffer reaches 100 logs (Stanza entries), it puts those 100 Stanza entries on a channel (logChan) for the Stanza adapter to retrieve it, convert into an OTLP LogResource with 100 LogRecords, and send it down the collector pipeline to another component (processor/connector/exporter). This conversion is currently pretty complex, involving multiple asynchronous workers and a bunch of channels:

If my understanding of Go channels is correct, if a batch of logs is waiting on any of these channels during an ungraceful shutdown, it will also be lost. This is because the Stanza File consumer marks a log as successfully sent (and updates the file offsets in persistent storage) as soon as the Stanza entry is picked up from the LogEmitter's logChan channel.

The recent fix eliminated some logical/implementation bugs that could cause logs to be lost during a graceful shutdown, but there still exists the possibility of losing logs during ungraceful shutdown. To eliminate this possibility, I can think of the following solutions:

  1. Do not update the offsets until the log is eventually sent out of the Stanza adapter into another collector component via a successful call to ConsumeLogs on next consumer. This could be done by providing a callback that will be called after ConsumeLogs.
  2. Make the Stanza adapter use persistent storage for the logs in transit, so that after a shutdown the logs in transit are restored from storage.
  3. Simplify the logic to convert the Stanza entry to a OTLP log record in the LogEmitter and call ConsumeLogs synchronously, removing all the asynchronicity and possibly also buffering.

In my view, the current implementation with multiple channels and asynchronous workers if overly complex without delivering a lot of value, instead creating the possibility of data loss. I would like to see the following improvements:

  1. Remove the channels and workers by having LogEmitter convert every Stanza entry to a log record and call ConsumeLogs synchronously. Leave the 100-log buffering in place for this change, to minimize the impact. Compare benchmarks.
  2. Remove batching in LogEmitter and compare the performance. This can introduce a significant performance impact, so needs to be considered carefully.
  3. (simultaneously to 2) Consider if it's feasible to modify the File consumer to send logs in batches. It seems natural (to me) for the File consumer to emit all logs read from one file in one pass as a single batch. The file consumer being able to emit batches of entries as opposed to single entries could alleviate the performance impact of removing batching from LogEmitter (possibly even improving the situation, as the "natural" batches could be larger than the current artificial 100 log limit). This would require the modification of all Stanza operators to be able to operate on batches of Stanza entries as opposed to single entry at a time. It seems a simple change to add a ProcessBatch method beside the existing Process method, with a for loop calling Process inside the ProcessBatch function.

@djaglowski
Copy link
Member

Great summary @andrzej-stencel. I'll add just a little detail as I understand.

  1. Remove the channels and workers by having LogEmitter convert every Stanza entry to a log record and call ConsumeLogs synchronously. Leave the 100-log buffering in place for this change, to minimize the impact. Compare benchmarks.

Assuming we make the changes described in (3), then the 100-log buffer will only need to exist for the sake of non-filelog receivers (journald, windows event log, syslog, udp, tcp, named pipe). Potentially at a later time, we can pull batching upstream into some of these as well. e.g. emit multiple logs from the same journald unit or WEL channel. The others either do or could support ingestion of multiple logs at once, but removing the emitter's log buffer will effectively be asking the sender of the logs to be more thoughtful about batching. In some cases this will not be possible for them to make changes, meaning they would have to accept every plog.Logs will contain only 1 LogRecord.

  1. Remove batching in LogEmitter and compare the performance. This can introduce a significant performance impact, so needs to be considered carefully.

I'm not sure it will be possible to entirely remove batching if we also implement the changes described in (3), which I think we should do. We have to be aware that logs read from a given file (or other source) are not necessarily related to the same resource and/or scope. So when a slice of stanza entries are converted into a plog.Logs, it'll still be necessary to batch them by resource and scope.

  1. (simultaneously to 2) Consider if it's feasible to modify the File consumer to send logs in batches. It seems natural (to me) for the File consumer to emit all logs read from one file in one pass as a single batch. The file consumer being able to emit batches of entries as opposed to single entries could alleviate the performance impact of removing batching from LogEmitter (possibly even improving the situation, as the "natural" batches could be larger than the current artificial 100 log limit). This would require the modification of all Stanza operators to be able to operate on batches of Stanza entries as opposed to single entry at a time. It seems a simple change to add a ProcessBatch method beside the existing Process method, with a for loop calling Process inside the ProcessBatch function.

In my opinion this could be a really nice improvement regardless of any other changes. The data coming out of the receiver will be more organized and intuitive. In terms of implementation, I think we can add a []string buffer to the Reader struct, and then emit the entire buffer, either when it reaches a max size (maybe configurable at a later time) or whenever we hit EOF. We can maintain a tentative offset which is saved only when this buffer is actually emitted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@andrzej-stencel andrzej-stencel

Labels
bug Something isn't working receiver/filelog release:required-for-ga Must be resolved before GA release
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
8 participants
@michalpristas @djaglowski @mx-psi @steelecliftonberry @ChrsMark @OverOrion @andrzej-stencel @swiatekm