Possible Memory leak issue with kube parser?

[umpa385]

Has anyone had issues with the kube log parser using up a lot of memory? My config is attached because its quite big. Wanted to know if there was some type of memory leak bug? It is using a lot more memory even compared to fluent which just doesn't seem right. I don 't have any errors on the vector config or logs for vector.

customConfig:
  data_dir: "/vector-data-dir"
  sources:
    syslog:
      type: "file"
      include:
      - "/var/log/syslog"
      - "/var/log/auth.log"
      - "/var/log/kern.log"
      - "/var/log/messages"
      read_from: "beginning"
    var_logs:
      type: "file"
      include:
      - "/var/log/*.log"
      exclude:
      - "/var/log/syslog"
      - "/var/log/auth.log"
      - "/var/log/kern.log"
      - "/var/log/messages"
      read_from: "beginning"
    containers:
      type: "kubernetes_logs"
      exclude_paths_glob_patterns:
      - "/var/log/pods/blend-system_fluent*/**"
      - "/var/log/pods/vector_vector*/**"
      - "/var/log/pods/blend_*apitest[0-9]*/**"
      - "/var/log/pods/blend_*e2e[0-9]*/**"
      - "/var/log/pods/blend_lending-webdriver-test*/**"
      - "/var/log/pods/blend_faye*/**"
      - "/var/log/pods/kube-system_datadog-[a-z0-9][a-z0-9][a-z0-9][a-z0-9][a-z0-9]_*/**"
      - "/var/log/pods/blend_load-test-task-pod*/**"
      - "/var/log/pods/osquery*/**"
      - "/var/log/pods/*_actions-runner-system_*/**"
      - "/var/log/pods/*_appsmith_*/**"
      - "/var/log/pods/*_velero_*/**"
      read_from: "beginning"
    osquery_results:
      type: "file"
      include:
      - "/var/log/osquery/osqueryd.results.log"
      read_from: "beginning"
    osquery_snapshots:
      type: "file"
      include:
      - "/var/log/osquery/osqueryd.snapshots.log"
      read_from: "beginning"
  transforms:
    syslog_parser:
      type: "remap"
      inputs:
      - syslog
      source: |
        parse_syslog!(.message)
    containers_ec2_metadata:
      type: "aws_ec2_metadata"
      inputs:
      - containers
      fields:
      - instance-id
      - local-ipv4
    containers_transform:
      type: "remap"
      drop_on_abort: false
      reroute_dropped: true
      inputs:
      - containers_ec2_metadata
      source: |
        .k8s.namespace_name = .kubernetes.pod_namespace
        .k8s.pod_name = .kubernetes.pod_name
        .k8s.pod_ip = .kubernetes.pod_ip
        .k8s.container_name = .kubernetes.container_name
        .k8s.container_image = .kubernetes.container_image
        del(.kubernetes)
        .kubernetes = del(.k8s)
        if is_json(to_string(.message) ?? "") {
          .parsed = parse_json!(.message)
        } else {
          .parsed = .message
        }
        del(.message)
        del(.source_type)
        del(.stream)
        .time = del(.timestamp)
        .env = "sandbox"
        .k8s_cluster_name = "temp.k8s.centrio.com"
        .instance_id = del(."instance-id")
        .ip_address = del(."local-ipv4")
#         del(.file)
    osquery_results_parser:
      type: "remap"
      inputs:
      - osquery_results
      source: |
        if exists(.message) {
          .parsed = parse_json!(.message)
        } else {
          log("JSON parse error in osquery_results", level: "error")
          .parsed = null
        }
    osquery_snapshots_parser:
      type: "remap"
      inputs:
      - osquery_snapshots
      source: |
        if exists(.message) {
          .parsed = parse_json!(.message)
        } else {
          log("JSON parse error in osquery_snapshots", level: "error")
          .parsed = null
        }
    osquery_results_filter:
      type: "filter"
      inputs:
      - osquery_results_parser
      condition: 'exists(.parsed)'
    osquery_snapshots_filter:
      type: "filter"
      inputs:
      - osquery_snapshots_parser
      condition: 'exists(.parsed)'

[jszwedko]

One guess would be internal metric cardinality because the kubernetes_logs source tags some internal metrics with high cardinality tags. You could try setting https://vector.dev/docs/reference/configuration/global-options/#expire_metrics_secs to expire those high cardinality metrics.

[umpa385]

will give that a shot ty!

[umpa385]

So it seems like the memory usage is much better (back to what is expected), but trying to understand what the note below means if you have more context on it. Thanks again for the help really appreciate it!

Note that internal counters that are expired but are later updated will have their values reset to zero. Be careful to set this value high enough to avoid expiring critical but infrequently updated internal counters.

[umpa385]

Also if you have any idea what the "default" for that value should be, it would be helpful. Really haven't used this before and trying to understand what it is used for and what problems it can cause if we set it to high or too low. Thanks again!

[jszwedko]

There are two things to keep in mind:

• If you are using the internal_metrics source, you'll want to configure it to be greater than the scrape_interval_secs.
• If you are using the prometheus_metrics sink, you'll want to configure it to be greater than the scrape interval of any clients scraping that endpoint

[umpa385]

We don't use either so I think we should be good.

[umpa385]

Sorry need to still get an answer for this.

Also if you have any idea what the "default" for that value should be, it would be helpful. Really haven't used this before and trying to understand what it is used for and what problems it can cause if we set it to high or too low. Thanks again!

Also would like to know what the internal counters are, if there is documentation on this that would be ideal.

[pront]

Are you referring to https://vector.dev/docs/reference/configuration/global-options/#expire_metrics_secs? That is None by default will translates to "metrics are never removed". Choosing a value here is a tradeoff between memory usage and stored metrics history. You can start with something reasonable like 100 seconds and take it from there.

If you are really curious, the implementation details are in lib/vector-core/src/metrics/recency.rs. For example, if a counter expires it is deleted from the registry, but that counter might be increased later again, so a new entry will be created from scratch for that counter.

[umpa385]

We are not using either source in the config. We don't use internal_metrics or prometheus_metrics, though making the change suggested did drop memory usage.

Are internal_metrics turned on by default regardless of config?

[jszwedko]

Are internal_metrics turned on by default regardless of config?

Correct, telemetry is always enabled. I swear we have an issue for making it possible to disable, but I'm not finding it now.

[chriskim06]

maybe this one #13692

[jszwedko]

maybe this one #13692

Ah, yes, there it is!