Feature Request: Autodiscover hints opt-in

[Evesy]

Hi,

We have recently started to use Filebeat's autodiscovery mechanism to scrape logs from Kubernetes containers, however we don't want all files to be processed by default. Rather we want to enable an opt-in process for pod logs to be processed by beats and shipped to Elasticsearch.

Currently we are able to achieve this with the below config:

    filebeat.autodiscover:
      providers:
        - type: kubernetes
          in_cluster: true
          tags:
            - "kubernetes"
          templates:
            - condition:
                contains:
                  kubernetes.annotations.logging: "true"
              config:
                - type: docker
                  tail_files: true
                  containers.ids:
                  - "${data.kubernetes.container.id}"
                  processors:
                  - add_kubernetes_metadata:
                      in_cluster: true
                  - add_cloud_metadata: ~

As expected, only logs from pods with the annotation kubernetes.annotations.logging == true are processed.

However in addition to the above config, we also want to make use of the hints based autodiscover to handle cases where logs are multiline, or perhaps certain lines need to be excluded.

If I simply add hints.enabled: true to the above provider config, all Kubernetes logs begin to get processed (https://discuss.elastic.co/t/filebeat-autodiscover-hints-breaking-template/137310/8?u=evesy), and in addition those events aren't enriched with the cloud metadata.

I've been informed of co.elastic.logs/disable: true as an option for the hints autodiscovery, however this would involve every pod that doesn't want logging to set this annotation (not everything deployed is in our control), making it an opt-out process.
What I'd love is the ability to perform an inverse of this logic to make it strictly an opt-in process (i.e. co.elastic.logs/enable: true)

I'm unable to currently find a way to do this.

In summary it'd be great if it was possible to use the hints autodiscovery to process Kubernetes logs but only for pods with a specific annotation on them

[jeffspahr]

I'd love to see this as well.

[Croolis]

Why do we want to use special flag to explicitly enable logging for some containers? Why do we agree on fact that hints are supposed to enable logging for all containers even though it's disabled in configuration of autodiscover? It looks like implementing workaround for bug instead of fixing it. Why hints break autodiscover configuration?

[exekias]

I agree with having a config flag that allows this behavior

[Stono]

I'd really like to see this behaviour, we have 400 pods, and only about 30 have an annotation to actually care about the logs... so we're reading 370 other logs and overloading filebeat without reason.

[ptemmer]

Bumped into this today as well. Enabling "hints" overrules the autoDiscover conditions, which is not desired.

All kube-system pods get automatically included as well. It's not workable having to edit each of those deployments to add the annotation to not log them

[jsoriano]

I think this issue is addressed by #10911, closing this. Please reopen if you think something else needs to be done.

[dudicoco]

@jsoriano #10911 does not solve the issue for all use cases:
We have two filebeat daemonsets in different versions, each shipping logs to different logstash hosts.
Using the co.elastic.logs/disable: false annotation is not specific enough in this case.

[jsoriano]

Hi @dudicoco,

@jsoriano #10911 does not solve the issue for all use cases:
We have two filebeat daemonsets in different versions, each shipping logs to different logstash hosts.
Using the co.elastic.logs/disable: false annotation is not specific enough in this case.

Why this annotation is not enough in this case? Could you give more details on what is the problem on this scenario?

If the problem is to have different selectors for each filebeat daemonset, you can use the hints.key setting, that allows to change the expected key in the annotations.

For example with this configuration:

filebeat.autodiscover:
  providers:
    - type: kubernetes
      hints.enabled: true
      hints.key: otherlogs

You can use annotations like these ones:

co.elastic.otherlogs/disable: false

So one filebeat would be attending to co.elastic.logs... annotations, and the other to co.elastic.otherlogs... annotations.

If you try this, please report results back, I think we are lacking documentation and testing on this feature 🙁

[dudicoco]

Hi @jsoriano, thanks for the prompt reply!

This feature is indeed undocumented.

Is there a way to do something similar:

1. I would like all containers to send logs by default via filebeat1, without any annotation specifying this.
2. I would like only specific containers to send logs via filebeat2, using the annotation co.elastic.otherlogs/enable: true.
3. Pods without the co.elastic.otherlogs/enable: true annotation should not send logs via filebeat2.

Thanks again :)

[jsoriano]

@dudicoco then maybe you don't even need to use hints.key.

You can configure filebeat1 with the following configuration so it collects logs from all containers:

filebeat.autodiscover:
  providers:
    - type: kubernetes
      hints.enabled: true

And filebeat2 with collection disabled by default:

filebeat.autodiscover:
  providers:
    - type: kubernetes
      hints.enabled: true
      hints.default_config.enabled: false

Then only logs of pods with co.elastic.logs/enable: true will be collected by filebeat2.

By the way, for questions on usage of Beats, or to discuss about possible bugs of missing features, please use https://discuss.elastic.co/ 🙂

[dudicoco]

@jsoriano I've already used a similar solution after digging in a bit more and it works! :)

It would be beneficial to add hints.key to the docs.

Thank you!

[jsoriano]

Nice to read it works for you, thanks! I have created an issue as follow up to improve the support of the hints.key option: #19540