Filebeat Azure module pipeline setup failing in 7.6.0

[robrankin]

Running Filebeat in Kubernetes as a Job using Elastic docker images.

Version 7.5.x - Pipeline setup succeeds
Version 7.6.0 - Pipeline setup fails

7.5.2 Job definition:

resources:
- apiVersion: batch/v1
  kind: Job
  metadata:
    name: filebeat-pipelines-752-init
  spec:
    ttlSecondsAfterFinished: 100
    template:
      spec:
        nodeSelector:
          kubernetes.io/os: linux
        containers:
        - name: filebeat-pipelines-752-init
          image: docker.elastic.co/beats/filebeat:7.5.2
          command: [ "filebeat", "setup", "--pipelines",
            "--modules",
            "azure",
            "-E", "output.logstash.enabled=false",
            "-E", 'output.elasticsearch.hosts=["<snip>:9200"]' ]
        restartPolicy: Never
    backoffLimit: 4

Output:

Loaded Ingest pipeline

7.6.0 Job defintion

resources:
- apiVersion: batch/v1
  kind: Job
  metadata:
    name: filebeat-pipelines-760-init
  spec:
    ttlSecondsAfterFinished: 100
    template:
      spec:
        nodeSelector:
          kubernetes.io/os: linux
        containers:
        - name: filebeat-pipelines-760-init
          image: docker.elastic.co/beats/filebeat:7.6.0
          command: [ "filebeat", "setup", "--pipelines",
            "--modules",
            "azure",
            "-E", "output.logstash.enabled=false",
            "-E", 'output.elasticsearch.hosts=["<snip>:9200"]' ]
        restartPolicy: Never
    backoffLimit: 4

Output

Exiting: Error getting config for fileset azure/activitylogs: Error interpreting the template of the input: template: text:2:22: executing "text" at <.connection_string>: map has no entry for key "connection_string"

[faec]

@narph Do you have context on this? Was there a change in 7.6 that might be causing this?

[narph]

@faec , there are some changes in 7.6.0 but from testing, it looks like it is trying to validate on all required fields which it did not do before.
I see this behavior when testing with aws as well, maybe with other modules as well:

Exiting: Error getting config for fileset aws/cloudtrail: Error interpreting the template of the input: template: text:4:14: executing "text" at <.queue_url>: map has no entry for key "queue_url"

Not sure if something has been changed overall or we need to set anything special in these modules in order to bypass these exceptions. I am looking into this further, at the moment a dirty workaround would be to assign values to these config options:

....-M "azure.activitylogs.var.connection_string='val'" -M "azure.activitylogs.var.storage_account=val'" -M "azure.activitylogs.var.storage_account_key='val'" .......

[narph]

hi @andrewkroh, we are encountering the issue above because of the recent changes here:
https://github.com/elastic/beats/blob/7.6/filebeat/fileset/fileset.go#L270 where you have added the missingkey option to return an error.
It was previously:
https://github.com/elastic/beats/blob/7.5/filebeat/fileset/fileset.go#L271

Before, the setup --modules azure would work just using invalid values but now every configuration option has to be entered.

What should be the behavior in this case? How should the azure config adhere to the new changes?

[andrewkroh]

The manifest for azure/activitylogs is missing declarations for some of the variables used in the template (connection_string, storage_account, and storage_account_key). All variables used in the template must be defined in the manifest (this was the case before my change). Because the variables are not defined in the fileset's manifest users cannot set the variables in their config files despite the config showing them as example. They only way to set these variables will be via the -M CLI flag.

The change that I made detects this condition and prevents you from getting into this undefined state. Prior to the change the undefined variables would have been automatically set to <no value> (e.g. the module config would literally contain connection_string: <no value>) masking the fact that a variable was missing from the manifest.

It sounds like these variables are optional. If that's the case then add them to the fileset's manifest with a default value.

The error you noted from aws/cloudtrail is intentional because the queue_url is a required variable. We want a hard failure if a user does not configure it. Hard failures like this were hard to implement before my change because a default value was required for all variables in the manifest.

[narph]

@andrewkroh , thanks for the quick reply and explanation. All the 3 configuration options are required (connection_string, storage_account, and storage_account_key) in this case so I assume the behavior is intentional.
The other behavior that I have noticed was that in the azure configuration file we have enabled only the activitylogs fileset by default.
After setting some values for the required configuration options (example in comment #16270 (comment)) it will still return a similar exception, prompting us to enter values for the required configuration options from the other disabled filesets:

Exiting: Error getting config for fileset azure/signinlogs: Error interpreting the template of the input: template: text:2:22: executing "text" at <.connection_string>: map has no entry for key "connection_string"

signinlogs in this case. Should it not ignore the disabled filesets?

[andrewkroh]

All the 3 configuration options are required (connection_string, storage_account, and storage_account_key) in this case so I assume the behavior is intentional.

Even if they are required options they still must be declared in the manifest to allow the config file to work properly.

Should it not ignore the disabled filesets?

AFAIK disabled filesets are ignored. If you are enabling the module via the CLI -modules flag then you may be enabling all filesets (try looking at debug output). I don't even think it uses the config file when you do this. You can try adding -M azure.*.enabled=false and -M azure.activitylogs.enabled=true to selectively enable one fileset (see test_modules.py as an example).

[narph]

@andrewkroh , I have defined all the missing var definitions in the manifest files and it seems that this fixed the issue.
Thanks again for the clarifications.