Is it possible to filter null values ? (and also how does the expect "result_key" action work ?)

[googleg]

Hi all,

I have an issue with the ingestion of network logs into ES using FluentBit and the Geoip2 filter to get IP geoloc.

The problem is, from time to time the IPs cannot be geo-localised, e.g. private network IPs, multicast IPs, or simply not found in the Geoip2 database.

When this happens, the Geoip2 filter returns "null" values that get then added to my records and this creates problem down the line for decoding and ingesting the data.

My question is: is there a way to detect null values and do something, even as silly as replacing it with an arbitrary value ?

I tried to play with the Expect filter, it works for detecting null values but I can't get anything useful done, only "warn" or "exit".

The documentation mentions a third action "result_key" (https://docs.fluentbit.io/manual/pipeline/filters/expect) but I don't understand if that could help me and if so how does that work ?

Thanks for your help.

[googleg]

Ok so I am responding to myself: the usage of the action result_key is explained and illustrated here: #5668

Now, I am using the td-agent-bit package for arm64 (Raspbian 11), and despite its version being advertised as 1.9.3 the list of possible actions does not include "result_key" (see below).

I am a bit confused because according to the documentation that action was introduced in fluent-bit 1.9 ??!?

`
Fluent Bit v1.9.3

• Copyright (C) 2015-2022 The Fluent Bit Authors
• Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
• https://fluentbit.io

HELP
expect filter plugin

DESCRIPTION
Validate expected keys and values

OPTIONS
key_exists check that the given key name exists in the record
> default: default, type: string

key_not_exists check that the given key name do not exists in the record
> default: default, type: string

key_val_is_null check that the value of the key is NULL
> default: default, type: string

key_val_is_not_null check that the value of the key is NOT NULL
> default: default, type: string

key_val_eq check that the value of the key equals the given value
> default: default, type: space delimited strings (minimum 1)

action action to take when a rule does not match: 'warn' or 'exit'
> default: warn, type: string
`

[googleg]

Ok I managed to solve this one: I finally realized that the package is now called fluent-bit instead of td-agent-bit. I have version 2.0.6 installed now and the expect filter action result_key works as expected.

I am still not able to meet my goal but I'll open an other discussion thread for that.

[googleg]

Regarding my original question/problem, I managed to work around the problem by converting the null values to text via a Lua filter and then using a Modify filter to change the "null" strings to arbitrary values.