FILTER LUA create new key values

[tecwisesistema]

Hi

On a Windows server running fluent-bit 2.1.2 with a log file in UTF-16le format, log forwarding does not fail, but because of extra bits, it embeds a NULL character between each character in the log label.

---

{"log": "\u0000l\u0000i\u0000b\u0000r\u0000a\u0000r\u0000y\u0000!\u0000R\u0000e\u0000p\u0000o\u0000r\u0000t\u0000S\u0000e\u0000r\u0000v\u0000e\u0000r\u0000_\u00000\u0000-\u00008\u0000!\u00001\u0000a\u00008\u0000c\u0000!\u00000\u00005\u0000/\u00002\u00004\u0000/\u00002\u00000\u00002\u00003\u0000-\u00001\u00004\u0000:\u00000\u00007\u0000:\u00004\u00002\u0000:\u0000:\u0000 \u0000i\u0000 \u0000I\u0000N\u0000F\u0000O\u0000:\u0000 \u0000R\u0000e\u0000n\u0000d\u0000e\u0000r\u0000F\u0000o\u0000r\u0000N\u0000e\u0000w\u0000S\u0000e\u0000s\u0000s\u0000i\u0000o\u0000n\u0000(\u0000'\u0000/\u0000M\u0000S\u0000B\u0000I\u0000/\u0000I\u0000s\u0000A\u0000l\u0000i\u0000v\u0000e\u0000'\u0000)\u0000\r\u0000"}

We have used an LUA Filter and now it registers in Loki without the NULL characters

-- LUA Script
function rem_utf16le(tag, timestamp, record)
record["log"] = string.gsub(record["log"],"[%z+]", "")
return 2, timestamp, record
end

"log": "library!ReportServer_0-10!20b8!05/25/2023-11:47:54:: i INFO: RenderForNewSession('/MSBI/IsAlive')\r",

My question is with LUA, can we parse the tag record to extract tags using a fixed key/value??

Thanks in advanced
Best regards

[agup006]

Yes, you can certainly embed the tag as another record within the message. in the official docs there is a section that goes over this scenario:

Within this section the lua function adds a tag:
https://docs.fluentbit.io/manual/pipeline/filters/lua#inline-configuration