Implement fluent forward protocol for input/sink

[sandervandegeijn]

Is your feature request related to a problem? Please describe.
The forward protocol is used in fluentd/fluent-bit setup to forward logs between instances. This is very useful for larger architectures. It would be nice to implement this for data prepper as well so data prepper can be used as an intermediate or endpoint before sending the data to Opensearch

Describe the solution you'd like
Implement input/sink

Describe alternatives you've considered (Optional)
http - overhead might be an issue

Additional context
none

[KarstenSchnitter]

It would be nice to have something like this. However, the http_source should be able to provide the use-case, probably with the help of some processors to adjust timestamps. I do not see support for http headers in the http_source though, that might be used by FluentBit to indicate the event "tag". Maybe providing a simple configuration example would be enough? Mapping the "date" to "@timestamp" could be one of the exemplary transformations. Comments on batching and compression would also be beneficial.

[sandervandegeijn]

You could use http, but for resource intensive workloads http can be quite intensive. This is why you see other logforwarders implementing custom protocols. A _bulk endpoint could be an alternative as well. But for interoperability supporting the forward protocol would be nice.

Headers would be useful to provide metadata to be able to route the data, but it won't solve the overhead of using http.

[KarstenSchnitter]

Do you have a performance comparison between the two protocols?

[dlvenable]

@ict-one-nl , Are you suggesting that Data Prepper have a source to support the FluentBit Forward output? I understand that this is more of an internal protocol. Would having a general TCP source such as requested in #2162 provide what you are looking for? Or do you specifically want FluentBit Forward?

[KarstenSchnitter]

@dlvenable, I think, that support for the both data formats JSON and MessagePack would be required along with additional features.

There is a better description for the protocol in the Fluentd docs. The full protocol definition is available at the Fluentd Wiki. You will find, that the server side of the protocol must support MessagePack and JSON message types and also must respond to heartbeat messages received via UDP, although the client can opt to not send those. All of this leads me to believe, that a specific plugin would be required. With a general TCP source, you can only address JSON message format without handshake and heartbeats. I presume, that this would decrease the overall maximum performance of the solution.

IMHO, Fluent Forward is a nice format for (semi-)local application, where routing the TCP and UDP connections is easy. When you have more complex network setups, e.g. providing DataPrepper as a service, it is much more complicated to support the protocol, regardless whether DataPrepper or Fluentd/Fluent Bit would be the server side.

[sandervandegeijn]

@ict-one-nl , Are you suggesting that Data Prepper have a source to support the FluentBit Forward output? I understand that this is more of an internal protocol. Would having a general TCP source such as requested in #2162 provide what you are looking for? Or do you specifically want FluentBit Forward?

I think not, for me it's about interoperability between the different logforwarders and being able to mix/match to make use of the strengths of each one while maintaining performance/interoperability.

Sometimes you have to deal with existing infrastructure which is based on another forwarding infrastructure. Having common protocols help, i.e. I'm using kafka which is supported by most log forwarders (but I can't put kafka between everything as an abstraction layer).