Fluentd / Fluentbit comparison to Loki / Promtail

[geota]

Fluentd is one primary players in this space. It would be worthwhile for potential users of loki / promtail to understand the differences between these two log aggregators / forwarders.

[javefang]

According to Grafana Lab guys, it looks like promtail will label each log message with pod labels while the fluentd plugin doesn't automatically and has to be configured (then it's hard to make it dynamic if pods have different labels I guess). I don't really understand why the fluentd plugin can't do the same thing. Some clarification from the maintainer will be great i think?

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[cyriltovena]

@rfratto this should be covered by our new documentation.

[mdgreenwald]

@rfratto this should be covered by our new documentation.

Do you have a link to the relevant documentation?

[rfratto]

@mdgreenwald

Sorry for the confusion, he's just saying I should add this information in our docs revamp that I'm working on this week.

[neoakris]

@rfratto
Any updates on this, and if you're updating docs anyways can you add answers to these questions in your docs or at least mention if it's possible to have promtail and fluentd work nice together with each other and AM (Alert Manager)?

Questions:
1.) Which is better today for generating AMAlerts from logs? Promtail or fluentd? Are there plans to make one better at sending Alerts on log messages to AM in the future?
2.) My understanding is both can create Prometheus metrics from logs and AM alerts can be generated from that. Are there plans to make it easier for Promtail to send alerts to AM directly? Current solutions seem like a workaround or will that be the end goal? (it being the end goal would make sense from a consistency standpoint.)
3.) I currently believe (from reading/no hands-on) fluentd is better at transforming logs and promtail is better at syncing tags. Can your documentation tell us a way of getting the best of both worlds?
Example: promtail --> autoconfigured from Prometheus Operator ServiceMonitor objects spawning an auto generated Prometheus Scrape Config --> (gain labels sync with prometheus) --> ship to fluentbit/fluentd --> (gain their mature ecosystem of mature / off the shelf ready to use log parser plugins etc.) --> use fluentd plugin to ship to Loki.

Side Benefit of the above strategy 3.):

• Maybe fluentd will be best/easier to config for generating AM alerts, maybe promtail will be best, by hitting both you can use either.
• If the Loki staff adopt it, it could simplify the scope of work around promtail.

My understanding is that:
1.) Kubernetes Ingress objects allow you to chunk a 200,000 nginx config into 500 snippets of LB config called Ingress objects.
2.) Prometheus Operator's ServiceMonitor objects work similar to Ingress objects, You can create a bunch of ServiceMonitor objects to auto-generate Prometheus Scrape config.
3.) Loki uses the same config as Prometheus, so you should be able to autoconfigure Promtail by pointing it to auto-generated Prometheus Scrape config in theory.
4.) The reason Promtail exists is for consistency of tags between Prometheus Metrics and Loki Log.
5.) fluentd has tons of plugins / pipelines / off the shelf transformations

[rfratto]

@neoakris My current PR is here with the current comparison document here.

I'm hesitant to add work-in-progress details to our documentation since there's not much point comparing a fully featured product to a hypothetical, but I'll try to address your questions to the best of my ability here.

I'm not sure that I would say we want Promtail to be better than fluentd since the use cases are different. As you've mentioned, Promtail is a tailored solution for Loki; since it shares code with Prometheus, you can end up switching between metrics and logs on the fly. But that doesn't necessarily mean that the Prometheus config and Loki config are exactly the same either:

1. Promtail can only scrape logs from the machine Promtail is running on since it needs to read files from disk. There's some little extra work required in the scrape configurations to get this working properly. Simplify promtail config #1012 was opened to try to help with this, though
2. Promtail may be configured with extra labels that don't make sense for metrics, like filename.

As far as alerting goes, we've fairly recently made Loki's API Prometheus-compatible. I believe this means you can now generate alerts from Loki queries, and I think @slim-bean has done some work playing around with this. There aren't any current plans to have Promtail interact directly with Alert Manager, but that's not a hard stance.

For getting Promtail to place nice with Fluentd, that's not a use case we're covering (yet?). Promtail can only ingest data from text files on disk/systemd and can only send it out to Loki. I can pretty confidently say that we can't use Fluentd to simplify the work around Promtail since we don't want to add any more required dependencies than we already have (in fact; we're working to minimize dependencies now).

We do want to give extra power to Promtail with its pipelines, and I think supporting external libraries would help with that. I've just opened #1055 to track this.

Despite all I said here, I'm definitely not an authoritative voice on Loki 🙂 I'd recommend for you to open up issues for any ideas you might have.

[adamdecaf]

FYI, there's a logging-operator project that just released a v2 which would be good to compare loki against. They seem to use fluentbit (format) as well, so it's in the same space as loki.

https://github.com/banzaicloud/logging-operator