Kafka authentication/encryption

[renehonig]

Motivation

I would like to use the Falcosidekick Kafka output to communicate with a public Kafka endpoint. This does require authentication and encryption to secure the connection between Falcosidekick and the Kafka endpoint.

Feature
Enable authentication and encryption for Kafka in Falcosidekick. Mutual TLS seems to be a good choice, as it has already been implemented for many Falcosidekick outputs and is supported by Kafka.

Alternatives

An alternative is to create a private tunnel between the nodes running Falcosidekick and Kafka. This adds additional complexity and is not a preferred solution.

[Issif]

@cpanato as you integrated the Kafka output, do you want/can to work on that? 🙏

[cpanato]

@Issif yes i can take care
just need to read something about kafka auth and encrypt. @renehonig do you have any handy docs? :)

[renehonig]

@cpanato , thanks for your quick response and help!

Here are some links that should get you started on Kafka security. Given that many other Falcosidekick outputs support mTLS, that might be a good place to start:

• official Kafka documentation: https://kafka.apache.org/documentation/#security
• good article about implementation Kafka security: https://rosowski.medium.com/kafka-security-fundamentals-adding-tls-to-your-event-driven-utility-belt-432307f4ff62
• popular Kafka Helm chart: https://github.com/bitnami/charts/tree/master/bitnami/kafka/#installing-the-chart
• popular Kafka k8s operator: https://strimzi.io/

Hope this helps, please let me know if you need more.

[cpanato]

thank you @renehonig

[dirien]

@cpanato, we could make also a little session togther. We use, for example most of the time SASL or client cert.

[WaldoFR]

Hi, I did not see this issue but I had the same problem.

I did implement SASL + TLS here : https://github.com/WaldoFR/falcosidekick however it seems that the project kafka-go is not very resilient, I got two major issues :

• Topic autocreate has not been implemented : WIP: Allow Writer to auto-create topics (Fixes #683) segmentio/kafka-go#700
• When a client is created, it try to interfer with all topics on brokers : Before writing Client request all Topics from the Cluster segmentio/kafka-go#690

I'd like to reimplement the client to use confluent-kafka-go instead wich seem more reliable. If everyone is ok I'll reimplement this part to use confluent's module and realize a PR.

Regards,

[Issif]

@WaldoFR first implementation of kafka output by @cpanato used confluent-kafka-go but we switched to current library because we had trouble with the C libraries dependency. See #139

[dirien]

@WaldoFR, Question would be, if confluent-kafka-go is compatible with other kafka services too? Like Strimzi or Aiven...

[renehonig]

@dirien, @WaldoFR , Thanks for your work on this issue. I think it is compatible with other Kafka distributions too. The readme says:

• Future proof - Confluent, founded by the
  creators of Kafka, is building a streaming platform
  with Apache Kafka at its core. It's high priority for us that client features keep
  pace with core Apache Kafka and components of the [Confluent Platform](https://www.confluent.io/product/compare/

Is it based on librdkafka, which is a generic Kafka library.

[WaldoFR]

There is a PR segmentio/kafka-go#700 in kafka-go module, it's not merged yet but it might solves both issues mentioned in my last post. I'm going to verify that today, if it solve both issues the best might be to keep this module to avoid new unexpected problems from an other module.

[cpanato]

@WaldoFR are you working on this? i did not have much time :( but if you handling this that will be awesome, let me know if you need any help

[poiana]

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale

[poiana]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh with /remove-lifecycle rotten.

Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle rotten

[poiana]

Rotten issues close after 30d of inactivity.

Reopen the issue with /reopen.

Mark the issue as fresh with /remove-lifecycle rotten.

Provide feedback via https://github.com/falcosecurity/community.
/close

[poiana]

@poiana: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue with /reopen.

Mark the issue as fresh with /remove-lifecycle rotten.

Provide feedback via https://github.com/falcosecurity/community.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[poiana]

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale

[poiana]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh with /remove-lifecycle rotten.

Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle rotten

[poiana]

Rotten issues close after 30d of inactivity.

Reopen the issue with /reopen.

Mark the issue as fresh with /remove-lifecycle rotten.

Provide feedback via https://github.com/falcosecurity/community.
/close

[poiana]

@poiana: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue with /reopen.

Mark the issue as fresh with /remove-lifecycle rotten.

Provide feedback via https://github.com/falcosecurity/community.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.