Maybe support Kafka abstraction ?

[joeblew99]

Saw this https://github.com/matrix-org/naffka/blob/master/naffka.go

Its a golang lib wrapping samara, but it got me thinking about the idea of allowing golang devs to easily try Kafka or Liftbridge via a unified API.

See here for Context:
https://github.com/matrix-org/dendrite/search?q=naffka&type=Issues

[tylertreat]

Is your thought to support the Kafka protocol in the server or provide a high-level client that allows switching between Kafka and Liftbridge?

Supporting the Kafka protocol would be really neat and would make adoption/migration really easy. My concern with that is there might be some impedance mismatch / leaky abstraction since, while conceptually similar, it's not an exact mapping and Kafka supports some features that Liftbridge doesn't currently.

Providing a high-level wrapper that implements a lowest common denominator API between Liftbridge and Kafka would be easier. I think this is what you're suggesting?

I also have plans to implement a Kafka bridge for Liftbridge, which would be a Kafka connector that maps topics in Kafka to streams in Liftbridge. This would allow Kafka users to start migrating to Liftbridge incrementally.

[joeblew99]

I agree about the impedance mismatch.
Yes a low level wrapper that does not support all the the Kafka functionality since yes there is an impedance mismatch.

For me my use case is CDC from Cockroach dB and Yugabtye.

yugabyte/yugabyte-db#2513

These and tidb are pretty common Google spanner like databases.
They all use avro schema and expect a Kafka system to be there to push / pull the CDC data to.

Of course there are probably 1000's of other systems that push to Kafka out there too for which liftbridge could offer a lifeline for where you don't want java.

[YourTechBud]

It would be super cool if Liftbridge could support the Kafka wire protocol. Adoption would become so much easier.

I understand there could be feature imparity initially, but the initial development efforts could be to bridge precisely that. Many like myself find Liftbridge as a Kafka alternative after all.

[tylertreat]

Closing this for now as there are currently no plans to support the Kafka protocol.