Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Apache Pulsar support #16

Merged
merged 11 commits into from
Jul 6, 2019
Merged

Add Apache Pulsar support #16

merged 11 commits into from
Jul 6, 2019

Conversation

bsideup
Copy link
Owner

@bsideup bsideup commented Oct 4, 2018

No description provided.

@bsideup bsideup changed the title Pulsar Add Apache Pulsar support Oct 4, 2018
Supplier<CompletionStage<Map<Integer, Long>>> offsetsProvider
) {
return Mono
.defer(() -> Mono.fromCompletionStage(((PulsarClientImpl) pulsarClient).getNumberOfPartitions(topic)))
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why so many casts? :)

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ask Pulsar guys :D

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For some reason this method is not a part of the interface

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just saw this what is the project trying to do ?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For some reason this method is not a part of the interface

Pulsar tries to hide the "partitions" complexity from the users in its unified messaging model. Applications don't have to know the "partitions" under the hood. That's why it was not part of the interface.

alternatively, you can use PulsarAdmin for querying the partition metadata : https://github.com/apache/pulsar/blob/master/pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/Topics.java#L263

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@aahmed-se we're using Liiklus at https://www.vivy.com as our CQRS gateway. Currently we run it with Kafka but plan to deploy Pulsar next to Kafka and move some of our workloads as a first step (with kafka-to-pulsar replication), this is why I added Pulsar support to Liiklus :)

@sijie I see. Thanks for explanation! I was using pulsar-kafka-compat as a ref, this is where I found this code. I didn't want to use PulsarAdmin given its "admin" functionality (and also another transport)

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bsideup I see. make senses. we can consider exposing "partitions" in the pulsar client api, will bring it back to the community to discuss /cc @merlimat

}

@Override
public boolean equals(Object o) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should be hashcode implemented too?

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works like that, only implemented because of the tests

# Conflicts:
#	tck/src/main/java/com/github/bsideup/liiklus/records/tests/ConsumerGroupTest.java
@bsideup bsideup removed this from the next milestone Oct 30, 2018
@bsideup bsideup added this to the next milestone Jun 22, 2019
@bsideup bsideup added the enhancement New feature or request label Jun 22, 2019
@bsideup bsideup modified the milestone: next Jul 5, 2019
@bsideup bsideup merged commit 1e8813d into master Jul 6, 2019
@bsideup bsideup deleted the pulsar branch July 6, 2019 19:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request plugin/pulsar
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants