Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PIP-135: Include MetadataStore backend for Etcd #13717

Closed
merlimat opened this issue Jan 12, 2022 · 3 comments
Closed

PIP-135: Include MetadataStore backend for Etcd #13717

merlimat opened this issue Jan 12, 2022 · 3 comments
Assignees
Labels
Milestone

Comments

@merlimat
Copy link
Contributor

Motivation

Since all the pieces that composed the proposal in PIP-45 were finally merged
and are currently ready for 2.10 release, it is now possible to add other
metadata backends that can be used to support a BookKeeper + Pulsar cluster.

One of the popular systems that is most commonly used in alternative of
ZooKeeper is Etcd, thus it makes sense to have this as the first non-zookeeper
implementation.

Goal

Provide a Etcd implementation for the MetadataStore API. This will allow
users to deploy Pulsar clusters using Etcd service for the metadata and it will
not require the presence of ZooKeeper.

Implementation

  • Use the existing JEtcd Java client library for Etcd
  • Extends the AbstractBatchedMetadataStore class, in order to reuse the
    transparent batching logic that will be shared with the ZookKeeper
    implementation.

Work in progress: #13225

@hpvd
Copy link

hpvd commented Jan 12, 2022

not sure:
is this somehow related/interesting: https://bookkeeper.apache.org/bps/BP-28-etcd-as-metadata-store/

edit: sorry, unqualified, since you work on bk..

@merlimat
Copy link
Contributor Author

@hpvd We went through a different route here:

  • Provide a generic KV pluggable interface for Pulsar metadata access (MetadataStore APIs)
  • Implement Bookkeeper metadata access on top of Pulsar MetadataStore interfaces.

BookKeeper is now able to use any of the Pulsar MetadataStore implementations.

@hpvd
Copy link

hpvd commented Jan 12, 2022

@merlimat
many thanks for explanation!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants