Skip to content

Latest commit

 

History

History
83 lines (76 loc) · 3.09 KB

README.md

File metadata and controls

83 lines (76 loc) · 3.09 KB

YugabyteDB CDC Pipeline

This repository contains simple steps to set up a YugabyteDB CDC pipeline to get the data from YSQL to a Kafka topic

Running the CDC pipeline

The following example uses a PostgreSQL database as the sink database, which will be populated using a JDBC Sink Connector.

  1. Start YugabyteDB This can be a local instance as well as a universe running on Yugabyte Anywhere. All you need is the IP of the nodes where the tserver and master processes are running.
export NODE=<IP-OF-YOUR-NODE>
export MASTERS=<MASTER-ADDRESSES>
  1. Create a table
CREATE TABLE employee (id INT PRIMARY KEY, first_name TEXT, last_name TEXT, dept_id SMALLINT);
  1. Create a stream ID using yb-admin
./yb-admin --master_addresses $MASTERS create_change_data_stream ysql.<namespace>
  1. Start the docker containers
docker-compose up
  1. Deploy the source connector:
curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" localhost:8083/connectors/ -d '{
  "name": "ybconnector",
  "config": {
    "tasks.max":"1",
    "connector.class": "io.debezium.connector.yugabytedb.YugabyteDBConnector",
    "database.hostname":"'$NODE'",
    "database.master.addresses":"'$MASTERS'",
    "database.port":"5433",
    "database.user": "yugabyte",
    "database.password":"yugabyte",
    "database.dbname":"yugabyte",
    "database.server.name":"dbserver1",
    "snapshot.mode":"never",
    "database.streamid":"4a585fa40740459e907ff7fd67497869",
    "table.include.list":"public.employee"
  }
}'

Note: Change the parameters according to the scenario you're working on. 6. Deploy the sink connector

curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" localhost:8083/connectors/ -d @jdbc-sink-pg.json
  1. To login to the PostgreSQL terminal, use:
docker run --network=yb-cdc-demo_default -it --rm --name postgresqlterm --link pg:postgresql --rm postgres:11.2 sh -c 'PGPASSWORD=postgres exec psql -h pg -p "$POSTGRES_PORT_5432_TCP_PORT" -U postgres'
  1. You can start performing operations on the source now and see them getting replicated in the PostgreSQL table.

Adding your own connector to the image

If you want to use your own custom connector to the docker image, be ready with the jar file which has all the files required for the connector and follow the given steps:

  1. Create a directory
mkdir ~/custom-image
  1. Copy the jar file to the directory
cp jarFileXYZ.jar ~/custom-image/
  1. Navigate to the directory and create a Dockerfile
cd ~/custom-image && touch Dockerfile
  1. Edit the contents of the Dockerfile
FROM quay.io/yugabyte/debezium-connector:latest

COPY jarFileXYZ.jar $KAFKA_CONNECT_PLUGINS_DIR/debezium-connector-yugabytedb/
  1. Build your docker image
docker build . -t my-custom-image

NOTE: Do not forget to edit the docker-compose file with this custom image name in case you want to use this newly created image with the docker compose commands