This repository is playground for the debezium-kafka-postgresql.
Read the following:
- Tutorial: https://debezium.io/documentation/reference/stable/tutorial.html
- Postgres connector config: https://debezium.io/documentation/reference/stable/connectors/postgresql.html#postgresql-example-configuration
- Avro serialization: https://debezium.io/documentation/reference/stable/configuration/avro.html
- Apicurio schema registry: https://www.apicur.io/registry/docs/apicurio-registry/2.4.x/index.html
- Examples: https://github.com/debezium/debezium-examples
docker compose up -d
curl -i -X POST \
-H "Accept:application/json" \
-H "Content-Type:application/json" \
http://localhost:58083/connectors/ \
-d @register-postgres.json
Alternatively, you can use the avro connector.
This connector uses the Apicurio
schema registry.
Firstly you need to create the apicurio
schema in the database
(see 7. Apicurio schema registry)
Attach the connector with:
curl -i -X POST \
-H "Accept:application/json" \
-H "Content-Type:application/json" \
http://localhost:58083/connectors/ \
-d @register-postgres-avro.json
curl -H "Accept:application/json" localhost:58083/connectors/
curl -i -X GET -H "Accept:application/json" localhost:58083/connectors/inventory-connector
You should see something like the following
HTTP/1.1 200 OK
Date: Thu, 06 Feb 2020 22:12:03 GMT
Content-Type: application/json
Content-Length: 531
Server: Jetty(9.4.20.v20190813)
{
"name": "inventory-connector",
...
"tasks": [
{
"connector": "inventory-connector",
"task": 0
}
]
}
kafka-ui dashboard: http://localhost:58000/
debezium-ui dashboard: http://localhost:58001/
The registry is not going to start up.
First, we need to create a schema with name apicurio
under the postgres
database.
Then just restart the service using:
docker compose start apicurio
apicurio-ui dashboard: http://localhost:58002/
docker compose down -v
Connector for PostgreSQL does not create events for schema changes.
Add column with default value should be avoided. Instead, we need to do the following:
- Alter the table and add the column with null value.
- Update all rows.
- Alter table set not null and default value.
alter table customers add column test text null;
update customers set test='test';
alter table customers alter column test set not null;
alter table customers alter column test set default 'test';
Check here: https://github.com/birdiecare/connect-smts Also: https://jcustenborder.github.io/kafka-connect-documentation/projects/kafka-connect-json-schema/transformations/FromJson.html
By default, the directory /kafka/connect
is used as plugin directory by the Debezium Docker image for Kafka Connect.
So any additional connectors you may wish to use should be added to that directory.
Alternatively,
you can add further directories to the plugin path
by specifying the KAFKA_CONNECT_PLUGINS_DIR
environment variable when starting the container
(e.g. -e KAFKA_CONNECT_PLUGINS_DIR=/kafka/connect/,/path/to/further/plugins
).
When using the container image for Kafka Connect provided by Confluent,
you can specify the CONNECT_PLUGIN_PATH
environment variable to achieve the same.