Skip to content

Latest commit

 

History

History
156 lines (134 loc) · 4.12 KB

setup-env.md

File metadata and controls

156 lines (134 loc) · 4.12 KB
Raw

Setup a local environment for Debezium connector spanner

For Mac

Docker (docker compose)

$ brew install docker 
$ brew install docker-compose

other tools we need git, jq, httpie, direv

$ brew install httpie
$ brew install direv
$ brew install git
$ brew install jq

Sdkman install

$ curl -s "https://get.sdkman.io" | bash
# Follow the instructions on-screen to complete installation.
# Next, open a new terminal or enter:
$ source "$HOME/.sdkman/bin/sdkman-init.sh"
$ sdk version

For Ubuntu

Docker (docker compose)

$ sudo apt update 
$ sudo apt install docker-ce docker-ce-cli containerd.io docker-compose-plugin

other tools we need git, jq, httpie, sdkman, direvent, unzip, zip

$ sudo apt install git jq httpie direvent unzip zip

Sdkman install

$ curl -s "https://get.sdkman.io" | bash
# Follow the instructions on-screen to complete installation.
# Next, open a new terminal or enter:
$ source "$HOME/.sdkman/bin/sdkman-init.sh"
$ sdk version

Sdkman is a tool to manage parallel versions of multiple SDKs

Commands to install:

Maven (sdkman)

$ sdk install mvnd

Java (sdkman)

$ sdk install java 11.0.16-tem

IDE setup (Intellij Idea)

install plugitns for IDE

plugins:

direnv
sonarlint

Environment variables

Local setup steps

Add

# Local
127.0.0.1 zookeeper
127.0.0.1 schema-registry
127.0.0.1 kafka
127.0.0.1 rest-proxy
127.0.0.1 control-center

to /etc/hosts on local host

.envrc:

## Examples  
export GOOGLE_APPLICATION_CREDENTIALS="$HOME/work/spanner/utility-operand-danyl.json”
export SPANNER_CONNECTOR_PROJECT_DIR="$HOME/work/spanner/debezium.connector.spanner”
export JAVA_DEBUG_PORT="*:5005”
export KAFKA_DEBUG=y # "y" - debug enabled, "n" - not enabled
export DEBUG_SUSPEND_FLAG=n # "y" - wait until debugger connected, "n" - do not wait

.sdkmanrc:

# Enable auto-env through the sdkman_auto_env config
# Add key=value pairs of SDKs to use below
java=11.0.16-tem

Run kafka in docker-compose

$ docker-compose up -d 
$ docker-compose ps

Maven build and test commands

# build binaries
$ mvn -B clean install 
# integration tests
$ mvn -pl debezium-connector-spanner-integration-tests verify -P integration-tests

$ mvn test --batch-mode -Dmaven.test.failure.ignore=true

Spanner source connector config commands:

#>
http POST localhost:8083/connectors < ./spanner-source.json
http GET localhost:8083/connectors/my_spanner_connector
http DELETE localhost:8083/connectors/my_spanner_connector

Wait for 30-60 sec. Check connector is working:

#>
http kafka-connect:8083/connectors/my_spanner_connector/status

Spanner source connector config example: spanner-source.json

{
    "name": "my_spanner_connector",
    "config": {
        "connector.class": "io.debezium.connector.spanner.SpannerConnector",
        "heartbeat.interval.ms": "300000",
        "gcp.spanner.change.stream": "TestStream",
        "gcp.spanner.project.id": "boxwood-weaver-353315",
        "gcp.spanner.instance.id": "kafka-connector",
        "gcp.spanner.database.id": "stage-1",
        "gcp.spanner.low-watermark.enabled": true,
        "gcp.spanner.low-watermark.update-period.ms": 1000
    }
}

Misc links