diff --git a/apps/README.md b/apps/README.md index 960f165..47affc5 100644 --- a/apps/README.md +++ b/apps/README.md @@ -5,8 +5,9 @@ | [s-users-pt](s-users-pt/README.md) | Backend | Service | Legacy application which contains the Portugal users | | [s-users-fr](s-users-fr/README.md) | Backend | Service | Legacy application which contains the Portugal | backend | Service | Legacy application which contains the France users | | [ms-users](ms-users/README.md) | Backend | Microservice | Microservice with the consolidate data | +| [ms-users-camel](ms-users-camel/README.md) | Backend | Nanoservice | Application that list all the documents in a MongoDB collection. Exposes a basic REST API with the GET method | | [front-users-pt](front-users-pt/README.md) | Frontent | --- | Portugal users frontend | | [front-users-fr](front-users-fr/README.md) | Frontent | --- | France users frontend | | [front-users](front-users/README.md) | Frontent | --- | Frontend where you can see the consolidate data | | [ns-users-change-reader](ns-users-change-reader/README.md) | Banckend | Nanoservice | Application that reads from a topic where KafkaConnect store the data. It's call to ms-users with the consolidated data | -| [mongodb-front](mongodb-front/README.md) | Frontend+Backend | Nanoservice | Application that list all the documents in a MongoDB collection. Exposes a basic REST API with the GET method, and includes a front at http://hostname/citizens.html | +| \ No newline at end of file diff --git a/apps/ms-users-camel/README.md b/apps/ms-users-camel/README.md index d07f883..63af95e 100644 --- a/apps/ms-users-camel/README.md +++ b/apps/ms-users-camel/README.md @@ -1,6 +1,6 @@ # CAMEL EXTENSION FOR QUARKUS: MongoDB Front -This is a basic application to list the citizens from the MongoDB collection. It exposes a basic REST API with GET method to retrieve the whole list. Also it includes a basic HTML front to show the results. +This is a basic application to list the citizens from the MongoDB collection. It exposes a basic REST API with GET method to retrieve the whole list. ## How to call the API diff --git a/apps/ms-users-camel/pom.xml b/apps/ms-users-camel/pom.xml index 84df845..e3f2054 100644 --- a/apps/ms-users-camel/pom.xml +++ b/apps/ms-users-camel/pom.xml @@ -45,14 +45,6 @@ import - - @@ -71,12 +63,6 @@ camel-quarkus-jackson - - - org.apache.camel.quarkus - camel-quarkus-jsonpath - - org.apache.camel.quarkus camel-quarkus-mongodb @@ -93,11 +79,6 @@ camel-quarkus-bean - - org.apache.camel.quarkus - camel-quarkus-timer - - org.apache.camel.quarkus camel-quarkus-log diff --git a/apps/ns-users-camel-reader/.dockerignore b/apps/ns-users-camel-reader/.dockerignore new file mode 100644 index 0000000..94810d0 --- /dev/null +++ b/apps/ns-users-camel-reader/.dockerignore @@ -0,0 +1,5 @@ +* +!target/*-runner +!target/*-runner.jar +!target/lib/* +!target/quarkus-app/* \ No newline at end of file diff --git a/apps/ns-users-camel-reader/.gitignore b/apps/ns-users-camel-reader/.gitignore new file mode 100644 index 0000000..8c7863e --- /dev/null +++ b/apps/ns-users-camel-reader/.gitignore @@ -0,0 +1,43 @@ +#Maven +target/ +pom.xml.tag +pom.xml.releaseBackup +pom.xml.versionsBackup +release.properties +.flattened-pom.xml + +# Eclipse +.project +.classpath +.settings/ +bin/ + +# IntelliJ +.idea +*.ipr +*.iml +*.iws + +# NetBeans +nb-configuration.xml + +# Visual Studio Code +.vscode +.factorypath + +# OSX +.DS_Store + +# Vim +*.swp +*.swo + +# patch +*.orig +*.rej + +# Local environment +.env + +# Plugin directory +/.quarkus/cli/plugins/ diff --git a/apps/ns-users-camel-reader/README.md b/apps/ns-users-camel-reader/README.md new file mode 100644 index 0000000..2be950b --- /dev/null +++ b/apps/ns-users-camel-reader/README.md @@ -0,0 +1,89 @@ +# Camel Quarkus and Debezium + +This project uses Quarkus, the Supersonic Subatomic Java Framework. If you want to learn more about Quarkus, please visit its website: https://quarkus.io/ . + +The microservice is implemented with Camel extensions for Quarkus. + +## Scenario + +![Big picture](pictures/diagram.png) + + +Process flow: + +1. The microservice consumes _Debezium_ events from a Kafka topic + + Events are extracted from a remote SQL database by a Debezium KafkaConnector already deployed and running. Debezium serializes the events using [AVRO](https://avro.apache.org/) and publishes the schemas into the [Apicurio Registry](https://www.apicur.io/registry/) and the events into a Kafka topic. + + The **Camel Kafka** component consumes the events from the topic and it is also connected to the Apicurio Registry to deserialize the events from AVRO to JSON. + +2. The microservice maps the consumed event into a target POJO/entity by using [AtlasMap](https://www.atlasmap.io/) + + The microservice configuration defines which mapper is used. + + There are two mappers included in the project, for two different source JSON schemas (`france` and `portugal`): + + france-to-central-mapping.adm + + portugal-to-central-mapping.adm + + Both of them map the values to the same target POJO (the `central` JSON schema). + + + The mapping also applies some basic transformations (trimming spaces and it changes some text to uppercase). + + Example: mapping records from the SQL _France_ table into the POJO + + ![Atlasmap - France schema to POJO mapping](pictures/france-mapping.png) + + Example: mapping records from the SQL _Portugal_ table into the POJO + + ![Atlasmap - Portugal schema to POJO mapping](pictures/portugal-mapping.png) + + +3. It enriches the POJO with additional data (just adding the last time the document was updated). + +4. Depending on the debezium event: + + + It creates/inserts a new document into a MongoDB collection persisting the POJO. + + It updates the previous existing document in MongoDB + + It deletes the document in MongoDB + + + +## Running the application in dev mode + +You can run your application in dev mode that enables live coding using: +```shell script +mvn compile quarkus:dev +``` + +> **_NOTE:_** Quarkus now ships with a Dev UI, which is available in dev mode only at http://localhost:8080/q/dev/. + +## Packaging and running the application + +The application can be packaged using: +```shell script +mvn package +``` +It produces the `quarkus-run.jar` file in the `target/quarkus-app/` directory. +Be aware that it’s not an _über-jar_ as the dependencies are copied into the `target/quarkus-app/lib/` directory. + +The application is now runnable using `java -jar target/quarkus-app/quarkus-run.jar`. + +If you want to build an _über-jar_, execute the following command: +```shell script +mvn package -Dquarkus.package.type=uber-jar +``` + +The application, packaged as an _über-jar_, is now runnable using `java -jar target/*-runner.jar`. + +## Creating a native executable + +You can create a native executable using: +```shell script +mvn package -Dnative +``` + +Or, if you don't have GraalVM installed, you can run the native executable build in a container using: +```shell script +mvn package -Dnative -Dquarkus.native.container-build=true +``` + +You can then execute your native executable with: `./target/code-with-quarkus-1.0.0-SNAPSHOT-runner` + +If you want to learn more about building native executables, please consult https://quarkus.io/guides/maven-tooling. + diff --git a/apps/ns-users-camel-reader/gitops/Chart.yaml b/apps/ns-users-camel-reader/gitops/Chart.yaml new file mode 100644 index 0000000..274bbb7 --- /dev/null +++ b/apps/ns-users-camel-reader/gitops/Chart.yaml @@ -0,0 +1,6 @@ +apiVersion: v2 +name: kafka-workshop-producer +description: A Helm chart for Kubernetes +type: application +version: 0.1.0 +appVersion: "1.16.0" \ No newline at end of file diff --git a/apps/ns-users-camel-reader/gitops/fr.dev.values.yaml b/apps/ns-users-camel-reader/gitops/fr.dev.values.yaml new file mode 100644 index 0000000..f4a1337 --- /dev/null +++ b/apps/ns-users-camel-reader/gitops/fr.dev.values.yaml @@ -0,0 +1,12 @@ +service: + name: ns-users-camel-reader-fr + version: 0.1.1 + +kafka: + topic: fr.users.user + +atlas: + mapper: france-to-central-mapping.adm + +config: + env: \ No newline at end of file diff --git a/apps/ns-users-camel-reader/gitops/pt.dev.values.yaml b/apps/ns-users-camel-reader/gitops/pt.dev.values.yaml new file mode 100644 index 0000000..62ba769 --- /dev/null +++ b/apps/ns-users-camel-reader/gitops/pt.dev.values.yaml @@ -0,0 +1,12 @@ +service: + name: ns-users-camel-reader-pt + version: 0.1.1 + +kafka: + topic: pt.users.user + +atlas: + mapper: portugal-to-central-mapping.adm + +config: + env: \ No newline at end of file diff --git a/apps/ns-users-camel-reader/gitops/templates/deployment.yaml b/apps/ns-users-camel-reader/gitops/templates/deployment.yaml new file mode 100644 index 0000000..a242c2e --- /dev/null +++ b/apps/ns-users-camel-reader/gitops/templates/deployment.yaml @@ -0,0 +1,62 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: {{ .Values.service.name }} + namespace: {{ .Release.Namespace }} + labels: + name: {{ .Values.service.name | quote }} + app: {{ .Values.service.name | quote }} + version: {{ .Values.service.version | quote }} +spec: + replicas: {{ .Values.deployment.replicas }} + selector: + matchLabels: + name: {{ .Values.service.name | quote }} + template: + metadata: + labels: + name: {{ .Values.service.name | quote }} + version: {{ .Values.service.version | quote }} + spec: + serviceAccountName: {{ .Values.service.name }} + containers: + - name: {{ .Values.service.name }} + image: {{ .Values.service.image }}:{{ .Values.service.version }} + envFrom: + - secretRef: + name: {{ .Values.service.name }} + env: + {{- range .Values.config.env }} + {{- $envItem := . -}} + {{- with $ }} + - name: {{ $envItem.name | upper | replace "-" "_" | quote}} + value: {{ $envItem.value | quote }} + {{- end }} + {{- end }} + - name: KAFKA_BOOTSTRAP_SERVERS + value: {{ .Values.kafka.bootstrap }} + - name: KAFKA_DBZ_TOPIC_NAME + value: {{ .Values.kafka.topic }} + - name: APICURIO_REGISTRY_URL + value: {{ .Values.kafka.api.avro }} + - name: MONGODB_COLLECTION + value: {{ .Values.mongodb.collection }} + - name: ATLASMAP_MAPPER + value: {{ .Values.atlas.mapper }} + ports: + - name: http + containerPort: {{ .Values.deployment.port }} + protocol: TCP + livenessProbe: + httpGet: + path: {{ .Values.deployment.health.liveness }} + port: http + failureThreshold: 10 + readinessProbe: + httpGet: + path: {{ .Values.deployment.health.readiness }} + port: http + failureThreshold: 10 + resources: + {{- toYaml .Values.deployment.resources | nindent 12 }} +--- \ No newline at end of file diff --git a/apps/ns-users-camel-reader/gitops/templates/secret.yaml b/apps/ns-users-camel-reader/gitops/templates/secret.yaml new file mode 100644 index 0000000..4f6fa23 --- /dev/null +++ b/apps/ns-users-camel-reader/gitops/templates/secret.yaml @@ -0,0 +1,12 @@ +apiVersion: v1 +kind: Secret +metadata: + name: {{ .Values.service.name }} + namespace: {{ .Release.Namespace }} + labels: + name: {{ .Values.service.name | quote }} + version: {{ .Values.service.version | quote }} +data: + MONGODB_DATABASE: YWRtaW4= + MONGODB_LOGIN_DATABASE: YWRtaW4xMjM0 + MONGODB_CONNECTION_STRING: bW9uZ29kYjovL2FkbWluOmFkbWluMTIzNEBtb25nb2RiOjI3MDE3 \ No newline at end of file diff --git a/apps/ns-users-camel-reader/gitops/templates/serviceaccount.yaml b/apps/ns-users-camel-reader/gitops/templates/serviceaccount.yaml new file mode 100644 index 0000000..288db74 --- /dev/null +++ b/apps/ns-users-camel-reader/gitops/templates/serviceaccount.yaml @@ -0,0 +1,6 @@ +apiVersion: v1 +kind: ServiceAccount +metadata: + name: {{ .Values.service.name }} + labels: + name: {{ .Values.service.name | quote }} diff --git a/apps/ns-users-camel-reader/gitops/values.yaml b/apps/ns-users-camel-reader/gitops/values.yaml new file mode 100644 index 0000000..4a98aa7 --- /dev/null +++ b/apps/ns-users-camel-reader/gitops/values.yaml @@ -0,0 +1,33 @@ +service: + name: no-default-value + image: quay.io/dborrego/cdc-ns-users-camel-reader + version: 0.1 + +deployment: + replicas: 1 + port: 8080 + health: + liveness: /q/health/live + readiness: /q/health/ready + resources: + limits: + cpu: 500m + memory: 1024Mi + requests: + cpu: 100m + memory: 256Mi + +kafka: + bootstrap: kafka-kafka-bootstrap.kafka:9092 + topic: no-default-value + api: + avro: http://apicurio-registry-service.kafka:8080/apis/registry/v2 + +atlas: + mapper: no-default-value + +mongodb: + collection: users + +config: + env: diff --git a/apps/ns-users-camel-reader/k8s/configuration.env b/apps/ns-users-camel-reader/k8s/configuration.env new file mode 100644 index 0000000..c933d65 --- /dev/null +++ b/apps/ns-users-camel-reader/k8s/configuration.env @@ -0,0 +1,10 @@ +APICURIO_REGISTRY_URL=http://demo-apicurioregistry-service.amqstreams.svc.cluster.local:8080/apis/registry/v2 +KAFKA_BOOTSTRAP_SERVERS=single-node-cluster-kafka-bootstrap.amqstreams.svc.cluster.local:9092 +KAFKA_DBZ_TOPIC_NAME=france.database.tablename + +ATLASMAP_MAPPER=france-to-central-mapping.adm + +MONGODB_CONNECTION_STRING=mongodb://root:mypassword@mongodb.mongodb.svc.cluster.local:27017 +MONGODB_LOGIN_DATABASE=admin +MONGODB_DATABASE=admin +MONGODB_COLLECTION=myCollection diff --git a/apps/ns-users-camel-reader/k8s/deployment.yaml b/apps/ns-users-camel-reader/k8s/deployment.yaml new file mode 100644 index 0000000..fddec58 --- /dev/null +++ b/apps/ns-users-camel-reader/k8s/deployment.yaml @@ -0,0 +1,56 @@ +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + labels: + app.openshift.io/runtime: quarkus + app.kubernetes.io/version: 1.0.0-SNAPSHOT + app.kubernetes.io/name: camel-quarkus-dbz + name: camel-quarkus-dbz +spec: + replicas: 1 + selector: + matchLabels: + app.kubernetes.io/version: 1.0.0-SNAPSHOT + app.kubernetes.io/name: camel-quarkus-dbz + template: + metadata: + labels: + app.openshift.io/runtime: quarkus + app.kubernetes.io/version: 1.0.0-SNAPSHOT + app.kubernetes.io/name: camel-quarkus-dbz + spec: + containers: + - env: + - name: JAVA_APP_JAR + value: /deployments/quarkus-run.jar + envFrom: + - secretRef: + name: camel-quarkus-dbz-configuration + image: quay.io/ryanezil/camel-quarkus-dbz:1.0.0-SNAPSHOT + imagePullPolicy: Always + livenessProbe: + failureThreshold: 3 + httpGet: + path: /q/health/live + port: 8080 + scheme: HTTP + initialDelaySeconds: 0 + periodSeconds: 30 + successThreshold: 1 + timeoutSeconds: 10 + name: camel-quarkus-dbz + ports: + - containerPort: 8080 + name: http + protocol: TCP + readinessProbe: + failureThreshold: 3 + httpGet: + path: /q/health/ready + port: 8080 + scheme: HTTP + initialDelaySeconds: 0 + periodSeconds: 30 + successThreshold: 1 + timeoutSeconds: 10 diff --git a/apps/ns-users-camel-reader/pictures/diagram.png b/apps/ns-users-camel-reader/pictures/diagram.png new file mode 100644 index 0000000..1d15ea4 Binary files /dev/null and b/apps/ns-users-camel-reader/pictures/diagram.png differ diff --git a/apps/ns-users-camel-reader/pictures/france-mapping.png b/apps/ns-users-camel-reader/pictures/france-mapping.png new file mode 100644 index 0000000..a80e8a2 Binary files /dev/null and b/apps/ns-users-camel-reader/pictures/france-mapping.png differ diff --git a/apps/ns-users-camel-reader/pictures/portugal-mapping.png b/apps/ns-users-camel-reader/pictures/portugal-mapping.png new file mode 100644 index 0000000..30b50c5 Binary files /dev/null and b/apps/ns-users-camel-reader/pictures/portugal-mapping.png differ diff --git a/apps/ns-users-camel-reader/pom.xml b/apps/ns-users-camel-reader/pom.xml new file mode 100644 index 0000000..c059f02 --- /dev/null +++ b/apps/ns-users-camel-reader/pom.xml @@ -0,0 +1,236 @@ + + + 4.0.0 + dev.ryanezil + ns-user-camel-reader + 1.0.0-SNAPSHOT + + 3.11.0 + 17 + 17 + 17 + UTF-8 + UTF-8 + + quarkus-bom + com.redhat.quarkus.platform + + 2.13.8.SP3-redhat-00001 + true + 3.0.0 + + + true + true + + + + + + + + ${quarkus.platform.group-id} + ${quarkus.platform.artifact-id} + ${quarkus.platform.version} + pom + import + + + ${quarkus.platform.group-id} + quarkus-camel-bom + ${quarkus.platform.version} + pom + import + + + + + + + + org.apache.camel.quarkus + camel-quarkus-microprofile-health + + + + org.apache.camel.quarkus + camel-quarkus-jackson + + + + + org.apache.camel.quarkus + camel-quarkus-jsonpath + + + + + + + org.apache.camel.quarkus + camel-quarkus-avro + + + + org.apache.camel.quarkus + camel-quarkus-mongodb + + + + + org.apache.camel.quarkus + camel-quarkus-atlasmap + + + + + org.apache.camel.quarkus + camel-quarkus-direct + + + + org.apache.camel.quarkus + camel-quarkus-bean + + + + + org.apache.camel.quarkus + camel-quarkus-kafka + + + + + org.apache.camel.quarkus + camel-quarkus-log + + + + + + io.quarkus + quarkus-apicurio-registry-avro + + + + + io.quarkus + quarkus-openshift + + + + + io.quarkus + quarkus-arc + + + io.quarkus + quarkus-junit5 + test + + + + + + + + + true + + + false + + redhat + https://maven.repository.redhat.com/ga + + + + + + true + + + false + + redhat + https://maven.repository.redhat.com/ga + + + + + + + + ${quarkus.platform.group-id} + quarkus-maven-plugin + ${quarkus.platform.version} + true + + + + build + generate-code + generate-code-tests + + + + + + maven-compiler-plugin + ${compiler-plugin.version} + + + -parameters + + + + + maven-surefire-plugin + ${surefire-plugin.version} + + + org.jboss.logmanager.LogManager + ${maven.home} + + + + + maven-failsafe-plugin + ${surefire-plugin.version} + + + + integration-test + verify + + + + ${project.build.directory}/${project.build.finalName}-runner + org.jboss.logmanager.LogManager + ${maven.home} + + + + + + + + + + + + + native + + + native + + + + false + native + + + + diff --git a/apps/ns-users-camel-reader/src/main/docker/Dockerfile.jvm b/apps/ns-users-camel-reader/src/main/docker/Dockerfile.jvm new file mode 100644 index 0000000..37804ac --- /dev/null +++ b/apps/ns-users-camel-reader/src/main/docker/Dockerfile.jvm @@ -0,0 +1,97 @@ +#### +# This Dockerfile is used in order to build a container that runs the Quarkus application in JVM mode +# +# Before building the container image run: +# +# ./mvnw package +# +# Then, build the image with: +# +# docker build -f src/main/docker/Dockerfile.jvm -t quarkus/ns-user-camel-reader-jvm . +# +# Then run the container using: +# +# docker run -i --rm -p 8080:8080 quarkus/ns-user-camel-reader-jvm +# +# If you want to include the debug port into your docker image +# you will have to expose the debug port (default 5005 being the default) like this : EXPOSE 8080 5005. +# Additionally you will have to set -e JAVA_DEBUG=true and -e JAVA_DEBUG_PORT=*:5005 +# when running the container +# +# Then run the container using : +# +# docker run -i --rm -p 8080:8080 quarkus/ns-user-camel-reader-jvm +# +# This image uses the `run-java.sh` script to run the application. +# This scripts computes the command line to execute your Java application, and +# includes memory/GC tuning. +# You can configure the behavior using the following environment properties: +# - JAVA_OPTS: JVM options passed to the `java` command (example: "-verbose:class") +# - JAVA_OPTS_APPEND: User specified Java options to be appended to generated options +# in JAVA_OPTS (example: "-Dsome.property=foo") +# - JAVA_MAX_MEM_RATIO: Is used when no `-Xmx` option is given in JAVA_OPTS. This is +# used to calculate a default maximal heap memory based on a containers restriction. +# If used in a container without any memory constraints for the container then this +# option has no effect. If there is a memory constraint then `-Xmx` is set to a ratio +# of the container available memory as set here. The default is `50` which means 50% +# of the available memory is used as an upper boundary. You can skip this mechanism by +# setting this value to `0` in which case no `-Xmx` option is added. +# - JAVA_INITIAL_MEM_RATIO: Is used when no `-Xms` option is given in JAVA_OPTS. This +# is used to calculate a default initial heap memory based on the maximum heap memory. +# If used in a container without any memory constraints for the container then this +# option has no effect. If there is a memory constraint then `-Xms` is set to a ratio +# of the `-Xmx` memory as set here. The default is `25` which means 25% of the `-Xmx` +# is used as the initial heap size. You can skip this mechanism by setting this value +# to `0` in which case no `-Xms` option is added (example: "25") +# - JAVA_MAX_INITIAL_MEM: Is used when no `-Xms` option is given in JAVA_OPTS. +# This is used to calculate the maximum value of the initial heap memory. If used in +# a container without any memory constraints for the container then this option has +# no effect. If there is a memory constraint then `-Xms` is limited to the value set +# here. The default is 4096MB which means the calculated value of `-Xms` never will +# be greater than 4096MB. The value of this variable is expressed in MB (example: "4096") +# - JAVA_DIAGNOSTICS: Set this to get some diagnostics information to standard output +# when things are happening. This option, if set to true, will set +# `-XX:+UnlockDiagnosticVMOptions`. Disabled by default (example: "true"). +# - JAVA_DEBUG: If set remote debugging will be switched on. Disabled by default (example: +# true"). +# - JAVA_DEBUG_PORT: Port used for remote debugging. Defaults to 5005 (example: "8787"). +# - CONTAINER_CORE_LIMIT: A calculated core limit as described in +# https://www.kernel.org/doc/Documentation/scheduler/sched-bwc.txt. (example: "2") +# - CONTAINER_MAX_MEMORY: Memory limit given to the container (example: "1024"). +# - GC_MIN_HEAP_FREE_RATIO: Minimum percentage of heap free after GC to avoid expansion. +# (example: "20") +# - GC_MAX_HEAP_FREE_RATIO: Maximum percentage of heap free after GC to avoid shrinking. +# (example: "40") +# - GC_TIME_RATIO: Specifies the ratio of the time spent outside the garbage collection. +# (example: "4") +# - GC_ADAPTIVE_SIZE_POLICY_WEIGHT: The weighting given to the current GC time versus +# previous GC times. (example: "90") +# - GC_METASPACE_SIZE: The initial metaspace size. (example: "20") +# - GC_MAX_METASPACE_SIZE: The maximum metaspace size. (example: "100") +# - GC_CONTAINER_OPTIONS: Specify Java GC to use. The value of this variable should +# contain the necessary JRE command-line options to specify the required GC, which +# will override the default of `-XX:+UseParallelGC` (example: -XX:+UseG1GC). +# - HTTPS_PROXY: The location of the https proxy. (example: "myuser@127.0.0.1:8080") +# - HTTP_PROXY: The location of the http proxy. (example: "myuser@127.0.0.1:8080") +# - NO_PROXY: A comma separated lists of hosts, IP addresses or domains that can be +# accessed directly. (example: "foo.example.com,bar.example.com") +# +### +FROM registry.access.redhat.com/ubi8/openjdk-17:1.16 + +ENV LANGUAGE='en_US:en' + + +# We make four distinct layers so if there are application changes the library layers can be re-used +COPY --chown=185 target/quarkus-app/lib/ /deployments/lib/ +COPY --chown=185 target/quarkus-app/*.jar /deployments/ +COPY --chown=185 target/quarkus-app/app/ /deployments/app/ +COPY --chown=185 target/quarkus-app/quarkus/ /deployments/quarkus/ + +EXPOSE 8080 +USER 185 +ENV JAVA_OPTS="-Dquarkus.http.host=0.0.0.0 -Djava.util.logging.manager=org.jboss.logmanager.LogManager" +ENV JAVA_APP_JAR="/deployments/quarkus-run.jar" + +ENTRYPOINT [ "/opt/jboss/container/java/run/run-java.sh" ] + diff --git a/apps/ns-users-camel-reader/src/main/docker/Dockerfile.legacy-jar b/apps/ns-users-camel-reader/src/main/docker/Dockerfile.legacy-jar new file mode 100644 index 0000000..1206980 --- /dev/null +++ b/apps/ns-users-camel-reader/src/main/docker/Dockerfile.legacy-jar @@ -0,0 +1,93 @@ +#### +# This Dockerfile is used in order to build a container that runs the Quarkus application in JVM mode +# +# Before building the container image run: +# +# ./mvnw package -Dquarkus.package.type=legacy-jar +# +# Then, build the image with: +# +# docker build -f src/main/docker/Dockerfile.legacy-jar -t quarkus/ns-user-camel-reader-legacy-jar . +# +# Then run the container using: +# +# docker run -i --rm -p 8080:8080 quarkus/ns-user-camel-reader-legacy-jar +# +# If you want to include the debug port into your docker image +# you will have to expose the debug port (default 5005 being the default) like this : EXPOSE 8080 5005. +# Additionally you will have to set -e JAVA_DEBUG=true and -e JAVA_DEBUG_PORT=*:5005 +# when running the container +# +# Then run the container using : +# +# docker run -i --rm -p 8080:8080 quarkus/ns-user-camel-reader-legacy-jar +# +# This image uses the `run-java.sh` script to run the application. +# This scripts computes the command line to execute your Java application, and +# includes memory/GC tuning. +# You can configure the behavior using the following environment properties: +# - JAVA_OPTS: JVM options passed to the `java` command (example: "-verbose:class") +# - JAVA_OPTS_APPEND: User specified Java options to be appended to generated options +# in JAVA_OPTS (example: "-Dsome.property=foo") +# - JAVA_MAX_MEM_RATIO: Is used when no `-Xmx` option is given in JAVA_OPTS. This is +# used to calculate a default maximal heap memory based on a containers restriction. +# If used in a container without any memory constraints for the container then this +# option has no effect. If there is a memory constraint then `-Xmx` is set to a ratio +# of the container available memory as set here. The default is `50` which means 50% +# of the available memory is used as an upper boundary. You can skip this mechanism by +# setting this value to `0` in which case no `-Xmx` option is added. +# - JAVA_INITIAL_MEM_RATIO: Is used when no `-Xms` option is given in JAVA_OPTS. This +# is used to calculate a default initial heap memory based on the maximum heap memory. +# If used in a container without any memory constraints for the container then this +# option has no effect. If there is a memory constraint then `-Xms` is set to a ratio +# of the `-Xmx` memory as set here. The default is `25` which means 25% of the `-Xmx` +# is used as the initial heap size. You can skip this mechanism by setting this value +# to `0` in which case no `-Xms` option is added (example: "25") +# - JAVA_MAX_INITIAL_MEM: Is used when no `-Xms` option is given in JAVA_OPTS. +# This is used to calculate the maximum value of the initial heap memory. If used in +# a container without any memory constraints for the container then this option has +# no effect. If there is a memory constraint then `-Xms` is limited to the value set +# here. The default is 4096MB which means the calculated value of `-Xms` never will +# be greater than 4096MB. The value of this variable is expressed in MB (example: "4096") +# - JAVA_DIAGNOSTICS: Set this to get some diagnostics information to standard output +# when things are happening. This option, if set to true, will set +# `-XX:+UnlockDiagnosticVMOptions`. Disabled by default (example: "true"). +# - JAVA_DEBUG: If set remote debugging will be switched on. Disabled by default (example: +# true"). +# - JAVA_DEBUG_PORT: Port used for remote debugging. Defaults to 5005 (example: "8787"). +# - CONTAINER_CORE_LIMIT: A calculated core limit as described in +# https://www.kernel.org/doc/Documentation/scheduler/sched-bwc.txt. (example: "2") +# - CONTAINER_MAX_MEMORY: Memory limit given to the container (example: "1024"). +# - GC_MIN_HEAP_FREE_RATIO: Minimum percentage of heap free after GC to avoid expansion. +# (example: "20") +# - GC_MAX_HEAP_FREE_RATIO: Maximum percentage of heap free after GC to avoid shrinking. +# (example: "40") +# - GC_TIME_RATIO: Specifies the ratio of the time spent outside the garbage collection. +# (example: "4") +# - GC_ADAPTIVE_SIZE_POLICY_WEIGHT: The weighting given to the current GC time versus +# previous GC times. (example: "90") +# - GC_METASPACE_SIZE: The initial metaspace size. (example: "20") +# - GC_MAX_METASPACE_SIZE: The maximum metaspace size. (example: "100") +# - GC_CONTAINER_OPTIONS: Specify Java GC to use. The value of this variable should +# contain the necessary JRE command-line options to specify the required GC, which +# will override the default of `-XX:+UseParallelGC` (example: -XX:+UseG1GC). +# - HTTPS_PROXY: The location of the https proxy. (example: "myuser@127.0.0.1:8080") +# - HTTP_PROXY: The location of the http proxy. (example: "myuser@127.0.0.1:8080") +# - NO_PROXY: A comma separated lists of hosts, IP addresses or domains that can be +# accessed directly. (example: "foo.example.com,bar.example.com") +# +### +FROM registry.access.redhat.com/ubi8/openjdk-17:1.16 + +ENV LANGUAGE='en_US:en' + + +COPY target/lib/* /deployments/lib/ +COPY target/*-runner.jar /deployments/quarkus-run.jar + +EXPOSE 8080 +USER 185 +ENV JAVA_OPTS="-Dquarkus.http.host=0.0.0.0 -Djava.util.logging.manager=org.jboss.logmanager.LogManager" +ENV JAVA_APP_JAR="/deployments/quarkus-run.jar" + +ENTRYPOINT [ "/opt/jboss/container/java/run/run-java.sh" ] diff --git a/apps/ns-users-camel-reader/src/main/docker/Dockerfile.native b/apps/ns-users-camel-reader/src/main/docker/Dockerfile.native new file mode 100644 index 0000000..b774473 --- /dev/null +++ b/apps/ns-users-camel-reader/src/main/docker/Dockerfile.native @@ -0,0 +1,27 @@ +#### +# This Dockerfile is used in order to build a container that runs the Quarkus application in native (no JVM) mode. +# +# Before building the container image run: +# +# ./mvnw package -Dnative +# +# Then, build the image with: +# +# docker build -f src/main/docker/Dockerfile.native -t quarkus/ns-user-camel-reader . +# +# Then run the container using: +# +# docker run -i --rm -p 8080:8080 quarkus/ns-user-camel-reader +# +### +FROM registry.access.redhat.com/ubi8/ubi-minimal:8.8 +WORKDIR /work/ +RUN chown 1001 /work \ + && chmod "g+rwX" /work \ + && chown 1001:root /work +COPY --chown=1001:root target/*-runner /work/application + +EXPOSE 8080 +USER 1001 + +ENTRYPOINT ["./application", "-Dquarkus.http.host=0.0.0.0"] diff --git a/apps/ns-users-camel-reader/src/main/docker/Dockerfile.native-micro b/apps/ns-users-camel-reader/src/main/docker/Dockerfile.native-micro new file mode 100644 index 0000000..ccd776e --- /dev/null +++ b/apps/ns-users-camel-reader/src/main/docker/Dockerfile.native-micro @@ -0,0 +1,30 @@ +#### +# This Dockerfile is used in order to build a container that runs the Quarkus application in native (no JVM) mode. +# It uses a micro base image, tuned for Quarkus native executables. +# It reduces the size of the resulting container image. +# Check https://quarkus.io/guides/quarkus-runtime-base-image for further information about this image. +# +# Before building the container image run: +# +# ./mvnw package -Dnative +# +# Then, build the image with: +# +# docker build -f src/main/docker/Dockerfile.native-micro -t quarkus/ns-user-camel-reader . +# +# Then run the container using: +# +# docker run -i --rm -p 8080:8080 quarkus/ns-user-camel-reader +# +### +FROM quay.io/quarkus/quarkus-micro-image:2.0 +WORKDIR /work/ +RUN chown 1001 /work \ + && chmod "g+rwX" /work \ + && chown 1001:root /work +COPY --chown=1001:root target/*-runner /work/application + +EXPOSE 8080 +USER 1001 + +ENTRYPOINT ["./application", "-Dquarkus.http.host=0.0.0.0"] diff --git a/apps/ns-users-camel-reader/src/main/java/dev/ryanezil/camel/MergeStrategy.java b/apps/ns-users-camel-reader/src/main/java/dev/ryanezil/camel/MergeStrategy.java new file mode 100644 index 0000000..f7844b0 --- /dev/null +++ b/apps/ns-users-camel-reader/src/main/java/dev/ryanezil/camel/MergeStrategy.java @@ -0,0 +1,39 @@ +package dev.ryanezil.camel; + +import org.apache.camel.AggregationStrategy; +import org.apache.camel.Exchange; + +import dev.ryanezil.camel.model.CommonUser; + +public class MergeStrategy implements AggregationStrategy { + + public Exchange aggregate(Exchange original, Exchange resource) { + + + CommonUser userFromDebezium = original.getIn().getBody(CommonUser.class); + CommonUser userFromMongo = resource.getIn().getBody(CommonUser.class); + + if(userFromMongo != null) { + + /* + * Apply your own mapping implementation for debeziun updates. + * + * MapStruct component might be used with Camel for object mapping between POJOs + */ + + if(userFromDebezium.getRemoteId() != null) userFromMongo.setRemoteId(userFromDebezium.getRemoteId()); + if(userFromDebezium.getFirstName() != null) userFromMongo.setFirstName(userFromDebezium.getFirstName()); + if(userFromDebezium.getLastName() != null) userFromMongo.setLastName(userFromDebezium.getLastName()); + if(userFromDebezium.getEnriched() != null) userFromMongo.setEnriched(userFromDebezium.getEnriched()); + if(userFromDebezium.getEmail() != null) userFromMongo.setEmail(userFromDebezium.getEmail()); + if(userFromDebezium.getPhone() != null) userFromMongo.setPhone(userFromDebezium.getPhone()); + if(userFromDebezium.getGender() != null) userFromMongo.setGender(userFromDebezium.getGender()); + + // Updated object from MongoDB is returned + return resource; + + } else return original; + + } + +} diff --git a/apps/ns-users-camel-reader/src/main/java/dev/ryanezil/camel/Route.java b/apps/ns-users-camel-reader/src/main/java/dev/ryanezil/camel/Route.java new file mode 100644 index 0000000..290e764 --- /dev/null +++ b/apps/ns-users-camel-reader/src/main/java/dev/ryanezil/camel/Route.java @@ -0,0 +1,129 @@ +package dev.ryanezil.camel; + +import java.time.Instant; + +import org.apache.camel.Exchange; +import org.apache.camel.Processor; +import org.apache.camel.builder.RouteBuilder; +import org.apache.camel.model.dataformat.JsonLibrary; + +import dev.ryanezil.camel.model.CommonUser; + +public class Route extends RouteBuilder { + + + @Override + public void configure() throws Exception { + + from("kafka:{{kafka.dbz.topic.name}}" + + // The consumer processor group ID + "?groupId={{kafka.application.groupid}}" + + // What to do when there is no initial offset in ZooKeeper or if an offset is out of range + // 'earliest' automatically reset the offset to the earliest offset + "&autoOffsetReset=earliest" + + // See this link: https://www.apicur.io/registry/docs/apicurio-registry/2.4.x/getting-started/assembly-configuring-kafka-client-serdes.html#registry-serdes-types-avro_registry + "&valueDeserializer=io.apicurio.registry.serde.avro.AvroKafkaDeserializer" + + "&keyDeserializer=io.apicurio.registry.serde.avro.AvroKafkaDeserializer" + + "&additionalProperties.apicurio.registry.url={{apicurio.registry}}" + + "&additionalProperties.apicurio.registry.avro-datum-provider=io.apicurio.registry.serde.avro.ReflectAvroDatumProvider" + ) + .log("Message received from Kafka : ${body}") + .log(" on the topic ${headers[kafka.TOPIC]}") + .log(" on the partition ${headers[kafka.PARTITION]}") + .log(" with the offset ${headers[kafka.OFFSET]}") + .log(" with the key ${headers[kafka.KEY]}") + .choice() + /* + * A database DELETE operation causes Debezium to generate two Kafka records: + * 1. A record that contains "op": "d", the before row data, and some other fields. + * 2. A tombstone record that has the same key as the deleted row and a value of null. + * This record is a marker for Apache Kafka. It indicates that log compaction can + * remove all records that have this key. + */ + .when(body().isNull()).log("Record ignored: the message body is NULL") + .otherwise().to("direct:process-record") + .endChoice(); + + + from("direct:process-record") + .convertBodyTo(String.class) + .setHeader("dbz_operation").jsonpath("$.op") + .setHeader("dbz_source_db").jsonpath("$.source.db") + .setHeader("dbz_source_table").jsonpath("$.source.table") + .log("Debezium event info:") + .log(" >> Debezium Operation = ${header.dbz_operation}") + .log(" >> SourceDB = ${header.dbz_source_db}") + .log(" >> SourceTABLE = ${header.dbz_source_table}") + .choice() + .when(body().isNull()) + .log("The received message body is nULL") + + .when(header("dbz_operation").isEqualTo("c")) + .log("DBZ op='c' - A record was created/inserted") + .setBody().jsonpath("$.after").marshal().json(true) + .log("The retrieved Debezium 'after' block is:\n${body}") + .to("direct:upsert-record") + + .when(header("dbz_operation").isEqualTo("u")) + .log("DBZ op='u' - A record was updated") + .setBody().jsonpath("$.after").marshal().json(true) + .log("The retrieved Debezium 'after' block is:\n${body}") + .to("direct:upsert-record") + + .when(header("dbz_operation").isEqualTo("d")) + .log("DBZ op='d' - A record was deleted") + .setBody().jsonpath("$.before").marshal().json(true) + .log("The retrieved Debezium 'before' block is:\n${body}") + .to("direct:deleted-record") + + .when(header("dbz_operation").isEqualTo("r")) + .log("DBZ op='r' - A record was read (Snapshot event): operation not implemented") + + .otherwise() + .log("Unknown Debezium operation") + + .end(); + + + from("direct:upsert-record") + //Mapping JSON format from DBZ to CommonUser + .toD("atlasmap:atlasmap-mappings/{{atlasmap.mapper}}") + .unmarshal().json(JsonLibrary.Jackson, CommonUser.class ) + .enrich("direct:enrich", new MergeStrategy()) + .process(new Processor() { + // Enriching the document setting the 'Document last update time' using a Camel Processor + public void process(Exchange exchange) throws Exception { + String sourceDatabase = exchange.getIn().getHeader("dbz_source_db",String.class); + String sourceTable = exchange.getIn().getHeader("dbz_source_table",String.class); + + CommonUser commonUser = exchange.getIn().getBody(CommonUser.class); + commonUser.setEnriched("Document updated at UTC time [" + Instant.now().toString() + +"] from remote table [" + sourceDatabase +"." + sourceTable +"]"); + } + }) + // SEE operations here: https://camel.apache.org/components/3.21.x/mongodb-component.html#_endpoint_query_option_operation + .to("mongodb:camelMongoClient?database={{mongodb.database}}&collection={{mongodb.users.collection}}&operation=save") + .log("SAVED MongoDB JSON ${body}"); + + from("direct:enrich") + // retrieves only one element from the collection whose _id field matches the content of the IN message body + .setBody(simple("${body.get_id()}")) + .log("Looking for MongoDB document with _id=[${body}]") + .to("mongodb:camelMongoClient?database={{mongodb.database}}&collection={{mongodb.users.collection}}&operation=findById") + //The returned object is a BSON object. We marshall it to JSON and then unmarshal the JSON to a CommonUser object. + .marshal().json(JsonLibrary.Jackson) + .unmarshal().json(JsonLibrary.Jackson, CommonUser.class ); + + + from("direct:deleted-record") + // The IN message body will act as the removal filter query + // Mapping JSON format from DBZ to CommonUser + .toD("atlasmap:atlasmap-mappings/{{atlasmap.mapper}}") + .log("DELETING body=[${body}]") + // SEE operations here: https://camel.apache.org/components/3.21.x/mongodb-component.html#_endpoint_query_option_operation + .to("mongodb:camelMongoClient?database={{mongodb.database}}&collection={{mongodb.users.collection}}&operation=remove") + .log("DELETED MongoDB JSON ${body}"); + + } //END configure() method + +} \ No newline at end of file diff --git a/apps/ns-users-camel-reader/src/main/java/dev/ryanezil/camel/model/CommonUser.java b/apps/ns-users-camel-reader/src/main/java/dev/ryanezil/camel/model/CommonUser.java new file mode 100644 index 0000000..ef7e76b --- /dev/null +++ b/apps/ns-users-camel-reader/src/main/java/dev/ryanezil/camel/model/CommonUser.java @@ -0,0 +1,109 @@ +package dev.ryanezil.camel.model; + +import java.io.Serializable; + +import com.fasterxml.jackson.annotation.JsonProperty; + +import io.quarkus.runtime.annotations.RegisterForReflection; + +@RegisterForReflection +public class CommonUser implements Serializable { + + @JsonProperty + private String _id; + + @JsonProperty + private Long remoteId; + + @JsonProperty + private String firstName; + + @JsonProperty + private String lastName; + + @JsonProperty + private String enriched; + + @JsonProperty + private String email; + + @JsonProperty + private String phone; + + @JsonProperty + private String gender; + + + public String get_id() { + return _id; + } + + public void set_id(String _id) { + this._id = _id; + } + + public Long getRemoteId() { + return remoteId; + } + + public void setRemoteId(Long remoteId) { + this.remoteId = remoteId; + } + + public String getFirstName() { + return firstName; + } + + public void setFirstName(String firstName) { + this.firstName = firstName; + } + + public String getLastName() { + return lastName; + } + + public void setLastName(String lastName) { + this.lastName = lastName; + } + + public String getEnriched() { + return enriched; + } + + public void setEnriched(String enriched) { + this.enriched = enriched; + } + + + public String getEmail() { + return email; + } + + public void setEmail(String email) { + this.email = email; + } + + public String getPhone() { + return phone; + } + + public void setPhone(String phone) { + this.phone = phone; + } + + public String getGender() { + return gender; + } + + public void setGender(String gender) { + this.gender = gender; + } + + @Override + public String toString() { + return "CommonUser [_id=" + _id + ", remoteId=" + remoteId + ", firstName=" + firstName + ", lastName=" + + lastName + ", enriched=" + enriched + ", email=" + email + ", phone=" + phone + ", gender=" + gender + + "]"; + } + +} diff --git a/apps/ns-users-camel-reader/src/main/resources/application.properties b/apps/ns-users-camel-reader/src/main/resources/application.properties new file mode 100644 index 0000000..b703fea --- /dev/null +++ b/apps/ns-users-camel-reader/src/main/resources/application.properties @@ -0,0 +1,63 @@ +# +# Quarkus +# +quarkus.banner.enabled = true +quarkus.log.level=INFO +quarkus.native.resources.includes=atlasmap-mappings/*.adm + +###################### +# KAFKA configuration +###################### +# Example here: https://github.com/apache/camel-quarkus-examples/blob/main/kafka/src/main/resources/application.properties +kafka.dbz.topic.name=${KAFKA_DBZ_TOPIC_NAME} +%dev.kafka.dbz.topic.name=france.database.table +kafka.application.groupid=camel-quarkus-dbz + +apicurio.registry=${APICURIO_REGISTRY_URL} +%dev.apicurio.registry=http://demo-apicurioregistry.amqstreams.router-default.apps-crc.testing/apis/registry/v2 + +camel.component.kafka.brokers=${KAFKA_BOOTSTRAP_SERVERS} +# anonymous access and plain communications for non-dev environment + +# DEV/Localhost +%dev.quarkus.kafka.devservices.enabled=false +%dev.camel.component.kafka.brokers=single-node-cluster-kafka-bootstrap-amqstreams.apps-crc.testing:443 +%dev.camel.component.kafka.security-protocol=SASL_SSL +%dev.camel.component.kafka.sasl-mechanism=SCRAM-SHA-512 +%dev.camel.component.kafka.sasl-jaas-config=org.apache.kafka.common.security.scram.ScramLoginModule required username=superuser password=adiI9tczRyaBI4VdXJke4LDHUTywBeir; +%dev.camel.component.kafka.ssl-truststore-location=/tmp/ca.p12 +%dev.camel.component.kafka.ssl-truststore-type=PKCS12 +%dev.camel.component.kafka.ssl-truststore-password=xxxxxxxxxxxx +#%dev.camel.component.kafka.additional-properties=SET-ADDITIONAL-PROPERTIES + +###################### +# MAPPING +###################### +%dev.atlasmap.mapper=france-to-central-mapping.adm +atlasmap.mapper=${ATLASMAP_MAPPER} + +######################### +# MONGODB configuration +######################### +# See: https://quarkus.io/guides/mongodb#quarkus-mongodb_configuration +# The Camel Quarkus MongoDB extension automatically registers a MongoDB client bean named 'camelMongoClient' +# It can be referenced in the mongodb endpoint URI connectionBean path paramete +%dev.quarkus.mongodb.connection-string=mongodb://root:example@localhost:27017 +%dev.quarkus.mongodb.database=admin +%dev.mongodb.database=admin +%dev.mongodb.users.collection=myCollection + +quarkus.mongodb.connection-string=${MONGODB_CONNECTION_STRING} +quarkus.mongodb.database=${MONGODB_LOGIN_DATABASE} +quarkus.mongodb.application-name=camel-quarkus-dbz +mongodb.database=${MONGODB_DATABASE} +mongodb.users.collection=${MONGODB_COLLECTION} + + + +#### OpenShift deployment extension +quarkus.openshift.route.expose=false +quarkus.openshift.deployment-kind=Deployment + +# Secret name with ENV VARS parameter values +quarkus.openshift.env.secrets=camel-quarkus-dbz-configuration diff --git a/apps/ns-users-camel-reader/src/main/resources/atlasmap-mappings/france-to-central-mapping.adm b/apps/ns-users-camel-reader/src/main/resources/atlasmap-mappings/france-to-central-mapping.adm new file mode 100644 index 0000000..d8b2a8d Binary files /dev/null and b/apps/ns-users-camel-reader/src/main/resources/atlasmap-mappings/france-to-central-mapping.adm differ diff --git a/apps/ns-users-camel-reader/src/main/resources/atlasmap-mappings/portugal-to-central-mapping.adm b/apps/ns-users-camel-reader/src/main/resources/atlasmap-mappings/portugal-to-central-mapping.adm new file mode 100644 index 0000000..1c33a6a Binary files /dev/null and b/apps/ns-users-camel-reader/src/main/resources/atlasmap-mappings/portugal-to-central-mapping.adm differ diff --git a/gitops/components/service-registry.yaml b/gitops/components/service-registry.yaml new file mode 100644 index 0000000..3a05249 --- /dev/null +++ b/gitops/components/service-registry.yaml @@ -0,0 +1,17 @@ +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: service-registry + namespace: openshift-gitops +spec: + destination: + namespace: kafka + server: "https://kubernetes.default.svc" + source: + path: gitops/tenants/service-registry + repoURL: "https://github.com/dbgjerez/workshop-cdc" + targetRevision: 13-ns-users-change-reader-schema-registry-integration + project: default + syncPolicy: + syncOptions: + - CreateNamespace=true \ No newline at end of file diff --git a/gitops/core/applications/ns-users-camel-reader.yaml b/gitops/core/applications/ns-users-camel-reader.yaml new file mode 100644 index 0000000..e1a3d7f --- /dev/null +++ b/gitops/core/applications/ns-users-camel-reader.yaml @@ -0,0 +1,37 @@ +apiVersion: argoproj.io/v1alpha1 +kind: ApplicationSet +metadata: + name: ns-users-camel-reader + namespace: openshift-gitops +spec: + generators: + - list: + elements: + - country: fr + env: dev + - country: pt + env: dev + template: + metadata: + name: 'ns-users-camel-reader-{{country}}' + labels: + app: 'ns-users-camel-reader-{{country}}' + type: backend + spec: + destination: + namespace: demo + server: "https://kubernetes.default.svc" + source: + path: apps/ns-users-camel-reader/gitops/ + repoURL: "https://github.com/dbgjerez/workshop-cdc" + targetRevision: 13-ns-users-change-reader-schema-registry-integration + helm: + valueFiles: + - '{{country}}.{{env}}.values.yaml' + project: default + syncPolicy: + automated: + prune: true + selfHeal: false + syncOptions: + - CreateNamespace=true \ No newline at end of file diff --git a/gitops/core/operators/service-registry-operator.yaml b/gitops/core/operators/service-registry-operator.yaml new file mode 100644 index 0000000..0873b37 --- /dev/null +++ b/gitops/core/operators/service-registry-operator.yaml @@ -0,0 +1,11 @@ +apiVersion: operators.coreos.com/v1alpha1 +kind: Subscription +metadata: + name: service-registry-operator + namespace: openshift-operators +spec: + channel: 2.x + installPlanApproval: Automatic + name: service-registry-operator + source: redhat-operators + sourceNamespace: openshift-marketplace \ No newline at end of file diff --git a/gitops/tenants/kafka-connect/kafka-connect.yaml b/gitops/tenants/kafka-connect/kafka-connect.yaml index d3b96ee..3c010e3 100644 --- a/gitops/tenants/kafka-connect/kafka-connect.yaml +++ b/gitops/tenants/kafka-connect/kafka-connect.yaml @@ -16,8 +16,11 @@ spec: - name: debezium-connector-mysql artifacts: - type: zip - url: https://maven.repository.redhat.com/ga/io/debezium/debezium-connector-mysql/2.1.4.Final-redhat-00001/debezium-connector-mysql-2.1.4.Final-redhat-00001-plugin.zip - sha512sum: cb8ca43cc19b0a6cd705ccf77f5d29757493a61b4398226ef8b02cfc5cbea8dd3b76321c5ae669d4b526379896d068097bfa7189d237ffefbc9fdc8a03596498 + url: https://maven.repository.redhat.com/ga/io/debezium/debezium-connector-mysql/2.3.4.Final-redhat-00001/debezium-connector-mysql-2.3.4.Final-redhat-00001-plugin.zip + sha512sum: 16d9b74375d2561ef851b681b9957783b13ae5bf78ed52008d314939d6f38701ffdbe0d1208bf9e4a441631643338b36ad353aed52bea34aeb61d83d9e756783 + - type: zip + url: https://maven.repository.redhat.com/ga/io/apicurio/apicurio-registry-distro-connect-converter/2.4.4.Final-redhat-00002/apicurio-registry-distro-connect-converter-2.4.4.Final-redhat-00002.zip + sha512sum: 8269c007e4b58d95db72ab8364c6a619d2762081a2d6b55102ccbf47f4c2219be6bfa1c212ae5bcc6cdfcaeae3415f6b0e4416c40bfd91270ef45dd1787b286a bootstrapServers: 'kafka-kafka-bootstrap:9092' config: group.id: kafka-connect \ No newline at end of file diff --git a/gitops/tenants/kafka-connect/users-fr.yaml b/gitops/tenants/kafka-connect/users-fr.yaml index 0288ed0..dcdf7d4 100644 --- a/gitops/tenants/kafka-connect/users-fr.yaml +++ b/gitops/tenants/kafka-connect/users-fr.yaml @@ -19,5 +19,16 @@ spec: schema.history.internal.kafka.bootstrap.servers: 'kafka-kafka-bootstrap:9092' database.history.kafka.bootstrap.servers: 'kafka-kafka-bootstrap:9092' database.history.kafka.topic: "history.demo.fr" - include.schema.changes: "false" + include.schema.changes: "false" + + # Register AVRO schema into Service Registry + schema.name.adjustment.mode: avro + key.converter: io.apicurio.registry.utils.converter.AvroConverter + key.converter.apicurio.registry.url: http://apicurio-registry-service:8080/apis/registry/v2 + key.converter.apicurio.registry.auto-register: true + key.converter.apicurio.registry.find-latest: true + value.converter: io.apicurio.registry.utils.converter.AvroConverter + value.converter.apicurio.registry.url: http://apicurio-registry-service:8080/apis/registry/v2 + value.converter.apicurio.registry.auto-register: true + value.converter.apicurio.registry.find-latest: true tasksMax: 1 \ No newline at end of file diff --git a/gitops/tenants/kafka-connect/users-pt.yaml b/gitops/tenants/kafka-connect/users-pt.yaml index 64d1181..5e78975 100644 --- a/gitops/tenants/kafka-connect/users-pt.yaml +++ b/gitops/tenants/kafka-connect/users-pt.yaml @@ -20,4 +20,15 @@ spec: database.history.kafka.bootstrap.servers: "kafka-kafka-bootstrap:9092" database.history.kafka.topic: "history.demo.pt" include.schema.changes: "false" + + # Register AVRO schema into Service Registry + schema.name.adjustment.mode: avro + key.converter: io.apicurio.registry.utils.converter.AvroConverter + key.converter.apicurio.registry.url: http://apicurio-registry-service:8080/apis/registry/v2 + key.converter.apicurio.registry.auto-register: true + key.converter.apicurio.registry.find-latest: true + value.converter: io.apicurio.registry.utils.converter.AvroConverter + value.converter.apicurio.registry.url: http://apicurio-registry-service:8080/apis/registry/v2 + value.converter.apicurio.registry.auto-register: true + value.converter.apicurio.registry.find-latest: true tasksMax: 1 diff --git a/gitops/tenants/service-registry/service-registry.yaml b/gitops/tenants/service-registry/service-registry.yaml new file mode 100644 index 0000000..d360b41 --- /dev/null +++ b/gitops/tenants/service-registry/service-registry.yaml @@ -0,0 +1,10 @@ +apiVersion: registry.apicur.io/v1 +kind: ApicurioRegistry +metadata: + name: apicurio-registry + namespace: kafka +spec: + configuration: + persistence: 'kafkasql' + kafkasql: + bootstrapServers: 'kafka-kafka-bootstrap:9092' \ No newline at end of file