13 ns users change reader schema registry integration

apps/README.md

@@ -5,8 +5,9 @@
 | [s-users-pt](s-users-pt/README.md)             | Backend  | Service      | Legacy application which contains the Portugal users                                                                    |
 | [s-users-fr](s-users-fr/README.md)             | Backend  | Service      | Legacy application which contains the Portugal              | backend  | Service      | Legacy application which contains the France users                                                                      |
 | [ms-users](ms-users/README.md)               | Backend  | Microservice | Microservice with the consolidate data                                                                                  |
+| [ms-users-camel](ms-users-camel/README.md) | Backend | Nanoservice | Application that list all the documents in a MongoDB collection. Exposes a basic REST API with the GET method |
 | [front-users-pt](front-users-pt/README.md)         | Frontent | ---          | Portugal users frontend                                                                                                 |
 | [front-users-fr](front-users-fr/README.md)         | Frontent | ---          | France users frontend                                                                                                   |
 | [front-users](front-users/README.md)            | Frontent | ---          | Frontend where you can see the consolidate data                                                                         |
 | [ns-users-change-reader](ns-users-change-reader/README.md) | Banckend | Nanoservice  | Application that reads from a topic where KafkaConnect store the data. It's call to ms-users with the consolidated data |
-| [mongodb-front](mongodb-front/README.md) | Frontend+Backend | Nanoservice | Application that list all the documents in a MongoDB collection. Exposes a basic REST API with the GET method, and includes a front at http://hostname/citizens.html  |
+|

apps/ms-users-camel/README.md

@@ -1,6 +1,6 @@
 # CAMEL EXTENSION FOR QUARKUS: MongoDB Front
 
-This is a basic application to list the citizens from the MongoDB collection. It exposes a basic REST API with GET method to retrieve the whole list. Also it includes a basic HTML front to show the results.
+This is a basic application to list the citizens from the MongoDB collection. It exposes a basic REST API with GET method to retrieve the whole list.
 
 ## How to call the API
 

apps/ms-users-camel/pom.xml

@@ -45,14 +45,6 @@
         <scope>import</scope>
       </dependency>
 
- <!-- <dependency>
-  <groupId>io.quarkus.platform</groupId>
-  <artifactId>quarkus-camel-bom</artifactId>
-  <version>3.2.6.Final</version>
-  <type>pom</type>
-  <scope>import</scope>
-</dependency>   -->
-
     </dependencies>
   </dependencyManagement>
   <dependencies>
@@ -71,12 +63,6 @@
       <artifactId>camel-quarkus-jackson</artifactId>
     </dependency>
 
-    <!-- jsonpath is used in routes to process Debezium message -->
-    <dependency>
-      <groupId>org.apache.camel.quarkus</groupId>
-      <artifactId>camel-quarkus-jsonpath</artifactId>
-    </dependency>
-
     <dependency>
       <groupId>org.apache.camel.quarkus</groupId>
       <artifactId>camel-quarkus-mongodb</artifactId>
@@ -93,11 +79,6 @@
       <artifactId>camel-quarkus-bean</artifactId>
     </dependency>
 
-    <dependency>
-      <groupId>org.apache.camel.quarkus</groupId>
-      <artifactId>camel-quarkus-timer</artifactId>
-    </dependency>
-
    <dependency>
     <groupId>org.apache.camel.quarkus</groupId>
     <artifactId>camel-quarkus-log</artifactId>

apps/ns-users-camel-reader/.dockerignore

@@ -0,0 +1,5 @@
+*
+!target/*-runner
+!target/*-runner.jar
+!target/lib/*
+!target/quarkus-app/*

apps/ns-users-camel-reader/.gitignore

@@ -0,0 +1,43 @@
+#Maven
+target/
+pom.xml.tag
+pom.xml.releaseBackup
+pom.xml.versionsBackup
+release.properties
+.flattened-pom.xml
+
+# Eclipse
+.project
+.classpath
+.settings/
+bin/
+
+# IntelliJ
+.idea
+*.ipr
+*.iml
+*.iws
+
+# NetBeans
+nb-configuration.xml
+
+# Visual Studio Code
+.vscode
+.factorypath
+
+# OSX
+.DS_Store
+
+# Vim
+*.swp
+*.swo
+
+# patch
+*.orig
+*.rej
+
+# Local environment
+.env
+
+# Plugin directory
+/.quarkus/cli/plugins/

apps/ns-users-camel-reader/README.md

@@ -0,0 +1,89 @@
+# Camel Quarkus and Debezium
+
+This project uses Quarkus, the Supersonic Subatomic Java Framework. If you want to learn more about Quarkus, please visit its website: https://quarkus.io/ .
+
+The microservice is implemented with Camel extensions for Quarkus.
+
+## Scenario
+
+![Big picture](pictures/diagram.png)
+
+
+Process flow:
+
+1. The microservice consumes _Debezium_ events from a Kafka topic
+   + Events are extracted from a remote SQL database by a Debezium KafkaConnector already deployed and running. Debezium serializes the events using [AVRO](https://avro.apache.org/) and publishes the schemas into the [Apicurio Registry](https://www.apicur.io/registry/) and the events into a Kafka topic.
+   + The **Camel Kafka** component consumes the events from the topic and it is also connected to the Apicurio Registry to deserialize the events from AVRO to JSON.
+
+2. The microservice maps the consumed event into a target POJO/entity by using [AtlasMap](https://www.atlasmap.io/)
+   + The microservice configuration defines which mapper is used.
+   + There are two mappers included in the project, for two different source JSON schemas (`france` and `portugal`):
+     + france-to-central-mapping.adm
+     + portugal-to-central-mapping.adm
+
+     Both of them map the values to the same target POJO (the `central` JSON schema).
+
+   + The mapping also applies some basic transformations (trimming spaces and it changes some text to uppercase).
+
+    Example: mapping records from the SQL _France_ table into the POJO
+
+    ![Atlasmap - France schema to POJO mapping](pictures/france-mapping.png)
+
+    Example: mapping records from the SQL _Portugal_ table into the POJO
+
+    ![Atlasmap - Portugal schema to POJO mapping](pictures/portugal-mapping.png)
+
+
+3. It enriches the POJO with additional data (just adding the last time the document was updated).
+
+4. Depending on the debezium event:
+
+   + It creates/inserts a new document into a MongoDB collection persisting the POJO.
+   + It updates the previous existing document in MongoDB
+   + It deletes the document in MongoDB
+
+
+
+## Running the application in dev mode
+
+You can run your application in dev mode that enables live coding using:
+```shell script
+mvn compile quarkus:dev
+```
+
+> **_NOTE:_**  Quarkus now ships with a Dev UI, which is available in dev mode only at http://localhost:8080/q/dev/.
+
+## Packaging and running the application
+
+The application can be packaged using:
+```shell script
+mvn package
+```
+It produces the `quarkus-run.jar` file in the `target/quarkus-app/` directory.
+Be aware that it’s not an _über-jar_ as the dependencies are copied into the `target/quarkus-app/lib/` directory.
+
+The application is now runnable using `java -jar target/quarkus-app/quarkus-run.jar`.
+
+If you want to build an _über-jar_, execute the following command:
+```shell script
+mvn package -Dquarkus.package.type=uber-jar
+```
+
+The application, packaged as an _über-jar_, is now runnable using `java -jar target/*-runner.jar`.
+
+## Creating a native executable
+
+You can create a native executable using: 
+```shell script
+mvn package -Dnative
+```
+
+Or, if you don't have GraalVM installed, you can run the native executable build in a container using: 
+```shell script
+mvn package -Dnative -Dquarkus.native.container-build=true
+```
+
+You can then execute your native executable with: `./target/code-with-quarkus-1.0.0-SNAPSHOT-runner`
+
+If you want to learn more about building native executables, please consult https://quarkus.io/guides/maven-tooling.
+

apps/ns-users-camel-reader/gitops/Chart.yaml

@@ -0,0 +1,6 @@
+apiVersion: v2
+name: kafka-workshop-producer
+description: A Helm chart for Kubernetes
+type: application
+version: 0.1.0
+appVersion: "1.16.0"

apps/ns-users-camel-reader/gitops/fr.dev.values.yaml

@@ -0,0 +1,12 @@
+service:
+  name: ns-users-camel-reader-fr
+  version: 0.1.1
+
+kafka:
+  topic: fr.users.user
+
+atlas:
+  mapper: france-to-central-mapping.adm
+
+config:
+  env:

apps/ns-users-camel-reader/gitops/pt.dev.values.yaml

@@ -0,0 +1,12 @@
+service:
+  name: ns-users-camel-reader-pt
+  version: 0.1.1
+
+kafka:
+  topic: pt.users.user
+
+atlas:
+  mapper: portugal-to-central-mapping.adm
+
+config:
+  env:

apps/ns-users-camel-reader/gitops/templates/deployment.yaml

@@ -0,0 +1,62 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: {{ .Values.service.name }}
+  namespace: {{ .Release.Namespace }}
+  labels:
+    name: {{ .Values.service.name | quote }}
+    app: {{ .Values.service.name | quote }}
+    version: {{ .Values.service.version | quote }}
+spec:
+  replicas: {{ .Values.deployment.replicas }}
+  selector:
+    matchLabels:
+      name: {{ .Values.service.name | quote }}
+  template:
+    metadata:
+      labels:
+        name: {{ .Values.service.name | quote }}
+        version: {{ .Values.service.version | quote }}
+    spec:
+      serviceAccountName: {{ .Values.service.name }}
+      containers:
+        - name: {{ .Values.service.name }}
+          image: {{ .Values.service.image }}:{{ .Values.service.version }}
+          envFrom:
+            - secretRef:
+                name: {{ .Values.service.name }}
+          env:
+          {{- range .Values.config.env }}
+          {{- $envItem := . -}}
+          {{- with $ }}
+            - name: {{ $envItem.name | upper |  replace "-" "_" | quote}}
+              value: {{ $envItem.value | quote }}
+          {{- end }}
+          {{- end }}
+          - name: KAFKA_BOOTSTRAP_SERVERS
+            value: {{ .Values.kafka.bootstrap }}
+          - name: KAFKA_DBZ_TOPIC_NAME
+            value: {{ .Values.kafka.topic }}
+          - name: APICURIO_REGISTRY_URL
+            value: {{ .Values.kafka.api.avro }}
+          - name: MONGODB_COLLECTION
+            value: {{ .Values.mongodb.collection }}
+          - name: ATLASMAP_MAPPER
+            value: {{ .Values.atlas.mapper }}
+          ports:
+            - name: http
+              containerPort: {{ .Values.deployment.port }}
+              protocol: TCP
+          livenessProbe:
+            httpGet:
+              path: {{ .Values.deployment.health.liveness }}
+              port: http
+            failureThreshold: 10
+          readinessProbe:
+            httpGet:
+              path: {{ .Values.deployment.health.readiness }}
+              port: http
+            failureThreshold: 10
+          resources:
+            {{- toYaml .Values.deployment.resources | nindent 12 }}
+---

apps/ns-users-camel-reader/gitops/templates/secret.yaml

@@ -0,0 +1,12 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: {{ .Values.service.name }}
+  namespace: {{ .Release.Namespace }}
+  labels:
+    name: {{ .Values.service.name | quote }}
+    version: {{ .Values.service.version | quote }}
+data:
+  MONGODB_DATABASE: YWRtaW4=
+  MONGODB_LOGIN_DATABASE: YWRtaW4xMjM0
+  MONGODB_CONNECTION_STRING: bW9uZ29kYjovL2FkbWluOmFkbWluMTIzNEBtb25nb2RiOjI3MDE3

apps/ns-users-camel-reader/gitops/templates/serviceaccount.yaml

@@ -0,0 +1,6 @@
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: {{ .Values.service.name }}
+  labels:
+    name: {{ .Values.service.name | quote }}

apps/ns-users-camel-reader/gitops/values.yaml

@@ -0,0 +1,33 @@
+service:
+  name: no-default-value
+  image: quay.io/dborrego/cdc-ns-users-camel-reader
+  version: 0.1
+
+deployment:
+  replicas: 1
+  port: 8080
+  health:
+    liveness: /q/health/live
+    readiness: /q/health/ready
+  resources:
+    limits:
+      cpu: 500m
+      memory: 1024Mi
+    requests:
+      cpu: 100m
+      memory: 256Mi
+
+kafka:
+  bootstrap: kafka-kafka-bootstrap.kafka:9092
+  topic: no-default-value
+  api:
+    avro: http://apicurio-registry-service.kafka:8080/apis/registry/v2
+
+atlas:
+  mapper: no-default-value
+
+mongodb:
+  collection: users
+
+config:
+  env:

apps/ns-users-camel-reader/k8s/configuration.env

@@ -0,0 +1,10 @@
+APICURIO_REGISTRY_URL=http://demo-apicurioregistry-service.amqstreams.svc.cluster.local:8080/apis/registry/v2
+KAFKA_BOOTSTRAP_SERVERS=single-node-cluster-kafka-bootstrap.amqstreams.svc.cluster.local:9092
+KAFKA_DBZ_TOPIC_NAME=france.database.tablename
+
+ATLASMAP_MAPPER=france-to-central-mapping.adm
+
+MONGODB_CONNECTION_STRING=mongodb://root:mypassword@mongodb.mongodb.svc.cluster.local:27017
+MONGODB_LOGIN_DATABASE=admin
+MONGODB_DATABASE=admin
+MONGODB_COLLECTION=myCollection