CP Data Contract - Data Quality Rule

Data Contract Documentation

Set up environment

docker-compose up -d

Register Schema with Rules

We register a schema with a defined rule:

curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" \
    --data '{
            "schemaType": "AVRO",
            "schema": "{\"type\":\"record\",\"name\":\"SensorData\",\"namespace\":\"com.kafkaStreamsExample\",\"fields\":[{\"name\":\"sensorId\",\"type\":{\"type\":\"string\",\"avro.java.string\":\"String\"}},{\"name\":\"type\",\"type\":{\"type\":\"string\",\"avro.java.string\":\"String\"}},{\"name\":\"value\",\"type\":\"float\"},{\"name\":\"unit\",\"type\":{\"type\":\"string\",\"avro.java.string\":\"String\"}},{\"name\":\"timestamp\",\"type\":[{\"type\":\"string\",\"avro.java.string\":\"String\"},\"null\"]}]}",
            "metadata": {
            "properties": {
            "owner": "Patrick Neff",
            "email": "pneff@confluent.io"
            }
        },
        "ruleSet": {
        "domainRules": [
            {
            "name": "checkSensor",
            "kind": "CONDITION",
            "type": "CEL",
            "mode": "WRITE",
            "expr": "message.sensorId == \"sensor_1\"",
            "params": {
            "dlq.topic": "sensor-data-raw-dlq"
            },
            "onFailure": "DLQ"
            }
            ]
        }
    }"' \
http://localhost:8081/subjects/sensor-data-raw-value/versions

Check Schemas

curl -X GET http://localhost:8081/schemas | jq

Run

 ./gradlew run  

The producer produces events continuously. The first 5 records pass the rule and are sent to the sensor-data-raw topic. The second 4 records fail and are sent to the dlq-topic.

Check C3 under localhost:9091. However, C3 only displays the schema fields not the rules etc. (yet)