Design and list out possible approaches for Data Reconciliation

[shashwat-egov]

Data in DIGIT flows from domain modules(microservices) to messaging queue. Persister(microservice which performs CRUD operations and adds data to our RDBMS instance i.e. PostgreSQL) and Indexer(microservice which performs ETL operations and adds data to our analytical database i.e. ElasticSearch) services (part of DIGIT Core) listen on these data emission events and persist data in PostgreSQL and ElasticSearch respectively.

Sometimes, due to errors, data records fail to persist in these databases. This might happen due to a variety of reasons like infra issues in ElasticSearch, Kafka or service downtime due to misconfiguration etc. Thus, we need to design and create a data reconciliation service that syncs data between the two database instances and keeps them both in sync with each other.