Singer Postgres --> Postgres replication demo

[sherifnada]

Demo of replicating data from 2 postgres taps (1 remote, 1 local docker-powered) to a local docker postgres db.
Some learnings:

Setting up a tap requires

1. Creating a python virtual environment for the tap (accomplished via install_connector.sh). This is not strictly required but is strongly recommended.
2. Supplying a json file which configures the tap. Configuration typically includes credentials, but can also include flags that control behavior.
3. Configuring which columns to replicate, a two-step process:
   1. Run “Discovery” to generate the Tap Catalog (it’s a superset of the schema, describes all “streams” you can pull from this tap). Store the catalog as a JSON. Discovery is not necessarily supported by all taps. I’m not certain how a tap is supposed to behave if discovery is not supported.
   2. Mutate the catalog file to indicate which streams to subscribe to, as well their replication mode i.e: whether to incrementally replicate a table or always replicate the full table. Incremental replication requires keeping state. This seems to be kept in a local file and supplied via the --state flag.
   3. I’ve found that the mutation of the catalog file is finicky for a few reasons:
      1. One must manually edit a long and fangled JSON file
      2. “legacy style” taps (postgres sql tap is considered legacy) assume a very slightly different schema of catalog file than newer taps.
4. Invoke the tap within its python env

Setting up a target requires:

1. Creating a virtual env.
2. Supplying a JSON configuration file.
3. Optionally supplying a state file
4. Piping output from stdout into an invocation of the target

This example demonstrates piping of data from two different sources: a publicly available remote postgres database, and a local database stood up with docker.

Generate your own catalog files to and edit them to do full table replication and select one of the tables to replicate in each database. You might need to tell the target database to persist empty tables in case you try to replicate a table that doesn’t exist yet.

Once you’re done, you can poke around the database, creds are in target_config.json.

To check this out, first run ./first_time_setup.sh then ./run.sh

[sherifnada]

To see the checked in catalogs see #3

[cgardens]

nicely done! we're 50% done with this product now 😉