Update reingestion workflows to load and report data

[stacimc]

Problem

As described in #1642, it is necessary to backfill data for our dated DAGs. This is being resolved in the short term by turning catchup on for those DAGs. The long term fix is to re-enable our existing ingestion workflows (example wikimedia_ingestion_workflow).

Description

Copying context from this comment:

These are daily DAGs that generate a list of days, generally weighted toward more recent days, and then runs just the pull_data step for that day.

The purpose of these DAGs was to reingest/update old data in a smart way, with an emphasis on updating more recent data first. I believe the intention was to update popularity data and such, but if we could get these DAGs working, they would also naturally backfill data over time.

The workflows only run the pull_data step; tsvs are generated but the data is not loaded into the DB. So we’d need to update the workflow to make sure the data is loaded. But as an additional benefit, we would be able to run these independently of the normal ingestion process and so would not interrupt typical ingestion.

We'll need to update the DAG factory for ingestion workflows to include the load_data steps, and ideally also report data at the end of reingestion.

Implementation

• 🙋 I would be interested in implementing this feature.