You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Add a column to processing_history to store the name of the final BQ dataset a set of pipeline output files was loaded into.
e.g. ingesting clinvar_vcv_2024_10_10_kyle_dev and clinvar_rcv_2024_10_10_kyle_dev each with xml_release_date=2024-10-10 creates a dataset called clinvar_2024_10_10_kyle_dev. This is included in the slack message but not persisted into the processing_history table. It can be inferred using the same logic that was used to decide it in bq-ingest, but persisting it in a column makes it easy to lookup from queries.
Store it both in the vcv and the rcv row.
This field can be set during the same UPDATE that runs to set the release_date field at the end of bq-ingest.
Add a column to
processing_history
to store the name of the final BQ dataset a set of pipeline output files was loaded into.e.g. ingesting
clinvar_vcv_2024_10_10_kyle_dev
andclinvar_rcv_2024_10_10_kyle_dev
each withxml_release_date=2024-10-10
creates a dataset calledclinvar_2024_10_10_kyle_dev
. This is included in the slack message but not persisted into the processing_history table. It can be inferred using the same logic that was used to decide it in bq-ingest, but persisting it in a column makes it easy to lookup from queries.Store it both in the vcv and the rcv row.
This field can be set during the same
UPDATE
that runs to set therelease_date
field at the end of bq-ingest.The text was updated successfully, but these errors were encountered: