You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository has been archived by the owner on Sep 30, 2024. It is now read-only.
When we have an error in the save step of our ingest pipeline—as we did with #353—we can end up with partially ingested data. For example, the Parquet write happens after the Zarr is written. If an error happens there, we end up with the data written to a new Zarr group, but not available in the database (or in the Parquet files, of course). So the data is not presented in the application.
We should set up our error handling so that if any part of the ingest pipeline fails, we clear out all of the data and log a useful message.
The text was updated successfully, but these errors were encountered:
When we have an error in the save step of our ingest pipeline—as we did with #353—we can end up with partially ingested data. For example, the Parquet write happens after the Zarr is written. If an error happens there, we end up with the data written to a new Zarr group, but not available in the database (or in the Parquet files, of course). So the data is not presented in the application.
We should set up our error handling so that if any part of the ingest pipeline fails, we clear out all of the data and log a useful message.
The text was updated successfully, but these errors were encountered: