You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Most of the time spent on the ingestion pipeline is taken up by the process of validating the fhirflat format, which requires creating a pydantic class object for each resource then re-creating the FHIRflat parquet.
Splitting up the pipeline into two steps (which can be run together using a single command if desired) should allow for greater parallelisation (see #44) and allows for data to be read in and edited before validation, without having to run the whole pipeline again for a few validation errors.
The text was updated successfully, but these errors were encountered:
Most of the time spent on the ingestion pipeline is taken up by the process of validating the fhirflat format, which requires creating a pydantic class object for each resource then re-creating the FHIRflat parquet.
Splitting up the pipeline into two steps (which can be run together using a single command if desired) should allow for greater parallelisation (see #44) and allows for data to be read in and edited before validation, without having to run the whole pipeline again for a few validation errors.
The text was updated successfully, but these errors were encountered: