Benchmark normalization speed #7741
Labels
autoteam
from/connector-ops
normalization
team/destinations
Destinations team's backlog
type/enhancement
New feature or request
Tell us about the problem you're trying to solve
Following up on #4286, we also need a test that can run normalization on a "larger" amount of data than the current integration tests to study the scalability of the generated models (whether full refresh or incremental).
See https://airbytehq.slack.com/archives/C01MFR03D5W/p1636199069220000
Describe the solution you’d like
Generate X numbers of new rows replicated in all destinations
Measure time/amount of data processed to process the X new rows
hopefully by avoiding reprocessing Y old rows from history (that may be quite large, unnecessary and expensive to work again on)
The text was updated successfully, but these errors were encountered: