-
Notifications
You must be signed in to change notification settings - Fork 1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Compute and write metrics for rows prior to store writes #763
Conversation
/test test-end-to-end-batch |
Does this fix #747? |
Sort of. It provides an alternative means to monitor elements being ingested when the sink is BigQuery. |
@zhilingc can we add this change in metrics to our documentation please? |
115d99e
to
823bd9c
Compare
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: woop, zhilingc The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
/lgtm |
What this PR does / why we need it:
In PR #568, ingestion metrics were shifted to be computed and written post-store-writes, in order to accurately reflect the values that are actually available to users.
However, for certain stores (like BQ), successfully written rows cannot be retrieved post-write, and as a result feature row metrics are not produced for ingestion jobs writing to such stores. This PR re-introduces inflight feature row metrics as an alternative source of monitoring.
Which issue(s) this PR fixes:
Fixes #
Does this PR introduce a user-facing change?: