Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compute and write metrics for rows prior to store writes #763

Merged
merged 2 commits into from
Jun 19, 2020

Conversation

zhilingc
Copy link
Collaborator

@zhilingc zhilingc commented Jun 2, 2020

What this PR does / why we need it:

In PR #568, ingestion metrics were shifted to be computed and written post-store-writes, in order to accurately reflect the values that are actually available to users.

However, for certain stores (like BQ), successfully written rows cannot be retrieved post-write, and as a result feature row metrics are not produced for ingestion jobs writing to such stores. This PR re-introduces inflight feature row metrics as an alternative source of monitoring.

Which issue(s) this PR fixes:

Fixes #

Does this PR introduce a user-facing change?:

Adds the tag "metrics_namespace" to the following metrics:
- feature_row_lag_ms_*
- feature_value_lag_ms_*
- feature_row_ingested_count
- feature_value_missing_count
- feature_value_*

The tag can contain either of:
- `Inflight` 
- `WriteToStoreSuccess`

@zhilingc
Copy link
Collaborator Author

zhilingc commented Jun 2, 2020

/test test-end-to-end-batch

@Yanson
Copy link
Contributor

Yanson commented Jun 2, 2020

Does this fix #747?

@mrzzy mrzzy added the compat/breaking Breaking user-facing change label Jun 3, 2020
@zhilingc
Copy link
Collaborator Author

zhilingc commented Jun 3, 2020

Does this fix #747?

Sort of. It provides an alternative means to monitor elements being ingested when the sink is BigQuery.

@woop woop mentioned this pull request Jun 4, 2020
@woop woop added this to the v0.6.0 milestone Jun 4, 2020
@woop
Copy link
Member

woop commented Jun 9, 2020

@zhilingc can we add this change in metrics to our documentation please?

@feast-ci-bot
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: woop, zhilingc

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@woop
Copy link
Member

woop commented Jun 19, 2020

/lgtm

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants