Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Persist normalization job failure reasons #14310

Closed
sherifnada opened this issue Jun 30, 2022 · 2 comments
Closed

Persist normalization job failure reasons #14310

sherifnada opened this issue Jun 30, 2022 · 2 comments
Assignees
Labels

Comments

@sherifnada
Copy link
Contributor

sherifnada commented Jun 30, 2022

Tell us about the problem you're trying to solve

If normalization fails, we currently only store the higher-level Temporal scheduling error in the Airbyte db, not the actual failure coming from Normalization (python error from the normalization code or DBT error coming from running normalization on the destination)

Describe the solution you’d like

We should use TRACE messaging to signal errors with attached failure reasons and use this to persists Normalization failure reasons correctly.

This then enables #14926 so we can track Normalization issues in Sentry

@sherifnada
Copy link
Contributor Author

sherifnada commented Jul 14, 2022

EOS update:

  • DBT errors are generally one liners, so parsing them should be doable, need to get a clearer picture of how this would work. DBT v1 has JSON stacktraces but not all destinations do that. So we can't solely rely on DBT v1 log formats. Oracle is the only destination whose DBT v1 support is lagging.
  • Decision: be okay with not parsing Oracle errors to Sentry. It will eventually get v1 support and be part of the Sentry incident reporting. So we'll build v1 log compatibility and accept not reporting Oracle errors for now.
  • Instrumenting normalization worker to parse stdout and therefore parse TRACE messages now mostly happening locally

@Phlair
Copy link
Contributor

Phlair commented Jul 21, 2022

Splitting this issue in two.
This issue now tracks the first part (PR: #14790)
This other issue tracks the second part (parsing and pushing to Sentry)

@Phlair Phlair changed the title Track normalization job failures via Sentry Persist normalization job failure reasons Jul 21, 2022
davinchia added a commit that referenced this issue Aug 12, 2022
…15587)

Bug fix from #14310 . The build was broken then from before #15497 so this was hidden.

- Add the new env vars to the integration test yamls.
- Also make sure to properly close the destination test container db.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants