Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add transactional json logging for dbt invocations #1799

Closed
drewbanin opened this issue Oct 2, 2019 · 0 comments · Fixed by #1806
Closed

Add transactional json logging for dbt invocations #1799

drewbanin opened this issue Oct 2, 2019 · 0 comments · Fixed by #1806
Labels
enhancement New feature or request

Comments

@drewbanin
Copy link
Contributor

Describe the feature

This PR adds a json logger to the dbt project. When json logging is used, dbt should emit "transactional" log lines that report on the state of the run. From these logs alone, external processes should be able to paint a UI with a superset of the stdout logging that dbt presents on the CLI. This includes:

  • The current state of the dbt project (internal, parsing, running)
  • The current state of the resources that have run, or are running:
  • state: running, complete
  • status: running, cancelled, success, fail, error, etc.
  • status_message: (the thing we print on stdout at resource completion)
  • timing: (started_at, finished_at)
  • error: if an error occurred in the scope of a resource, the error (as a string)

Implementation notes:

  • log lines that occur during each of internal, parsing, and running should be tagged with an attribute, run_state (or similar), so they can be collected and rendered together
  • internal is for log lines that occur outside the scope of parsing and running
  • these log lines should be easily identifiable from the log stream, eg with the channel attribute, or similar
  • we can either cram this information into the extra dict, or we can represent this data some other way if that makes more sense.
@drewbanin drewbanin added enhancement New feature or request triage and removed triage labels Oct 2, 2019
@drewbanin drewbanin added this to the Louisa May Alcott milestone Oct 2, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant