You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
For each dbt model, there is a "bytes processed" section, for example:
[CREATE TABLE (166.3m rows, 278.4 GB processed) in 185.94s]
...
[CREATE TABLE (157.1k rows, 342.0 MB processed) in 18.64s]
However, the summary log is something like this: Finished running 11 incremental models, 2 table models, 1 view model, 2 hooks in 1159.15s.
Is it possible to sum up the total bytes processed (and total rows), for example: Finished running 11 incremental models, 2 table models, 1 view model, 2 hooks. Total of 247m rows, 1.4 TB processed in 1159.15s.
Describe alternatives you've considered
none
Additional context
It seems like the summary line was missed when adding bytes processed in v0.18.
Who will this benefit?
Developers who want to keep an eye on bytes processed for cost reasons.
Are you interested in contributing this feature?
Not at this point, but once we have more dbt expertise in our company, we will contribute like we do with other projects.
The text was updated successfully, but these errors were encountered:
Thanks for writing this up @switzer, I think it's a totally reasonable request.
Before we do this, we need to add to dbt-core's run results a dictionary of adapter-specific information, which dbt-bigquery can populate with structured information about bytes processed, as proposed in #2747. There is also an existing feature request (#2079) that wants the more structured logging, as well as the reporting a total in the summary stats line.
I'm going to close this issue as a duplicate, but please comment here if you believe this reflects something that is missing from those two.
Describe the feature
For each dbt model, there is a "bytes processed" section, for example:
However, the summary log is something like this:
Finished running 11 incremental models, 2 table models, 1 view model, 2 hooks in 1159.15s.
Is it possible to sum up the total bytes processed (and total rows), for example:
Finished running 11 incremental models, 2 table models, 1 view model, 2 hooks. Total of 247m rows, 1.4 TB processed in 1159.15s.
Describe alternatives you've considered
none
Additional context
It seems like the summary line was missed when adding bytes processed in v0.18.
Who will this benefit?
Developers who want to keep an eye on bytes processed for cost reasons.
Are you interested in contributing this feature?
Not at this point, but once we have more dbt expertise in our company, we will contribute like we do with other projects.
The text was updated successfully, but these errors were encountered: