Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bytes processed in summary line after a dbt run #2905

Closed
switzer opened this issue Nov 22, 2020 · 1 comment
Closed

Bytes processed in summary line after a dbt run #2905

switzer opened this issue Nov 22, 2020 · 1 comment
Labels
duplicate This issue or pull request already exists enhancement New feature or request

Comments

@switzer
Copy link

switzer commented Nov 22, 2020

Describe the feature

For each dbt model, there is a "bytes processed" section, for example:

[CREATE TABLE (166.3m rows, 278.4 GB processed) in 185.94s]
...
[CREATE TABLE (157.1k rows, 342.0 MB processed) in 18.64s]

However, the summary log is something like this:
Finished running 11 incremental models, 2 table models, 1 view model, 2 hooks in 1159.15s.

Is it possible to sum up the total bytes processed (and total rows), for example:
Finished running 11 incremental models, 2 table models, 1 view model, 2 hooks. Total of 247m rows, 1.4 TB processed in 1159.15s.

Describe alternatives you've considered

none

Additional context

It seems like the summary line was missed when adding bytes processed in v0.18.

Who will this benefit?

Developers who want to keep an eye on bytes processed for cost reasons.

Are you interested in contributing this feature?

Not at this point, but once we have more dbt expertise in our company, we will contribute like we do with other projects.

@switzer switzer added enhancement New feature or request triage labels Nov 22, 2020
@jtcohen6
Copy link
Contributor

Thanks for writing this up @switzer, I think it's a totally reasonable request.

Before we do this, we need to add to dbt-core's run results a dictionary of adapter-specific information, which dbt-bigquery can populate with structured information about bytes processed, as proposed in #2747. There is also an existing feature request (#2079) that wants the more structured logging, as well as the reporting a total in the summary stats line.

I'm going to close this issue as a duplicate, but please comment here if you believe this reflects something that is missing from those two.

@jtcohen6 jtcohen6 added duplicate This issue or pull request already exists and removed triage labels Nov 23, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
duplicate This issue or pull request already exists enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants