dbt compile / run is really slow

[luck02]

Please make sure to fill out either the issue template or the feature template and delete the other one!

Issue

when I issue a dbt run --models etc.folders or dbt run ... it takes as much as a minute or three to complete.

Issue description

dbt run --models etc.model or dbt compile --models etc.model

Results

There are about 65 models in that directory and I would expect those to run in a reasonable time (5-15 seconds?)

It actually takes closer to 2-3 minutes.

System information

The output of dbt --version:

installed version: 0.11.1
   latest version: 0.12.1

The operating system you're running on:
OSX 10.13.6 (17G2307)

The python version you're using (probably the output of python --version)
Python 2.7.10

Steps to reproduce

run dbt compile / run in a larger project.

In this case we have about 400 models total in the project, and about 65 in the folder I'm trying to compile.

Ask:

Is this normal execution time and is expected? It's problematic as we're working with some autogenerated models and I'm trying to exclude the generated files that came out with issues (unrelated problem). DBT only compiles to the first failure which means I have to wait a couple of minutes for my feedback loop.

Are there any changes I could make that would expedite my execution? Thanks!

--edit--
I am aware that v0.12.11 is released and we'll upgrade soon-ish... However our production is still on 0.11.1 and I'd like to be in sync with prod. However is 0.12 has improvements that will help I'd definitely move now.

[drewbanin]

Thanks for the report @luck02! I'd definitely recommend at least trying 0.12.1 out locally -- this version drastically cuts down on the number of queries that dbt needs to execute in compile and run, which should meaningfully decrease your runtime.

Are you by chance using Snowflake? In my experience, Snowflake projects frequently take especially long to compile because the introspective queries (eg. against information_schema.tables) take a super long time to return. In 0.12.x, dbt caches these query results, so things should move a lot faster.

Can you give 0.12.1 a shot and let me know if there's any change in performance?

[luck02]

hrm, 12.1 isn't available on brew just yet.

--edit-- disregard, it's just not tagged as 12.1. IE I can't @0.12.1 install it... but brew install dbt gets 0.12.1

[luck02]

Ok, @drewbanin 12.1 is much faster, thanks for the headsup!

[drewbanin]

@luck02 awesome, glad to hear it! We're making speed one of our priorities going forwards. More to come on that front with #1128 and #1020

[luck02]

I mean, don't get me wrong, it's not super fast either but it's much better... Still on the slow side. Thanks for helping me out!

[Savbrom]

Hi guys, I was wondering if anybody could help me. When running the klaviyo package for dbt and setting the lookback window to 7 days (336 hours), all dbt models are executed except the last one. Visual Studio Code stops running before all models could be built. Any suggestions?