-
Notifications
You must be signed in to change notification settings - Fork 1.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add a generate_database_name() macro #1695
Comments
Hey @whisperstream - I think this is a good idea! We historically have recommended building all dbt models into a single database - can you tell me more about why you're interested in building models across databases? I'm all for building the functionality described in this feature request, but I also want to get a handle on what a typical use case looks like :) |
Mostly this comes from an organizational perspective. My (current) plan is to have 3 databases, staging, bizready and marts. Within each of the first two databases I use one schema per source of data. I'm working on a large enterprise project and I'm trying to enable multiple teams to contribute dbt scripts for the data sources they have access to. This is further complicated by having two tiers of dbt developers. Tier 1 - dev_raw Tier 2 - dev analyst I could use different dbt_project.yml for the different dev types but I want to keep the lineage data when generating docs. It seems that for most companies, it's assumed the dbt developer has access to run the whole dbt_project.yml in one go, in my case, only the integration and production environments will have this capability. If there's a better way of structuring the project am open to that too, just haven't found/thought of another way and so worked around it by creating an |
Thanks for the context @whisperstream - this is really interesting! One approach that may be interesting: you can use multiple different dbt projects and they can state each other as dependencies. Check out the docs on packages here if you haven't seen them already. In your case, the Tier 2 package could "import" the Tier 2 package. That would still let you I think there's merit to implementing a |
@drewbanin thanks for that, let me try and arrange my current project into a multi package model and see if that works too. |
Ok back again. I think packages help separate out the projects into logical parts, but I still need to do the rewriting. For Tier1 devs - the But for Tier2 devs, even when models are separated into different packages I still need to have an Another semi related item I wondered was if
That way if people have more complicated things to do with ref, they can do it in a more seamless way, that also keeps everything appearing as standard dbt syntax. |
Hey @whisperstream - check out this thread. It's a long one, but we set forth some approaches in this thread to make it possible to override Can you show me what your
What happens when a Tier 2 analyst runs a Tier 1 model? Do they just exclude those models from their dbt runs? Can you also share what you would want your I think the pattern you're describing is a really good one and I'm super interested in making sure that dbt does a good job of facilitating it! I appreciate you taking the time :) |
Let me clean up my xref macro a bit and add some comments, in the mean time I'll try and explain this part So just to preface this, what I describe below (and the condition in Normally in dbt you create a hierarchy of models, i.e.
This assumes you have access to So when a tier2 developer executes:
dbt will try and execute the whole DAG, which will fail because tier2 can't access the
That way the tier2 developer doesn't need access to anything including or proceeding bizready because they don't have access rights to it anyway, but they're still able to contribute to and execute dbt for any Conversely a tier1 developer or a production process executing the same |
Ok here's what my xref macro is looking like: all bizready tables are prefixed with In prod mode or when Originally I was getting the custom schema name if the developer mode was Not sure if you have some better ways I might solve this use case, but being able to oevrride the
|
…abase-name add generate_database_name macro (#1695)
Describe the feature
In the current version (v14.0), there exists a
{{ generate_schema_name_for_env }}
macro which works very well in dev mode, allowing a production run to write to the specified schema but writing all tables and views to a dev schema when in dev mode. In the same way we need a{{generate_database_name_for_env}}
macro for when a database is configured in thedbt_project.yml
file.Currently:
If I have in my
dbt_project.yml
file a section inmodels
that reads:and in my
profiles.yml
file I have:Then my models in dev mode will be written using
mart_db
instead ofdev_db
.Describe alternatives you've considered
Right now to solve this I've created alternative
ref
macro calledxref
to override this behavior but it feels a bit clunky to do this and I will have to tell out dbt devs to all use{{ xref('some table') }}
instead of the inbuiltref
function.Additional context
Not database specific, it's a dbt issue.
Who will this benefit?
Anyone who wants to specify a set of production databases in their
dbt_project.yml
file in the same way that they might already do for their schemas using the existing{{generate_database_name_for_env}}
macro but who also wants to have dbt write all tables and views into a single schema when in dev_mode.The text was updated successfully, but these errors were encountered: