Feature/alias #800

drewbanin · 2018-06-18T22:49:56Z

Reopened from #651, #771

Description (and original code courtesy of @abelsonlive). Thanks also to @jon-rtr for helping bring this branch up-to-date with development.

WHAT

This P.R. implements model aliasing RE: #637.

WHY

To enable more control over how dbt materializes tables.

HOW

This P.R. adds an alias parameter to the model config. If not present, this parameter will default to the name of the model (AKA the parsed file path of the model). This implementation is preferable because it maintains the use of the filename in ref calls, thereby enabling dbt users to have distinct model names, but non-distinct table names.

...

Model aliasing is implemented for seed and model resources. An alias can be specified in dbt_project.yml or in a {{ config(alias=...) }} block. The name attribute of a Relation object will reflect the alias, so {{ this.name }} will return a model alias if one is present.

…er/dbt into feature/model-aliasing

…o model-aliasing $ git merge kickstarter/feature/model-aliasing CONFLICT (content): Merge conflict in dbt/utils.py CONFLICT (modify/delete): dbt/include/global_project/macros/materializations/table.sql deleted in HEAD and modified in kickstarter/feature/model-aliasing. Version kickstarter/feature/model-aliasing of dbt/include/global_project/macros/materializations/table.sql left in tree. CONFLICT (modify/delete): dbt/include/global_project/macros/materializations/bigquery.sql deleted in HEAD and modified in kickstarter/feature/model-aliasing. Version kickstarter/feature/model-aliasing of dbt/include/global_project/macros/materializations/bigquery.sql left in tree. 1. dbt/utils.py Some major changes are being introduced in 0.10.1: Implement relations api (#727) #727 5344f54#diff-196bbfafed32edaf1554550f65111f87 The Relation class was extracted into ... ./dbt/api/object.py class APIObject(dict) ./dbt/adapters/default/relation.py class DefaultRelation(APIObject) ./dbt/adapters/bigquery/relation.py class BigQueryRelation(DefaultRelation) ./dbt/adapters/snowflake/relation.py class SnowflakeRelation(DefaultRelation) Changing node.get('name') to node.get('alias') ... ./dbt/adapters/default/relation.py ./dbt/adapters/bigquery/relation.py ./dbt/adapters/snowflake/relation.py 2. dbt/include/global_project/macros/materializations/table.sql This was renamed to ... ./dbt/include/global_project/macros/materializations/table/table.sql 3. dbt/include/global_project/macros/materializations/bigquery.sql This was split into ... ./dbt/include/global_project/macros/materializations/table/bigquery_table.sql and ... ./dbt/include/global_project/macros/materializations/view/bigquery_view.sql 4. other instances of model['name'] The following file also mention model['name'] and probably need to change as well ... ./dbt/include/global_project/macros/materializations/archive/archive.sql ./dbt/include/global_project/macros/materializations/seed/bigquery.sql ./dbt/include/global_project/macros/materializations/seed/seed.sql Added comentary to ... ./dbt/exceptions.py 5. further changes Revert model.get('alias') to model.get('name') ... print_test_result_line in ./dbt/ui/printer.py (since in this context schema is NOT being used) Change model.get('name') to model.get('alias') ... print_seed_result_line in ./dbt/ui/printer.py (since in this context schema is also being used) Change node.get('name') to node.get('alias') ... _node_context in ./dbt/node_runners.py (since in this context schema is also being used) call_get_missing_columns in ./dbt/node_runners.py (since in this context schema is also being used) call_already_exists in ./dbt/node_runners.py (since in this context schema is also being used) 6. linting import lines must be under 80 characters https://www.python.org/dev/peps/pep-0328/

abelsonlive · 2018-06-19T04:12:49Z

Wooooot!

cmcarthur

if you get rid of that get_custom_alias macro and just call node['alias'] directly, this is approved. i don't think overriding that macro in your project is going to work properly right now

cmcarthur · 2018-06-27T16:24:43Z

Makefile

+	rm -rf logs/
+	rm -rf target/
+	find . -type f -name '*.pyc' -delete
+	find . -type d -name '__pycache__' -depth -delete


this looks really useful

cmcarthur · 2018-06-27T16:29:20Z

dbt/include/global_project/macros/etc/get_custom_alias.sql

+    {% set alias = config.get('alias', default_alias) %}
+    {{ return(alias) }}
+
+{%- endmacro %}


does this actually work? looks like there are a lot of places (e.g. archive) where we use node['alias'] directly

cmcarthur · 2018-06-28T14:09:56Z

long-run we should replace node['alias'] with model.alias but whatever.

unrelated: your build is failing.

drewbanin · 2018-07-02T21:33:20Z

@cmcarthur ok, I ripped out get_custom_alias. I think the better approach is to jinja-compile the alias field, but we should likely do that for many dbt fields, so I think we can handle that in a different PR. I was trying to achieve a way to dynamically change the name of the resulting table for eg. date partitioning on BQ. I definitely agree that it was not the right approach here

- add test for duped model names in different schemas

drewbanin · 2018-07-03T14:32:37Z

these tests are failing with a very upsetting:

On expect_value_foo_alias_tablename__foo: 

select count(*)
from "test1530625544_aliases_026"."foo"
where tablename != 'foo'

dbt: DEBUG: Postgres error: failed to find conversion function from unknown to text

This works locally for me, so I think it must be a postgres version thing. Just going to keep adding typecasts until it works /shrug

cmcarthur · 2018-07-03T18:39:00Z

test/integration/026_aliases_test/macros/cast.sql

+
+{% macro bigquery__string_literal(s) %}
+    cast('{{ s }}' as string)
+{% endmacro %}


random: let's move this macro https://github.com/fishtown-analytics/dbt-utils/blob/master/macros/cross_db_utils/safe_cast.sql from dbt-utils to dbt proper.

yeah, agree, that would be super useful. Do you think that should happen in this PR or a different one?

cmcarthur · 2018-07-03T19:14:41Z

test/integration/026_aliases_test/test_aliases.py

+    def project_config(self):
+        return {
+            "macro-paths": ['test/integration/026_aliases_test/macros'],
+        }


i really like this as a test case for this feature

cmcarthur · 2018-07-03T19:15:37Z

@drewbanin this looks great. i left a couple random thoughts here but you can ship this.

Implement model aliasing Co-authored-by: Brian Abelson <abelsonlive@users.noreply.github.com> Co-authored-by: Jonathan Kaczynski <jon-rtr@users.noreply.github.com> automatic commit by git-black, original commits: 540631f

Brian Abelson and others added 19 commits February 6, 2018 15:39

tests working on newer postgres image

2b1378c

rename test

3726cd1

passing tests; breaking no other tests. looks good!

4fb70c1

cleanup and typos

76ced75

extra line

81e644a

make alias optional, fix parser tests

ecb7c86

add requested changes

69e6c9a

fix describe_node

75b91a8

cleanup

b11ebb0

Merge branch 'development' into feature/model-aliasing

9fd89d7

fix logging line

a2b5f73

Merge branch 'feature/model-aliasing' of https://github.com/kickstart…

64f420d

…er/dbt into feature/model-aliasing

try to get alias from config

5040c88

Temporarily bumping the library version.

874d1b9

Adding a make clean command.

06d459a

Temporarily bumping the library version.

144f453

Merge branch 'development' into feature/alias

20b84c4

bq/seed updates, add tests, catch dupe aliases

ff84f2a

drewbanin requested a review from cmcarthur June 18, 2018 22:50

drewbanin added this to the Betsy Ross (unreleased) milestone Jun 18, 2018

drewbanin added enhancement New feature or request code review labels Jun 18, 2018

tests passing

6c6f18a

rm alias from macros

f39b5a6

drewbanin force-pushed the feature/alias branch from cb94c4a to f39b5a6 Compare June 23, 2018 20:05

cmcarthur reviewed Jun 28, 2018

View reviewed changes

drewbanin assigned cmcarthur Jul 2, 2018

rip out generate_model_alias macro;

d8a2e57

drewbanin added 3 commits July 2, 2018 23:01

remove get_alias macro integration test

73a474e

- add test for duped model names in different schemas

fix tests for pg, add tests for snowflake/bq

f872ed0

hopefull fix for circle postgres errors

e5739b3

actually pass literals thru macro

f2ef85a

cmcarthur added estimate: 1 and removed estimate: 1 labels Jul 3, 2018

cmcarthur reviewed Jul 3, 2018

View reviewed changes

drewbanin assigned drewbanin and unassigned cmcarthur Jul 3, 2018

cmcarthur reviewed Jul 3, 2018

View reviewed changes

cmcarthur approved these changes Jul 3, 2018

View reviewed changes

drewbanin added 3 commits July 3, 2018 16:16

revert tox.init change

d274196

merge development

307a4e1

fix unit tests

ed9f884

drewbanin merged commit 540631f into development Jul 4, 2018

drewbanin deleted the feature/alias branch July 4, 2018 16:36

drewbanin mentioned this pull request Jul 4, 2018

alias models to change their names in the database #637

Closed

drewbanin mentioned this pull request Jul 19, 2018

Betsy ross changelog #869

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/alias #800

Feature/alias #800

drewbanin commented Jun 18, 2018 •

edited

Loading

abelsonlive commented Jun 19, 2018

cmcarthur left a comment

cmcarthur Jun 27, 2018

cmcarthur Jun 27, 2018

cmcarthur commented Jun 28, 2018

drewbanin commented Jul 2, 2018

drewbanin commented Jul 3, 2018

cmcarthur Jul 3, 2018

drewbanin Jul 3, 2018

cmcarthur Jul 3, 2018

cmcarthur commented Jul 3, 2018

Feature/alias #800

Feature/alias #800

Conversation

drewbanin commented Jun 18, 2018 • edited Loading

WHAT

WHY

HOW

abelsonlive commented Jun 19, 2018

cmcarthur left a comment

Choose a reason for hiding this comment

cmcarthur Jun 27, 2018

Choose a reason for hiding this comment

cmcarthur Jun 27, 2018

Choose a reason for hiding this comment

cmcarthur commented Jun 28, 2018

drewbanin commented Jul 2, 2018

drewbanin commented Jul 3, 2018

cmcarthur Jul 3, 2018

Choose a reason for hiding this comment

drewbanin Jul 3, 2018

Choose a reason for hiding this comment

cmcarthur Jul 3, 2018

Choose a reason for hiding this comment

cmcarthur commented Jul 3, 2018

drewbanin commented Jun 18, 2018 •

edited

Loading