Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add default source nodes rendering #1107

Merged
merged 34 commits into from
Aug 14, 2024
Merged
Changes from 1 commit
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
9d4c2cf
Add default source nodes rendering
arojasb3 Jul 19, 2024
d925d40
Update ResourceType reference with constant
arojasb3 Jul 19, 2024
070b4d5
Add MixIn Operator + freshness template rendering
arojasb3 Jul 21, 2024
021a834
Add render_source_nodes to settings.py
arojasb3 Jul 22, 2024
56a77ed
update jaffle shop source with tests
arojasb3 Jul 22, 2024
6c208c1
Add source node rendering param to render config
arojasb3 Jul 25, 2024
8ad040c
Update existing tests with new source rendering
arojasb3 Jul 26, 2024
395ecc6
Update project hash in tests
arojasb3 Jul 26, 2024
e9e3793
Merge branch 'main' into add-source-nodes
arojasb3 Jul 29, 2024
4fcb9a2
update extra context after merge
arojasb3 Jul 29, 2024
760d3f8
update test hash after merge
arojasb3 Jul 29, 2024
9c42e0d
Merge branch 'main' into add-source-nodes
arojasb3 Jul 30, 2024
91ae0f4
adding quotes in ls command + source test
arojasb3 Aug 2, 2024
734666a
Merge branch 'main' into add-source-nodes
arojasb3 Aug 2, 2024
d8983f8
Update freshness param in source tests
arojasb3 Aug 2, 2024
203c534
update dbt ls command for 1.5.4
arojasb3 Aug 6, 2024
d788c04
Merge branch 'main' into add-source-nodes
arojasb3 Aug 6, 2024
6465a6f
Merge branch 'main' into add-source-nodes
arojasb3 Aug 9, 2024
093b6e5
🎨 [pre-commit.ci] Auto format from pre-commit.com hooks
pre-commit-ci[bot] Aug 9, 2024
41546f0
Change default source rendering to None
arojasb3 Aug 9, 2024
5a062e2
Merge branch 'add-source-nodes' of github.com:arojasb3/astronomer-cos…
arojasb3 Aug 9, 2024
4866f61
Update project hash
arojasb3 Aug 9, 2024
ce6da81
Add suggestions + workflow env variables
arojasb3 Aug 13, 2024
dab5229
Run ci on add-source-nodes branch
pankajastro Aug 13, 2024
d6ad582
Run ci on add-source-nodes branch
pankajastro Aug 13, 2024
1dd4b82
intentionally fail the sql job
pankajastro Aug 13, 2024
1c6ae65
Revert env changes
pankajastro Aug 13, 2024
59574fc
🎨 [pre-commit.ci] Auto format from pre-commit.com hooks
pre-commit-ci[bot] Aug 13, 2024
ff65e5b
Merge branch 'main' into add-source-nodes
pankajastro Aug 13, 2024
ad0e0c2
Add tests
pankajastro Aug 13, 2024
b0b4020
Add test
pankajastro Aug 13, 2024
7fd92ab
🎨 [pre-commit.ci] Auto format from pre-commit.com hooks
pre-commit-ci[bot] Aug 13, 2024
96492c8
Revert source render param
pankajastro Aug 13, 2024
bacd19d
Add no cover
pankajastro Aug 13, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add suggestions + workflow env variables
  • Loading branch information
arojasb3 committed Aug 13, 2024
commit ce6da818dc46b25570cf4ec6d04998995caa2e5d
3 changes: 3 additions & 0 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
@@ -163,6 +163,7 @@ jobs:
POSTGRES_DB: postgres
POSTGRES_SCHEMA: public
POSTGRES_PORT: 5432
SOURCE_RENDERING_BEHAVIOR: all

- name: Upload coverage to Github
uses: actions/upload-artifact@v2
@@ -234,6 +235,7 @@ jobs:
POSTGRES_DB: postgres
POSTGRES_SCHEMA: public
POSTGRES_PORT: 5432
SOURCE_RENDERING_BEHAVIOR: all

- name: Upload coverage to Github
uses: actions/upload-artifact@v2
@@ -377,6 +379,7 @@ jobs:
POSTGRES_DB: postgres
POSTGRES_SCHEMA: public
POSTGRES_PORT: 5432
SOURCE_RENDERING_BEHAVIOR: all

- name: Upload coverage to Github
uses: actions/upload-artifact@v2
2 changes: 1 addition & 1 deletion cosmos/airflow/graph.py
Original file line number Diff line number Diff line change
@@ -132,7 +132,7 @@
execution_mode: ExecutionMode,
args: dict[str, Any],
use_task_group: bool = False,
source_rendering_behavior: SourceRenderingBehavior = SourceRenderingBehavior.ALL,
source_rendering_behavior: SourceRenderingBehavior = SourceRenderingBehavior.NONE,
) -> TaskMetadata | None:
"""
Create the metadata that will be used to instantiate the Airflow Task used to run the Dbt node.
@@ -167,14 +167,14 @@
and node.has_test is False
):
return None
task_id = f"{node.name}_source"
args["select"] = f"source:{node.resource_name}"
args.pop("models")
if use_task_group is True:
task_id = node.resource_type.value
if node.has_freshness is False and source_rendering_behavior == SourceRenderingBehavior.ALL:

Check warning on line 175 in cosmos/airflow/graph.py

Codecov / codecov/patch

cosmos/airflow/graph.py#L170-L175

Added lines #L170 - L175 were not covered by tests
# render sources without freshness as empty operators
return TaskMetadata(id=task_id, operator_class="airflow.operators.empty.EmptyOperator")

Check warning on line 177 in cosmos/airflow/graph.py

Codecov / codecov/patch

cosmos/airflow/graph.py#L177

Added line #L177 was not covered by tests
else:
task_id = f"{node.name}_{node.resource_type.value}"
if use_task_group is True:
2 changes: 1 addition & 1 deletion cosmos/config.py
Original file line number Diff line number Diff line change
@@ -60,7 +60,7 @@ class RenderConfig:
:param dbt_project_path: Configures the DBT project location accessible on the airflow controller for DAG rendering. Mutually Exclusive with ProjectConfig.dbt_project_path. Required when using ``load_method=LoadMode.DBT_LS`` or ``load_method=LoadMode.CUSTOM``.
:param dbt_ls_path: Configures the location of an output of ``dbt ls``. Required when using ``load_method=LoadMode.DBT_LS_FILE``.
:param enable_mock_profile: Allows to enable/disable mocking profile. Enabled by default. Mock profiles are useful for parsing Cosmos DAGs in the CI, but should be disabled to benefit from partial parsing (since Cosmos 1.4).
:param source_rendering_behavior: Determines how source nodes are rendered when using cosmos default source node rendering (ALL, NONE, WITH_TESTS_OR_FRESHNESS). Defaults to "ALL" (since Cosmos 1.6).
:param source_rendering_behavior: Determines how source nodes are rendered when using cosmos default source node rendering (ALL, NONE, WITH_TESTS_OR_FRESHNESS). Defaults to "NONE" (since Cosmos 1.6).
"""

emit_datasets: bool = True
2 changes: 1 addition & 1 deletion docs/configuration/source-nodes-rendering.rst
Original file line number Diff line number Diff line change
@@ -4,7 +4,7 @@ Source Nodes Rendering
================

.. note::
This feature is only available for dbt-core >= 1.5.
This feature is only available for dbt-core >= 1.5 and cosmos >= 1.6.0.

By default, Cosmos does not render dbt sources automatically. Instead, you need to configure the rendering of sources explicitly.
You can control this behavior using the ``source_rendering_behavior`` field in the ``RenderConfig`` object. This is how it works:
Loading