DbtTaskGroup - Kubernetes Execution with RenderConfig

[harshag7dm]

When I use Kubernetes Execution in DbtTaskGroup and try to filter out the Models using tag, it doesn't work. There is no Kubernetes pod being run.
If I remove the RenderConfig, then all the models are run?
Is this a bug or am I missing something?

[dosubot]

It sounds like you're encountering an issue where using RenderConfig with DbtTaskGroup in a Kubernetes environment doesn't trigger the execution of pods when filtering models by tag. This could be due to a configuration issue or a more complex problem within the interaction of DbtTaskGroup, RenderConfig, and Kubernetes Execution.

To address this, ensure that your DbtTaskGroup is correctly configured to include the tag filter in its arguments for dbt tasks. The RenderConfig should be set up properly with the necessary class arguments and Kubernetes-specific configurations. For detailed guidance, refer to the documentation on configuring RenderConfig and Kubernetes execution in the Cosmos repository.

Additionally, check for any recent changes or updates in the Cosmos repository that might affect this functionality. Specifically, look into the changelog for updates related to Docker and Kubernetes operators, as these could impact how DbtTaskGroup operates with Kubernetes Execution.

If the issue persists, examining the logs or error messages for any clues on why the Kubernetes pod is not being initiated could provide further insights. The DbtTestKubernetesOperator class, for example, includes methods designed to handle warnings and extract log issues, which might reveal underlying problems.

If you can provide more details about your configuration and any specific error messages you're encountering, it would help in diagnosing and resolving the issue more effectively.

Details

Sources

• All models are rendered in the sql compiled (although only one model needs to be rendered due to the select tag in DbtTaskGroup)
• cosmos/operators/kubernetes.py
• docs/configuration/render-config.rst
• dev/dags/basic_cosmos_task_group.py
• docs/getting_started/kubernetes.rst
• dev/dags/user_defined_profile.py
• CHANGELOG.rst
• cosmos/converter.py

About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

[tatiana]

@harshag7dm please, could you confirm which version of Cosmos you've tried, with which DAG?

[tatiana]

To try to reproduce this:

1. Run https://astronomer.github.io/astronomer-cosmos/getting_started/kubernetes.html#kubernetes
2. Modify https://github.com/astronomer/cosmos-example/blob/main/dags/jaffle_shop_kubernetes.py to have RenderConfig with a selector using tag (e.g. https://github.com/astronomer/cosmos-demo/blob/main/dags/filtering/customers_tag.py)
3. Confirm we can reproduce the issue
4. Fix this, if we can

[pankajastro]

This is a legitimate issue. I can replicate it using the steps outlined above. I’ll create a PR to address it.

[pankajastro]

I investigated further and found that when the manifest is unavailable and the execution mode isn't set to ExecutionMode.LOCAL, we're using the custom parser to load the dbt project. However, this custom parser currently does not support selectors.

astronomer-cosmos/cosmos/dbt/graph.py

Lines 323 to 329 in 2994dba

	if execution_mode == ExecutionMode.LOCAL and self.profile_config:
	try:
	self.load_via_dbt_ls()
	except FileNotFoundError:
	self.load_via_custom_parser()
	else:
	self.load_via_custom_parser()

The workaround I tried

• Use load_via_dbt_ls for other execution modes as well, although I'm uncertain about any potential side effects.
• Extract the tag in the custom parser method and assign it to the node. This is currently set to empty.

  astronomer-cosmos/cosmos/dbt/graph.py

  Line 557 in 2994dba

  tags=[],

In both cases, the DAG rendering is like below for dag https://github.com/astronomer/cosmos-example/blob/main/dags/jaffle_shop_kubernetes.py with addition param render_config

render_config=RenderConfig(
            select=["tag:customers"],
        ),