[Regression] node.external.partitions returned as string instead of dict

[thomasgjerdekog]

Is this a regression in a recent version of dbt-core?

• I believe this is a regression in dbt-core functionality
• I have searched the existing issues, and I could not find an existing issue for this regression

Current Behavior

In dbt 1.8.9 it seems like the behavior of node.external.partitions has changed. It used to return a list of dicts, but now it seems to return a list of json strings.
This seems to be causing problems for dbt-external-tables when trying to create external tables with partitions defined.

I'm using this macro to compare the behaviors:

{% macro get_partitions() -%}
    {% set source_nodes = graph.sources.values() if graph.sources else [] %}
    {% for node in source_nodes %}
        {% if node.external %}
            {% if node.external.partitions %}
                {{print(node.external.partitions)}}
            {% endif %}
        {% endif %}
    {% endfor %}
{%- endmacro %}

dbt 1.8.8:

dbt run-operation get_partitions
11:23:26  Running with dbt=1.8.8
11:23:27  Registered adapter: snowflake=1.8.4
11:23:31  Found 1412 models, 2 snapshots, 6 seeds, 5 operations, 129 data tests, 619 sources, 2 exposures, 730 macros
[{'name': 'dl_partition', 'data_type': 'string', 'expression': "<redacted>"}]

dbt 1.8.9:

dbt run-operation get_partitions
11:21:38  Running with dbt=1.8.9
11:21:38  Registered adapter: snowflake=1.8.4
11:21:43  Found 1396 models, 2 snapshots, 6 seeds, 5 operations, 129 data tests, 611 sources, 2 exposures, 735 macros
['{\'name\': \'dl_partition\', \'data_type\': \'string\', \'expression\': "<redacted>"}']

Expected/Previous Behavior

I expect node.external.partitions to be a list of dicts so that the values can be accessed by looping the list and addressing with i.e. partition.name

Steps To Reproduce

1. Install dbt 1.8.9
2. Create a yaml source file called testtable.yml with external and partition set. I.e.:

version: 2
sources:
- name: TEST
  schema: STAGE
  tables:
  - name: TABLE
    external:
      partitions:
      - name: dl_partition
        data_type: string
        expression: split_part(METADATA$FILENAME, '/', 2)

3. Create a test macro called get_partitions.sql:

{% macro get_partitions() -%}
    {% set source_nodes = graph.sources.values() if graph.sources else [] %}
    {% for node in source_nodes %}
        {% if node.external %}
            {% if node.external.partitions %}
                {{print(node.external.partitions)}}
            {% endif %}
        {% endif %}
    {% endfor %}
{%- endmacro %}

4. Run the macro:
   dbt run-operation get_partitions

Relevant log output

No response

Environment

- OS: Windows 11 23H2/Ubuntu 22.04.5 LTS
- Python: Python 3.12.4
- dbt (working version): 1.8.8
- dbt (regression version): 1.8.9

Which database adapter are you using with dbt?

snowflake

Additional Context

No response

[dbeatty10]

Thanks for reporting this @thomasgjerdekog !

I'm not seeing an obvious code difference between dbt-core 1.8.8 and 1.8.9 that explains what you're seeing, so I'm guessing it is via some dependency.

Could you share all the dbt versions for your dependencies in each of those different environments?

Assuming that you're using pip to install dbt-core, you can run something like this to get those versions:

pip list | grep '^dbt-'

Output might look something like this:

dbt-adapters               1.4.0
dbt-common                 1.7.0
dbt-core                   1.8.8
dbt-postgres               1.8.2
dbt-semantic-interfaces    0.5.1

[thomasgjerdekog]

Hi @dbeatty10
Sure!
Old (working):

pip freeze | grep "^dbt-"
dbt-adapters==1.7.0
dbt-common==1.10.0
dbt-core==1.8.8
dbt-extractor==0.5.1
dbt-osmosis==0.12.8
dbt-semantic-interfaces==0.5.1
dbt-snowflake==1.8.4

New (not working):

pip freeze | grep "^dbt-"
dbt-adapters==1.10.2
dbt-common==1.13.0
dbt-core==1.8.9
dbt-extractor==0.5.1
dbt-osmosis==0.12.8
dbt-semantic-interfaces==0.5.1
dbt-snowflake==1.8.4

I tried experimenting a bit with downgrading the dbt-adapters and dbt-common packages prior to submitting this issue, but I wasn't able to resolve the install conflicts since dbt-core 1.8.9 depends on protobuf >= 5.0 while versions of dbt-adapters prior to 1.10.0 depend on protobuf < 5.0