[Bug] Crash when dbt runs `show grants` statement due to column name capitalization mismatch #1086

lamalex · 2024-08-03T15:49:51Z

Is this a new bug in dbt-spark?

I believe this is a new bug in dbt-spark
I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

My DBT build is failing (in certain environments, have not been able to pinpoint what the difference is) due to column name capitalization mismatch from a show grants statement.

In my dbt-spark library i've added a print on the grants table before columns are accessed by name, and I am seeing lower cased column names instead of upper-case (which does not match when I run this statement in databricks sql console!)

| column     | data_type |
| ---------- | --------- |
| principal  | Text      |
| actiontype | Text      |
| objecttype | Text      |
| objectkey  | Text      |

if I change the column names in the adapter my models build successfully.
This is occurring on my workstation, as well as in our Dagster instance, but is not occurring in a Databricks workflow job (which are all using dbt-core 1.8.4, and dbt-databricks 1.8.3(or 4), and dbt-spark 1.8.0).

Expected Behavior

not crash

Steps To Reproduce

run dbt build --profile databricks --select tag:elementary --log-level debug

Relevant log output

15:35:00  Using databricks connection "model.elementary.metadata"
15:35:00  On model.elementary.metadata: /* {"app": "dbt", "dbt_version": "1.8.4", "dbt_databricks_version": "1.8.4", "databricks_sql_connector_version": "3.1.2", "profile_name": "databricks", "target_name": "dev", "node_id": "model.elementary.metadata"} */


    show grants on `launi-databricks_catalog_nonprod`.`zzz_launi_alauni`.`metadata`


15:35:00  Databricks adapter: Cursor(session-id=01ef51ad-d40d-1dc7-be88-71c780641be1, command-id=Unknown) - Created cursor
15:35:01  SQL status: OK in 0.4000000059604645 seconds
15:35:01  Databricks adapter: Cursor(session-id=01ef51ad-d40d-1dc7-be88-71c780641be1, command-id=01ef51ad-f259-1668-889b-9ea8e4fa88ba) - Closing cursor
| column     | data_type |
| ---------- | --------- |
| principal  | Text      |
| actiontype | Text      |
| objecttype | Text      |
| objectkey  | Text      |

15:35:01  Databricks adapter: DatabricksDBTConnection(id=5338871184, session-id=01ef51ad-d40d-1dc7-be88-71c780641be1, name=model.elementary.metadata, idle-time=4.0531158447265625e-06s, acquire-count=0, language=sql, thread-identifier=(60914, 12901707776), compute-name=) - Released connection
15:35:01  Unhandled error while executing target/run/elementary/models/edr/system/metadata.sql
'Principal'
15:35:01  Traceback (most recent call last):
  File "/Users/alauni/Code/launi/launi-dbt/.venv/lib/python3.12/site-packages/dbt/task/base.py", line 368, in safe_run
    result = self.compile_and_execute(manifest, ctx)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alauni/Code/launi/launi-dbt/.venv/lib/python3.12/site-packages/dbt/task/base.py", line 314, in compile_and_execute
    result = self.run(ctx.node, manifest)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alauni/Code/launi/launi-dbt/.venv/lib/python3.12/site-packages/dbt/task/base.py", line 415, in run
    return self.execute(compiled_node, manifest)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alauni/Code/launi/launi-dbt/.venv/lib/python3.12/site-packages/dbt/task/run.py", line 298, in execute
    result = MacroGenerator(
             ^^^^^^^^^^^^^^^
  File "/Users/alauni/Code/launi/launi-dbt/.venv/lib/python3.12/site-packages/dbt/clients/jinja.py", line 84, in __call__
    return self.call_macro(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alauni/Code/launi/launi-dbt/.venv/lib/python3.12/site-packages/dbt_common/clients/jinja.py", line 298, in call_macro
    return macro(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alauni/Code/launi/launi-dbt/.venv/lib/python3.12/site-packages/jinja2/runtime.py", line 768, in __call__
    return self._invoke(arguments, autoescape)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alauni/Code/launi/launi-dbt/.venv/lib/python3.12/site-packages/jinja2/runtime.py", line 782, in _invoke
    rv = self._func(*arguments)
         ^^^^^^^^^^^^^^^^^^^^^^
  File "<template>", line 75, in macro
  File "/Users/alauni/Code/launi/launi-dbt/.venv/lib/python3.12/site-packages/jinja2/sandbox.py", line 394, in call
    return __context.call(__obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alauni/Code/launi/launi-dbt/.venv/lib/python3.12/site-packages/jinja2/runtime.py", line 303, in call
    return __obj(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alauni/Code/launi/launi-dbt/.venv/lib/python3.12/site-packages/dbt/clients/jinja.py", line 84, in __call__
    return self.call_macro(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alauni/Code/launi/launi-dbt/.venv/lib/python3.12/site-packages/dbt_common/clients/jinja.py", line 298, in call_macro
    return macro(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alauni/Code/launi/launi-dbt/.venv/lib/python3.12/site-packages/jinja2/runtime.py", line 768, in __call__
    return self._invoke(arguments, autoescape)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alauni/Code/launi/launi-dbt/.venv/lib/python3.12/site-packages/jinja2/runtime.py", line 782, in _invoke
    rv = self._func(*arguments)
         ^^^^^^^^^^^^^^^^^^^^^^
  File "<template>", line 25, in macro
  File "/Users/alauni/Code/launi/launi-dbt/.venv/lib/python3.12/site-packages/jinja2/sandbox.py", line 394, in call
    return __context.call(__obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alauni/Code/launi/launi-dbt/.venv/lib/python3.12/site-packages/jinja2/runtime.py", line 303, in call
    return __obj(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alauni/Code/launi/launi-dbt/.venv/lib/python3.12/site-packages/dbt/clients/jinja.py", line 84, in __call__
    return self.call_macro(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alauni/Code/launi/launi-dbt/.venv/lib/python3.12/site-packages/dbt_common/clients/jinja.py", line 298, in call_macro
    return macro(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alauni/Code/launi/launi-dbt/.venv/lib/python3.12/site-packages/jinja2/runtime.py", line 768, in __call__
    return self._invoke(arguments, autoescape)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alauni/Code/launi/launi-dbt/.venv/lib/python3.12/site-packages/jinja2/runtime.py", line 782, in _invoke
    rv = self._func(*arguments)
         ^^^^^^^^^^^^^^^^^^^^^^
  File "<template>", line 48, in macro
  File "/Users/alauni/Code/launi/launi-dbt/.venv/lib/python3.12/site-packages/jinja2/sandbox.py", line 394, in call
    return __context.call(__obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alauni/Code/launi/launi-dbt/.venv/lib/python3.12/site-packages/jinja2/runtime.py", line 303, in call
    return __obj(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alauni/Code/launi/launi-dbt/.venv/lib/python3.12/site-packages/dbt/adapters/databricks/utils.py", line 78, in wrapper
    return func(*new_args, **new_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alauni/Code/launi/launi-dbt/.venv/lib/python3.12/site-packages/dbt/adapters/spark/impl.py", line 493, in standardize_grants_dict
    grantee = row["Principal"]
              ~~~^^^^^^^^^^^^^
  File "/Users/alauni/Code/launi/launi-dbt/.venv/lib/python3.12/site-packages/agate/mapped_sequence.py", line 88, in __getitem__
    return self.dict()[key]
           ~~~~~~~~~~~^^^^^
KeyError: 'Principal'



### Environment

```markdown
- OS: MacOS 14.6
- Python: 3.12.3
- dbt-core: 1.8.4
- dbt-spark: 1.8.0

Additional Context

No response

The text was updated successfully, but these errors were encountered:

…in grants table response Match case to column names returned rather than hardcoding. Fixes: dbt-labs#1086

amychen1776 · 2024-08-28T18:02:10Z

@lamalex Can I confirm with you that you're using dbt-databricks and dbt-spark for your Databricks workflows? Is there a reason why you aren't just using dbt-databricks?

lamalex · 2024-08-28T23:06:59Z

@amychen1776 i am using dbt-databricks, but dbt-spark is a dependency of dbt-databricks

amychen1776 · 2024-08-29T12:24:31Z

@lamalex You're very correct here. However you're currently using dbt-databricks and the issue you're raising is in comparison to the Databricks interface. It would make the most sense to start with the Databricks team - since I can't transfer issues to repositories outside of my org, would you please open up an issue on the Databricks repo for them to take a look?

lamalex · 2024-08-31T02:32:39Z

@amychen1776 are you saying that although the code for this bug exists in dbt-spark, it's the dbt-databricks team who should be responsible for the fix?

amychen1776 · 2024-09-05T19:49:36Z

Yes - they can also open up issues and prs for this adapter as needed but for initial triage, it should fall to the dbt-databricks team first. Thank you for opening up the issue!

lamalex · 2024-09-05T23:31:57Z

@amychen1776 Is there something wrong with the PR i opened? I mean I’m not asking anyone else to fix it!

amychen1776 · 2024-09-18T13:53:30Z

@lamalex not at all! It's more I want to make sure Databricks is aware of this

lamalex added bug Something isn't working triage labels Aug 3, 2024

lamalex added a commit to lamalex/dbt-spark that referenced this issue Aug 3, 2024

Update standardize_grants_dict to handle potential case mismatches …

c1bae97

…in grants table response Match case to column names returned rather than hardcoding. Fixes: dbt-labs#1086

lamalex mentioned this issue Aug 3, 2024

Update standardize_grants_dict to handle potential case mismatches in grants table response #1087

Open

4 tasks

lamalex added a commit to lamalex/dbt-spark that referenced this issue Aug 3, 2024

Update standardize_grants_dict to handle potential case mismatches …

281fd60

…in grants table response Match case to column names returned rather than hardcoding. Fixes: dbt-labs#1086

amychen1776 added awaiting_response and removed triage labels Aug 28, 2024

github-actions bot added triage and removed awaiting_response labels Aug 28, 2024

amychen1776 added awaiting_response and removed triage labels Aug 29, 2024

github-actions bot added triage and removed awaiting_response labels Aug 31, 2024

lamalex mentioned this issue Aug 31, 2024

[Bug] Crash when dbt runs show grants statement due to column name capitalization mismatch databricks/dbt-databricks#782

Open

2 tasks

amychen1776 closed this as not planned Won't fix, can't repro, duplicate, stale Sep 5, 2024

amychen1776 removed the triage label Sep 5, 2024

amychen1776 added the grants Issues related to dbt's grants functionality label Sep 18, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug] Crash when dbt runs `show grants` statement due to column name capitalization mismatch #1086

[Bug] Crash when dbt runs `show grants` statement due to column name capitalization mismatch #1086

lamalex commented Aug 3, 2024

amychen1776 commented Aug 28, 2024

lamalex commented Aug 28, 2024

amychen1776 commented Aug 29, 2024

lamalex commented Aug 31, 2024

amychen1776 commented Sep 5, 2024

lamalex commented Sep 5, 2024

amychen1776 commented Sep 18, 2024 •

edited

Loading

[Bug] Crash when dbt runs show grants statement due to column name capitalization mismatch #1086

[Bug] Crash when dbt runs show grants statement due to column name capitalization mismatch #1086

Comments

lamalex commented Aug 3, 2024

Is this a new bug in dbt-spark?

Current Behavior

Expected Behavior

Steps To Reproduce

Relevant log output

Additional Context

amychen1776 commented Aug 28, 2024

lamalex commented Aug 28, 2024

amychen1776 commented Aug 29, 2024

lamalex commented Aug 31, 2024

amychen1776 commented Sep 5, 2024

lamalex commented Sep 5, 2024

amychen1776 commented Sep 18, 2024 • edited Loading

[Bug] Crash when dbt runs `show grants` statement due to column name capitalization mismatch #1086

[Bug] Crash when dbt runs `show grants` statement due to column name capitalization mismatch #1086

amychen1776 commented Sep 18, 2024 •

edited

Loading