[CT-2889] Macros executed via run-operation cannot ref private models

[nicholasyager]

Is this a new bug in dbt-core?

• I believe this is a new bug in dbt-core
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

Suppose you have a model (order.sql) setup in a group (example) with access set to private. If you execute a macro that refs this private model, dbt will abort with an AttributeError: 'Macro' object has no attribute 'group' exception.

19:22:02  Traceback (most recent call last):
  File "/redacted/dbt_project/venv/lib/python3.9/site-packages/dbt/adapters/snowflake/connections.py", line 255, in exception_handler
    yield
  File "/redacted/dbt_project/venv/lib/python3.9/site-packages/dbt/adapters/base/impl.py", line 1043, in execute_macro
    result = macro_function(**kwargs)
  File "/redacted/dbt_project/venv/lib/python3.9/site-packages/dbt/clients/jinja.py", line 330, in __call__
    return self.call_macro(*args, **kwargs)
  File "/redacted/dbt_project/venv/lib/python3.9/site-packages/dbt/clients/jinja.py", line 257, in call_macro
    return macro(*args, **kwargs)
  File "/redacted/dbt_project/venv/lib/python3.9/site-packages/jinja2/runtime.py", line 763, in __call__
    return self._invoke(arguments, autoescape)
  File "/redacted/dbt_project/venv/lib/python3.9/site-packages/jinja2/runtime.py", line 777, in _invoke
    rv = self._func(*arguments)
  File "<template>", line 35, in macro
  File "/redacted/dbt_project/venv/lib/python3.9/site-packages/jinja2/sandbox.py", line 393, in call
    return __context.call(__obj, *args, **kwargs)
  File "/redacted/dbt_project/venv/lib/python3.9/site-packages/jinja2/runtime.py", line 298, in call
    return __obj(*args, **kwargs)
  File "/redacted/dbt_project/venv/lib/python3.9/site-packages/dbt/context/providers.py", line 259, in __call__
    return self.resolve(name, package, version)
  File "/redacted/dbt_project/venv/lib/python3.9/site-packages/dbt/context/providers.py", line 508, in resolve
    if not self.model.group or self.model.group != target_model.group:
AttributeError: 'Macro' object has no attribute 'group'

This issue seems to originate in the resolve method in RuntimeRefResolver. Specifically, this chunk of logic:

        elif (
            target_model.resource_type == NodeType.Model
            and target_model.access == AccessType.Private
            # don't raise this reference error for ad hoc 'preview' queries
            and self.model.resource_type != NodeType.SqlOperation
            and self.model.resource_type != NodeType.RPCCall  # TODO: rm
        ):
            if not self.model.group or self.model.group != target_model.group:
                raise DbtReferenceError(
                    unique_id=self.model.unique_id,
                    ref_unique_id=target_model.unique_id,
                    group=cast_to_str(target_model.group),
                )

When running a macro via run-operation, target_model is not a ParsedNode, but rather a Macro, which do not have a concept of a group. Hence, the AttributeError.

Expected Behavior

Given that macros are not part of the group and access abstraction, I would expect macros executed within a run-operation context to be able to ref any model in a dbt project without verifying group membership.

Steps To Reproduce

1. Create a model, and configure it to be in a group.
2. Create a macro that ref's the grouped model.
3. Attempt to execute the macro via run-operation.

Relevant log output

See above in current behavior

Environment

- OS: MacOS 13.5
- Python: 3.9.6
- dbt: 1.5.4

Which database adapter are you using with dbt?

snowflake

Additional Context

No response

[jtcohen6]

@nicholasyager Thanks for opening! :)

It does feel a bit weird that macros can reference private models, if called from within other models in the same group, but not if the same macro is called via run-operation.

I'm inclined to say that we should support this, same as how we added support for compilation/preview of --inline queries that reference private models:

• Fix: dbt show --inline with private models #7838

I'll queue this up for the next milestone of multi-project work. If the fix is quite simple (and I think it is, right where you're pointing to!) we could backport for inclusion in a v1.6.x patch.

[graciegoheen]

Notes from refinement:

• macros need to all be loaded first - the groups won't have been loaded with the macros are loaded for parsing
• this is during execution not parse time

[b-per]

Hi team. Any update on when this might be updated/fixed?

The label says backport 1.7 but 1.8 is out now.

[jtcohen6]

@b-per I've just created a new epic for multi-project work over the next few months, and I'm including this on there. I see this lower priority than other bugs / QoL improvements that have been coming up more often, but definitely open to hearing that I should be prioritizing it more highly.

At this point, I'd be inclined to "roll forward" with this fix, and include it in the subsequent minor version.

I don't think this would be a very big lift, so I'd say it's also fair game for community contribution (by you or someone else) :)

[b-per]

Yes, my comment about seeing the backport 1.7 label was more whether it was still an issue considered for the future.
I agree that it doesn't look critical enough to backport it to previous versions.

Should we add the community label then?

And I'll check the new epic! 👀