[CT-3096] Refactor selection + dbt list output, for better consistency across node types

[jtcohen6]

Based on conversation in #8546 / #8589 (comment)

What's the problem?

It's a bit odd that, for the purposes of listing & selection, we so strongly distinguish between:

1. "Logical" node types (models/seeds/snapshots/tests/analyses), which dbt actually compiles/builds
2. "Pointer" node types (sources/exposures/semantic_models/metrics), for which dbt just resolves & stores config

(these are my recently invented terms, not officially established anywhere, and by no means set in stone)

Specific behaviors

• The default output of dbt list includes {node_type}:{node_package}.{node_name} for the "pointer" node types, but then {node_fqn} for the "logical" node types
• It's possible to select a "pointer" node by its type (e.g. source:*, metric:*), but this is not possible for "logical" nodes
• The fqn:* selection method only includes "logical" node types

As an example:

$ dbt -q ls --resource-type model semantic_model
jaffle_shop.marts.customers
jaffle_shop.marts.metricflow_time_spine
jaffle_shop.marts.order_items
jaffle_shop.marts.orders
jaffle_shop.staging.stg_customers
jaffle_shop.staging.stg_locations
jaffle_shop.staging.stg_order_items
jaffle_shop.staging.stg_orders
jaffle_shop.staging.stg_products
jaffle_shop.staging.stg_supplies
semantic_model:jaffle_shop.customers
semantic_model:jaffle_shop.locations
semantic_model:jaffle_shop.order_item
semantic_model:jaffle_shop.orders
semantic_model:jaffle_shop.stg_products

The models are displayed by FQN. The semantic_models are displayed by semantic_model:{package}.{name}. The idea is, you could copy-paste / pipe those string outputs and use as selectors. But wouldn't it be better if it were more consistent, and you could use them as selectors?

Why is it like this?

This feels like an organic outgrowth of

• "Logical" nodes are more closely tied to files (a model is 1:1 with a file), and their FQNs are therefore more meaningfully linked to their relative file path
• "Pointer" nodes are yaml config, many to a file. Their FQN matters less, compared with their node type, package/project namespace, and unique identifier. (In the case of sources, the unique identifier is source_name + table name).

How could this be better?

Some potential acceptance criteria:

• All node types already have an FQN. So the fqn:* selector should actually include all nodes. Then we wouldn't need this wonky default selector.
• It should be possible to select all node types using the {node_type}: syntax - either {node_type}:{node_fqn_part}, or {node_type}:* to select all nodes of that type. (Today, the workaround for "select all snapshots" is something like --select config.materialized:snapshot, which is janky and not really documented.)
• We should support a resource_type:{type} selection method, similar to the --resource-type flag, but as an actual method that can be combined with other selection methods in unions/intersections.
• The default output of dbt list should change to {node_type}:{node_fqn} for all node types — or the default --output should be unique_id instead of selector.
• dbt ls should include all resource types by default. The fact that it excludes analyses by default (and only analyses) is pretty confusing.

Out of scope

Unrelated pet peeves about dbt list:

• The output behavior that conditionally calls print or fire_event depending on the --log-format (!)
• [CT-2767] [Feature] Use --quiet by default for dbt list --output json #7994

[ChenyuLInx]

• all behavior change will be gated by one flag selection_include_analysis(tbd).
• points:
  • 3
  • 4?
  • don't do it
  • 2
  • 2