source freshness precomputes metadata-based freshness in batch, if possible

[MichelleArk]

resolves #8705

Problem

Executing a query per source during source freshness was slow, even for faster metadata-based queries

Solution

• Now that a batch version of freshness calculation is available on the base adapter (Introduce TableLastModifiedMetadataBatch and implement BaseAdapter.calculate_freshness_from_metadata_batch dbt-adapters#127), leverage it in the source freshness task by:
  • On task startup -- pre-compute freshness responses for metadata sources, and save those on the runner as a cache
  • In the runner execution, lookup the metadata source by relation in the cache. If it exists, skip the per-node calculation.
  • This approach preserves node-level logging for all nodes, including those that have had source freshness precomputed:

🎩

❯ dbt source freshness
22:18:07  Running with dbt=1.8.0-b1
22:18:07  target not specified in profile 'snowflake', using 'default'
22:18:08  Registered adapter: snowflake=1.8.0-a1
22:18:08  Found 13 models, 6 seeds, 20 data tests, 6 sources, 13 metrics, 688 macros, 6 semantic models, 3 unit tests
22:18:08
22:18:09  Generating metadata freshness for 6 sources
22:18:12
22:18:12  Concurrency: 4 threads (target='default')
22:18:12
22:18:12  1 of 6 START freshness of ecom.raw_customers ................................... [RUN]
22:18:12  2 of 6 START freshness of ecom.raw_items ....................................... [RUN]
22:18:12  3 of 6 START freshness of ecom.raw_orders ...................................... [RUN]
22:18:12  4 of 6 START freshness of ecom.raw_products .................................... [RUN]
22:18:12  1 of 6 PASS freshness of ecom.raw_customers .................................... [PASS in 0.01s]
22:18:12  2 of 6 PASS freshness of ecom.raw_items ........................................ [PASS in 0.01s]
22:18:12  4 of 6 PASS freshness of ecom.raw_products ..................................... [PASS in 0.01s]
22:18:12  5 of 6 START freshness of ecom.raw_stores ...................................... [RUN]
22:18:12  6 of 6 START freshness of ecom.raw_supplies .................................... [RUN]
22:18:12  6 of 6 PASS freshness of ecom.raw_supplies ..................................... [PASS in 0.00s]
22:18:13  5 of 6 WARN freshness of ecom.raw_stores ....................................... [WARN in 0.69s]
22:18:13  3 of 6 WARN freshness of ecom.raw_orders ....................................... [WARN in 0.71s]
22:18:13
22:18:13  Finished running 6 sources in 0 hours 0 minutes and 5.27 seconds (5.27s).
22:18:13  Done.

(twice as fast on jaffle shop! should be even better on projects with more sources)

Checklist

• I have read the contributing guide and understand what's expected of me
• I have run this code in development and it appears to resolve the stated issue
• This PR includes tests, or tests are not required/relevant for this PR
• This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc) or this PR has already received feedback and approval from Product or DX
• This PR includes type annotations for new and modified functions

Should not be merged before dbt-labs/dbt-adapters#127 is merged & released in dbt-adapters

[github-actions]

Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the contributing guide.

[codecov]

Codecov Report

Attention: Patch coverage is 80.95238% with 8 lines in your changes are missing coverage. Please review.

Project coverage is 88.08%. Comparing base (ebc22fa) to head (f95f86c).
Report is 15 commits behind head on main.

Files	Patch %	Lines
core/dbt/task/freshness.py	80.95%	8 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #9749      +/-   ##
==========================================
- Coverage   88.13%   88.08%   -0.05%     
==========================================
  Files         178      178              
  Lines       22449    22498      +49     
==========================================
+ Hits        19785    19818      +33     
- Misses       2664     2680      +16     

Flag	Coverage Δ
integration	85.40% <52.38%> (-0.16%)	⬇️
unit	62.02% <57.14%> (+0.13%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.