Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow setting of event_time as a top-level property of a model with a field_name attribute #10634

Closed
1 task done
QMalcolm opened this issue Aug 29, 2024 · 2 comments
Closed
1 task done
Assignees

Comments

@QMalcolm
Copy link
Contributor

QMalcolm commented Aug 29, 2024

Housekeeping

  • I am a maintainer of dbt-core

Short description

In #10624 we're building a new type of incremental model strategy, microbatch. This will be an awesome system wherein we do reads and writes for incremental via partitions. As part of that, we need a column to partition by. We're gonna call the key for defining that column field_name, and it should live under a top level config property event_time. To specify an event time field name in a model's yaml, it would look something like the following

models:
 - name: my_model
   config:
     event_time: 
       field_name: my_time_field # <-- This is the key/value we're talking about
       ...
   columns: 
     - name: my_time_field
       data_type: timestamp

If you were to specify it in the model's .sql, it would look something like the following

{{
    config(
        materialized = 'incremental',
        event_time = {"field_name": "my_time_field"}, # <-- This is the key/value we're talking about
    )
}}
...

Acceptance criteria

The event_time.field_name configuration is setable via .sql and .yml for models, and this propagates to the eventual node in the manifest

Suggested Tests

  • Set event_time.field_name in the configuration of a model .sql file and ensure it propagates to the eventual node in the manifest
  • Set event_time.field_name in a model .yml file and ensure it propagates to the eventual node in the manifest

Impact to Other Teams

Affects CA: New optional property on model nodes

Will backports be required?

No

Context

No response

@QMalcolm QMalcolm changed the title Allow setting of event_time as a top-level property of a model Allow setting of event_time as a top-level property of a model with a field_name attribute Aug 29, 2024
@graciegoheen
Copy link
Contributor

We can close out this old issue as a dupe!

@QMalcolm
Copy link
Contributor Author

QMalcolm commented Sep 4, 2024

Closing as duplicate of #9490

@QMalcolm QMalcolm closed this as completed Sep 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants