fix for events incremental #51

fivetran-reneeli · 2024-10-25T16:09:35Z

PR Overview

This PR will address the following Issue/Feature:
#45 #48

This PR will result in the following new package version:

v0.13.0

Please provide the finalized CHANGELOG entry which details the relevant changes included in this PR:

Breaking Changes (`--full-refresh` required after upgrading)

Added a field called first_open_or_click_event_at in the iterable__user_campaign model. This timestamp shows the first time a user interacted with a campaign, recording the earliest occurring event out of 'emailOpen', 'emailClick', and 'pushOpen'. (PR #50)
Corrected the incremental filter in iterable__events model to now use the created_on date field instead of the created_at timestamp.
- Previously, this would potentially exclude late-arriving new records from populating in the end models if they had an older created_at value than what was present in the model. Switching to created_on widens the criteria.
Updated upstream stg_iterable__user_history model from materializing as a table to a view in order to improve performance.
In order to ensure no issues a --full-refresh is required after upgrading.

Under the Hood

In addition to using created_on in the incremental logic in iterable__events, we introduced a iterable_lookback_window variable to increase the window for accommodating potential late-arriving records. The default is 7 days prior to the maximum created_on value present in the iterable__events model, but you may customize this by setting the var iterable_lookback_window in your dbt_project.yml. See the Lookback Window section of the README for more details.
Added a section in the README about the iterable__event_metrics variable and how to use it to specify which event metrics to pivot out.
Removes created_on from the uniqueness test in iterable__events. Uniqueness is now tested solely on unique_event_id, a surrogate key made up of event_id (_fivetran_id in the raw table, which is a Fivetran-created unique identifier derived from hashing campaign_id, created_at, and event_name) and _fivetran_user_id (a Fivetran-created column derived from a hash of user_id and/or email).
Modified the event seed data to more accurately represent real-life data, with a unique _fivetran_id for each campaign_id, created_at, and event_name.

PR Checklist

Basic Validation

Please acknowledge that you have successfully performed the following commands locally:

dbt run –full-refresh && dbt test
dbt run (if incremental models are present) && dbt test

Before marking this PR as "ready for review" the following have been applied:

The appropriate issue has been linked, tagged, and properly assigned
All necessary documentation and version upgrades have been applied
docs were regenerated (unless this PR does not include any code or yml updates)
BuildKite integration tests are passing
[na] Detailed validation steps have been provided below

Detailed Validation

Please share any and all of your validation steps:

see internal ticket thread for validation in addition to the below:

This error in consistency_user_campaign is expected due to the addition of a new field first_open_or_click_event_at in this branch.

If you had to summarize this PR in an emoji, which would it be?

💃

User Campaign - First open or click event time

… first_open_or_click_event_at, update docs

fivetran-joemarkiewicz

@fivetran-reneeli thanks for working through this. This is looking really good I just have a few comments and suggestions before approving. Also, would you please be able to run the validation tests for this package and share the results in the PR description to confirm there are no unexpected changes between dev and prod.

CHANGELOG.md

fivetran-joemarkiewicz · 2024-10-28T21:33:59Z

CHANGELOG.md

+- Adds a field called `first_open_or_click_event_at` in the `iterable__user_campaign` model. This timestamp shows the first time a user interacted with a campaign, recording the earliest occurring event out of 'emailOpen', 'emailClick', and 'pushOpen'. ([PR #50](https://github.com/fivetran/dbt_iterable/pull/50))
+
+## Bug Fix 
+- Updates the incremental logic in `iterable__events` to use the `created_on` date field instead of the `created_at` timestamp. Previously, this would potentially exclude late-arriving new records from populating in the end models if they had an older `created_at` value than what was present in the model. Switching to `created_on` widens the criteria.


This should be listed in the breaking change section

README.md

fivetran-joemarkiewicz · 2024-10-28T21:53:17Z

packages.yml

+  #   version: [">=0.10.0", "<0.11.0"]
+
+  - git: https://github.com/fivetran/dbt_iterable_source.git
+    revision: rm_src_freshness
+    warn-unpinned: false


Reminder to swap before merge

models/iterable__events.sql

README Update iterable event metrics

fivetran-reneeli · 2024-10-29T15:32:20Z

Thanks @fivetran-joemarkiewicz this is ready for re-review. I updated with the validation test results. Actually, as expected, there was one test failure on consistency_user_campaign, due to the addition of the new first_open_or_click_event_at field.

…/created_on definitions in yml, plus update changelog

fivetran-joemarkiewicz

@fivetran-reneeli thanks for working through these changes! I have a few comments before this is ready for approval. Let me know if you have questions and once this is ready for final review.

CHANGELOG.md

models/iterable__events.sql

CHANGELOG.md

fivetran-joemarkiewicz · 2024-10-30T19:58:48Z

CHANGELOG.md

+# dbt_iterable v0.13.0
+[PR #51](https://github.com/fivetran/dbt_iterable/pull/51) includes the following updates:
+
+## Breaking Changes


Suggested change

## Breaking Changes

## Breaking Changes (`--full-refresh` required after upgrading)

integration_tests/dbt_project.yml

models/iterable__events.sql

Co-authored-by: Joe Markiewicz <74217849+fivetran-joemarkiewicz@users.noreply.github.com>

…nt_Data

fivetran-joemarkiewicz

@fivetran-reneeli this is really close, just a few small changes before approval. Let me know once their updated. Thanks!

CHANGELOG.md

README.md

models/iterable.yml

Co-authored-by: Joe Markiewicz <74217849+fivetran-joemarkiewicz@users.noreply.github.com>

fivetran-joemarkiewicz

@fivetran-reneeli changes look good to me! Just a small CHANGELOG edit request and a notice that I made a small README change to close a codeblock. Once the CHANGELOG edit is made this is ready for release review.

CHANGELOG.md

fivetran-joemarkiewicz · 2024-11-04T15:31:54Z

README.md

+    - "emailBounce"
+    - "inAppClick"
+    - "pushUninstall"
+```


This ending codeblock was not captured before. It seems the suggestion didn't like the ending codeblock in a codeblock. I just committed this directly to your branch to address the issue.

oh gotcha, thank you!

Co-authored-by: Joe Markiewicz <74217849+fivetran-joemarkiewicz@users.noreply.github.com>

CHANGELOG.md

fivetran-avinash · 2024-11-04T20:40:23Z

integration_tests/seeds/event_data.csv

@fivetran-reneeli We should probably have the seed files match on both source and transform packages. Can you update one or the other? Not sure what works best here for more robust testing.

Thanks @fivetran-avinash , I believe they're the same now

CHANGELOG.md

fivetran-avinash

@fivetran-reneeli Nice work! A few comments before approval.

CHANGELOG.md

fivetran-avinash

@fivetran-reneeli Oops, something went wonky here--had two tabs open with comments/suggestions and submitted both. Let me know if you have any questions.

Co-authored-by: Avinash Kunnath <108772760+fivetran-avinash@users.noreply.github.com>

fivetran-avinash

@fivetran-reneeli Ready to merge!

justin-fundrise and others added 7 commits October 9, 2024 16:53

adding missing variable

cf4ef97

new field for open or click

c59f8d5

changelog and versions

ed87003

fix for events incremental

c438cb7

versioning, documentation, readme updates, yml, add new field

21f080e

new schema and docs

88ebc6f

small changelog update

23e9078

fivetran-joemarkiewicz assigned fivetran-reneeli Oct 25, 2024

fivetran-joemarkiewicz self-requested a review October 25, 2024 19:24

fivetran-reneeli and others added 3 commits October 28, 2024 11:10

Merge branch 'release/v0.13.0' into first-open-click

6e8cd2c

Merge pull request #50 from justin-fundrise/first-open-click

1c57bd0

User Campaign - First open or click event time

merge customer PR, change field name from first_open_or_click_time to…

6a55ec5

… first_open_or_click_event_at, update docs

fivetran-joemarkiewicz requested changes Oct 28, 2024

View reviewed changes

fivetran-reneeli and others added 4 commits October 29, 2024 11:03

update order of macro inputs and update changelog

329642c

Merge branch 'release/v0.13.0' into readme_update_missing_var

a397489

Merge pull request #49 from justin-fundrise/readme_update_missing_var

ad21bec

README Update iterable event metrics

docs and readme

e446d71

This was linked to issues Oct 29, 2024

FEATURE - add iterable__event_metrics configuration instructions to README #10

Closed

[Bug] Review the incremental logic in iterable__events #45

Closed

[Feature] User Campaign First Open or Click Time #48

Closed

fivetran-reneeli mentioned this pull request Oct 29, 2024

[Feature] Update incremental partition from daily to weekly in iterable__events #52

Open

4 tasks

fivetran-reneeli added 2 commits October 29, 2024 22:07

add created_at to incremental logic unique key and correct created_at…

350b9c9

…/created_on definitions in yml, plus update changelog

update docs

f0bb24e

fivetran-joemarkiewicz requested changes Oct 30, 2024

View reviewed changes

fivetran-reneeli and others added 5 commits October 31, 2024 11:41

Update CHANGELOG.md

f6699c9

Co-authored-by: Joe Markiewicz <74217849+fivetran-joemarkiewicz@users.noreply.github.com>

remove created_at from unique test on event model and update seed eve…

2100686

…nt_Data

update lookback_window var to prefix w iterable

bbb79f6

updates and docs

f26b05c

adjust language in changelog

90a305f

fivetran-joemarkiewicz requested changes Nov 1, 2024

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

CHANGELOG.md Show resolved Hide resolved

README.md Outdated Show resolved Hide resolved

models/iterable.yml Show resolved Hide resolved

fivetran-reneeli and others added 6 commits November 1, 2024 15:56

Update CHANGELOG.md

bf2a4a4

Co-authored-by: Joe Markiewicz <74217849+fivetran-joemarkiewicz@users.noreply.github.com>

Update README.md

2c032bc

Co-authored-by: Joe Markiewicz <74217849+fivetran-joemarkiewicz@users.noreply.github.com>

changelog updates

a316934

new schema

c91f5dc

changelog

d168b36

Add end of codeblock

c0ea5e9

fivetran-joemarkiewicz approved these changes Nov 4, 2024

View reviewed changes

Update CHANGELOG.md

4224134

Co-authored-by: Joe Markiewicz <74217849+fivetran-joemarkiewicz@users.noreply.github.com>

fivetran-avinash reviewed Nov 4, 2024

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

fivetran-avinash reviewed Nov 4, 2024

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

fivetran-avinash reviewed Nov 4, 2024

View reviewed changes

CHANGELOG.md Show resolved Hide resolved

fivetran-avinash reviewed Nov 4, 2024

View reviewed changes

fivetran-reneeli and others added 5 commits November 4, 2024 17:43

Update CHANGELOG.md

73656ec

Co-authored-by: Avinash Kunnath <108772760+fivetran-avinash@users.noreply.github.com>

Update CHANGELOG.md

e6ff0fa

Co-authored-by: Avinash Kunnath <108772760+fivetran-avinash@users.noreply.github.com>

Update CHANGELOG.md

a38124f

Co-authored-by: Avinash Kunnath <108772760+fivetran-avinash@users.noreply.github.com>

Update CHANGELOG.md

04f1d87

Co-authored-by: Avinash Kunnath <108772760+fivetran-avinash@users.noreply.github.com>

new schema

d9b5c0e

fivetran-avinash approved these changes Nov 5, 2024

View reviewed changes

change deps

59b28e7

fivetran-reneeli merged commit e2a6b56 into main Nov 5, 2024
8 checks passed

fivetran-reneeli deleted the release/v0.13.0 branch November 5, 2024 17:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix for events incremental #51

fix for events incremental #51

fivetran-reneeli commented Oct 25, 2024 •

edited

Loading

fivetran-joemarkiewicz left a comment

fivetran-joemarkiewicz Oct 28, 2024

fivetran-reneeli Oct 29, 2024

fivetran-joemarkiewicz Oct 28, 2024

fivetran-avinash Nov 4, 2024

fivetran-reneeli commented Oct 29, 2024

fivetran-joemarkiewicz left a comment

fivetran-joemarkiewicz Oct 30, 2024

fivetran-reneeli Oct 31, 2024

fivetran-joemarkiewicz left a comment

fivetran-joemarkiewicz left a comment

fivetran-joemarkiewicz Nov 4, 2024

fivetran-reneeli Nov 4, 2024

fivetran-avinash Nov 4, 2024

fivetran-reneeli Nov 4, 2024

fivetran-avinash left a comment

fivetran-avinash left a comment •

edited

Loading

fivetran-avinash left a comment

	## Breaking Changes
	## Breaking Changes (`--full-refresh` required after upgrading)

fix for events incremental #51

fix for events incremental #51

Conversation

fivetran-reneeli commented Oct 25, 2024 • edited Loading

PR Overview

Breaking Changes (--full-refresh required after upgrading)

Under the Hood

PR Checklist

Basic Validation

Detailed Validation

If you had to summarize this PR in an emoji, which would it be?

fivetran-joemarkiewicz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fivetran-reneeli commented Oct 29, 2024

fivetran-joemarkiewicz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fivetran-joemarkiewicz left a comment

Choose a reason for hiding this comment

fivetran-joemarkiewicz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fivetran-avinash left a comment

Choose a reason for hiding this comment

fivetran-avinash left a comment • edited Loading

Choose a reason for hiding this comment

fivetran-avinash left a comment

Choose a reason for hiding this comment

fivetran-reneeli commented Oct 25, 2024 •

edited

Loading

Breaking Changes (`--full-refresh` required after upgrading)

fivetran-avinash left a comment •

edited

Loading