Add framework for instant email notifications of failing tests

[jeancochrane]

This PR adds support for a new meta.notify attribute to our dbt schema.yml syntax that can control automated email notifications for failing tests. It introduces a new test iasworld_pardat_spot_is_null that uses this system, along with a seed representing existing test failures; when combined, these two features allow us to send realtime notifications when a new value starts failing the test, without spamming us with notifications for known rows that are already failing.

I'll forward you a sample email so that you can see what the notification looks like. Here are the logs for the workflow that generated that notification.

Here's an overview for how this system works:

Sending a notification for a failing test

1. Get a dbt variable representing an SNS topic for the group of team members we want to email when this test fails
   • If no variable/topic combo exists yet for this group, a few config steps are required:
     1. Create the SNS topic
     2. Define a new empty dbt variable in dbt_project.yml that will store the ARN for the topic
     3. Set the ARN to a repo secret so that it is hidden but we can access it from GitHub workflows
     4. Update the scripts/run_iasworld_data_tests.py call in the test-dbt-models workflow to set the dbt variable you created in step 2 to the secret you created in step 3
   • If a variable/topic already exists for this group, no changes are necessary; just grab its dbt variable name
2. Update the test definition to add a meta.notify attribute whose value is the dbt variable name from step 1
3. When the test-dbt-models workflow runs, it will send a notification for any failures to the subscribers of the SNS topic from step 1

Restricting failure notifications to remove known failures

1. Add a seed representing the rows that we already know are failing for a given test
2. Adjust the interface for the generic test to add support for an anti_join argument that can join to another model and only return failures that are not already represented in that model
3. Update the test to supply the anti_join argument added in step 2 and point it to the seed added in step 1
4. When the test runs, it will only fail for rows that do not have a representation in the seed from step 1

Note that we have sketched out a simplified design for this system in #595, but we're planning to merge this version anyway so that we can get started configuring tests for notifications as soon as possible.

[rross0]

This is really cool stuff!

[jeancochrane]

One downside of this approach (setting topic ARNs as vars and overriding them with secrets during workflow execution) is that we'll have to update this bloated JSON dict every time we add a new topic. I don't think it's bad enough to outweigh the benefits of this approach, but it's a downside for sure.

[dfsnow]

suggestion (non-blocking): Why not just make a jq call or similar that maps the dbt vars to their expected env var equivalent, such that if we add a new dbt var it will automatically search for the corresponding env var? E.g.

• Add data_test_iasworld_commercial_sns_topic searches for $COMMERCIAL_SNS_TOPIC
• Add data_test_iasworld_asmt_sns_topic searches for $ASMT_SNS_TOPIC

[jeancochrane]

Hmm, this is an interesting idea, but I'm not totally convinced it's worth the complexity just yet. Let's plan to discuss in-person tomorrow.

[jeancochrane]

Based on our in-person discussion, I'm going to table this until we need to add more topics. I've added it to the future design in #595 to preserve this discussion.

[jeancochrane]

This unholy jq command was the simplest way I could think of parsing failures by topic and queuing up each one to send, but I'm open to ideas that will be more understandable to laypeople!

[dfsnow]

This is definitely a little bit cursed. What about doing it in Python? Also, let's add some conditionals on this (like those on L68) so that we don't accidentally spam folks if this workflow is triggered a bunch.

[jeancochrane]

Great ideas on both fronts! I gated this step behind a conditional in e5a615e and factored the esoteric jq call into a Python script in 8fbac89. Note that I decided to implement the gate via an input variable, so we'll have to update the Spark job in order to enable it. I think this is a feature rather than a bug, however, since it means we can merge this change without enabling it immediately.

[jeancochrane]

Requesting @dfsnow for a preliminary look at the design before I polish things up and add docs.

[dfsnow]

This is a great start! The variable passing stuff seems like a necessary evil, but the seed + anti-join stuff strikes me as a bit clunky. Let's brainstorm to see if we can't simplify things and then meet later this week. In the meantime, see my other comments for more minor actions.

[dfsnow]

This is definitely a little bit cursed. What about doing it in Python? Also, let's add some conditionals on this (like those on L68) so that we don't accidentally spam folks if this workflow is triggered a bunch.

[dfsnow]

Just so I understand correctly: test failures are grouped by topic, such that different failing tests would end up in the same single email ala:

           - {table_name}: {description} ({status})
                * {fail1_key1}: {fail1_value1}, {fail1_key2}: {fail1_value2}
           - {table_name2}: {description} ({status})
                * {fail1_key1}: {fail1_value1}, {fail1_key2}: {fail1_value2}
                * {fail2_key1}: {fail2_value1}, {fail2_key2}: {fail2_value2}

Is that right?

[jeancochrane]

Yup, that's right! To make extra sure we're on the same page, "topic" here specifically means SNS topic, i.e. tests that are tagged with the same SNS topic representing a group of recipients will be merged into one email that goes out to those recipients, regardless of the category of the tests.

[dfsnow]

suggestion (non-blocking): Why not just make a jq call or similar that maps the dbt vars to their expected env var equivalent, such that if we add a new dbt var it will automatically search for the corresponding env var? E.g.

• Add data_test_iasworld_commercial_sns_topic searches for $COMMERCIAL_SNS_TOPIC
• Add data_test_iasworld_asmt_sns_topic searches for $ASMT_SNS_TOPIC

[dfsnow]

suggestion (non-blocking): Let's add a call to action to the body as well. Something like "These were autogenerated by the Data Team. You will stop receiving these emails once the failures below are fixed" and then also our contact info at the bottom.

[jeancochrane]

I like that, done in c5096a8. I added our names instead of contact info, to avoid committing our email addresses to the source code; let me know if you don't mind the risk of spam and I can add email addresses in.