[Q1C2] More consistent, configurable tests

[jtcohen6]

The next version of dbt is going to focus on improving dbt test, along the lines of:

• More consistency between schema and data tests
• More consistency between tests and other dbt resources
• Straightforward, compelling extensions around filters, limits, expectation thresholds
• Improved development-time debugging by storing failing records in the database
• Rethinking, as appropriate, the YAML specs currently used to define tests

There are many open issues in the v0.20.0 milestone with the tests tag. This epic will track our broader initiative over the next several weeks, as it's likely we'll consolidate several open issues and close/rewrite others.

[jtcohen6]

@kwigley I wanted to summarize our conversation yesterday about how we're thinking of proceeding:

1. A test materialization, that performs node compilation + context management currently in python
   a. Hook it up to one-off tests (f.k.a. "data tests", Move data test "rendering" out of python compilation step #3154)
   b. Hook it up to generic macro tests (f.k.a. "schema tests," Use test materialization when executing ✨generic✨ tests #3192)
   c. Materialization handles the report-back of adapter response / result status (part of [CT-1635] Populate adapter_response for tests + sources #2964)
2. A new test block, to replace test__ macros (Schema test blocks #1173)
3. Net-new configs and resource properties that will make all tests more powerful and consistent (e.g. Threshold argument for Test Severity #2219, Allow instances of generic data tests to be documented #2578, parts of Schema test blocks #1173, with more issues TK)
4. Storing test failures in the database for quicker debugging in development (Storing test results in the database #2593)

Why this order of operations? We believe 1+2 represent foundational work that will pay dividends by smoothing the path toward the bigger wins in 3+4. Ideally, we ship the latter in v0.20.0 as reward/justification for the breaking changes earlier on. Even if we don't get there in time, we'll have built the right foundation to add them later on.

---

Smaller opportunities

• Should we change the syntax of the relationships tests to make it more consistent with other Jinja-in-YAML?
• Should we add/expand builtin generic tests (i.e. upstream some of the functionality from dbt-utils)?

---

Honorable mentions

Bigger things we should do before v1.0, but which I think are out of scope of the current effort:

• Smarter selection for tests with multiple parents (Run tests IFF all first-order parents are selected #2891)
• An "ur-"invocation that runs models, tests tests, and does all the rest (Generalized dbt build command #2743)
• Rationalizing node configs and resource properties (Set configs in schema.yml files #2401). Schema tests blur the lines here already today, and some of the work in step 3 above is going to make us feel the arbitrariness of this distinction even more.

---

Future art

Lots more to say, think, and do here over the long run. Not prerequisites to v1.0.

• Unit testing SQL with dbt constructs (Unit Testing SQL in DBT #2354)
• Regression testing, perhaps leveraging deferred references / a comparison manifest (Automating Non Regression Test : How to get a ref() to the deferred version of the selected model ? #2740)
• Expecting Jinja/python compilation errors to unit/integration test Jinja macros (Way to have test pass if compile error raised #3060)

[joellabes]

• Should we add/expand builtin generic tests (i.e. upstream some of the functionality from dbt-utils)?

At a minimum, I'd love to see the _where versions of each of the core tests brought in as first-class citizens