[DOCS] Choosing and configuring DataConnectors #3533

NathanFarmer · 2021-10-12T16:54:58Z

Changes proposed in this pull request:

DEVREL-213
Migrate "How to configure an InferredAssetDataConnector" and put under test
Migrate "How to configure a ConfiguredAssetDataConnector" and put under test
Migrate "How to configure a RuntimeDataConnector" and put under test
Migrate "How to choose which DataConnector to use" and put under test

Notable changes not in this pull request:

Examples for SQL (everything here is for file-system-like)

Definition of Done

Please delete options that are not relevant.

My code follows the Great Expectations style guide
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have added unit tests where applicable and made sure that new and existing tests are passing.
I have run any local integration tests and made sure that nothing is broken.

netlify · 2021-10-12T16:55:03Z

✔️ Deploy Preview for niobium-lead-7998 ready!

🔨 Explore the source changes: 4033abc

🔍 Inspect the deploy log: https://app.netlify.com/sites/niobium-lead-7998/deploys/61816c132ab0fd0007b22542

😎 Browse the preview: https://deploy-preview-3533--niobium-lead-7998.netlify.app

github-actions · 2021-10-12T16:55:20Z

HOWDY! This is your friendly 🤖 CHANGELOG bot 🤖

Please don't forget to add a clear and succinct description of your change under the Develop header in docs_rtd/changelog.rst, if applicable. This will help us with the release process. See the Contribution checklist in the Great Expectations documentation for the type of labels to use!

✨ Thank you! ✨

…3533)

talagluck

This is awesome overall! I think we might need some more explanation on some specific topics. We may also need a bit more detail explaining when to use Configured vs Inferred. Is it just that you can use Inferred when Data Assets are not divided equally in the same location, but where there's inconsistency, you need to use Configured? Inferred are actually more configurable than I had expected, so making this more explicit would be helpful.

docs/guides/connecting_to_your_data/how_to_choose_which_dataconnector_to_use.md

docs/guides/connecting_to_your_data/how_to_configure_an_inferredassetdataconnector.md

docs/guides/connecting_to_your_data/how_to_configure_a_runtimedataconnector.md

docs/guides/connecting_to_your_data/how_to_configure_an_inferredassetdataconnector.md

…sset (#3533)

…m:great-expectations/great_expectations into docs/DEVREL-213/everything-dataconnectors

docs/guides/connecting_to_your_data/how_to_configure_an_inferredassetdataconnector.md

alexsherstinsky

Some comments inside.
I did not see any examples that use the Batch filtering capabilities (including the use of a custom Python based filter query.

talagluck

Looks great overall! Just a few more minor nits. I haven't gone through the testing piece in detail either.

talagluck · 2021-10-26T20:35:20Z

docs/guides/connecting_to_your_data/how_to_choose_which_dataconnector_to_use.md

+| InferredAssetDataConnectors | ConfiguredAssetDataConnectors |
+| --- | --- |
+| InferredAssetFilesystemDataConnector | ConfiguredAssetFilesystemDataConnector |
+| InferredAssetFilePathDataConnector | ConfiguredAssetFilePathDataConnector |
+| InferredAssetAzureDataConnector | ConfiguredAssetAzureDataConnector |
+| InferredAssetGCSDataConnector | ConfiguredAssetGCSDataConnector |
+| InferredAssetS3DataConnector | ConfiguredAssetS3DataConnector |
+| InferredAssetSqlDataConnector | ConfiguredAssetSqlDataConnector |


Love this table.

docs/guides/connecting_to_your_data/how_to_choose_which_dataconnector_to_use.md

docs/guides/connecting_to_your_data/how_to_configure_an_inferredassetdataconnector.md

talagluck · 2021-10-26T20:48:04Z

docs/guides/connecting_to_your_data/how_to_configure_a_configuredassetdataconnector.md

+
+The configuration would also work with a regex capturing the entire filename (e.g. `pattern: (.*)\.csv`).  However, capturing the month on its own allows for `batch_identifiers` to be used to retrieve a specific Batch of the Data Asset. For more information about capture groups, refer to the Python documentation on [regular expressions](https://docs.python.org/3/library/re.html#re.Match.group).
+
+Later on we could retrieve the data in `yellow_tripdata_2019-02.csv` of `yellow_tripdata` as its own batch using `context.get_validator()` by specifying `{"month": "2019-02"}` as the `batch_identifier`.


This is wild - I hadn't realized that you could specify batch_identifiers at the top level of the get_validator - I've only seen it specified within the batch_request.

docs/guides/connecting_to_your_data/how_to_configure_a_configuredassetdataconnector.md

…gure_a_configuredassetdataconnector.md

NathanFarmer · 2021-10-27T14:24:08Z

@alexsherstinsky

I did not see any examples that use the Batch filtering capabilities (including the use of a custom Python based filter query.

Is this referring to the data_connector_query parameter of BatchRequest?

If so, I have added an example to how_to_configure_an_inferredassetdataconnector.md lines 268-291.

…m:great-expectations/great_expectations into docs/DEVREL-213/everything-dataconnectors

talagluck

LGTM! Just one question/possible update, but I don't want it to gate. Amazing work!

docs/guides/connecting_to_your_data/how_to_configure_an_inferredassetdataconnector.md

talagluck

LGTM! Just one question/possible update, but I don't want it to gate. Amazing work!

alexsherstinsky

LGTM (strong work!)

Shinnnyshinshin

Looks wonderful :)

…-tests-for-docs * develop: DevRel - docs fixes (#3498) Remove --no-spark flag from docs tests (#3625) [DOCS] Choosing and configuring DataConnectors (#3533) [BUGFIX] Be able to use spark execution engine with spark reuse flag (#3541)

[DOCS] How to configure an InferredAssetDataConnector

6f05c13

Nathan Farmer and others added 6 commits October 12, 2021 12:56

[DOCS] Change log (#3533)

9507500

[DOCS] How to configure a ConfiguredAssetDataConnector (#3533)

c0bd2d1

[DOCS] How to choose which DataConnector to use (#3533)

e76cc75

[DOCS] How to configure a RuntimeDataConnector (#3533)

9485973

[DOCS] Add option for RuntimeDataConnector to how-to-choose document (#…

c3a4f34

…3533)

Merge branch 'develop' into docs/DEVREL-213/everything-dataconnectors

39e8b39

NathanFarmer requested a review from talagluck October 13, 2021 16:55

Merge branch 'develop' into docs/DEVREL-213/everything-dataconnectors

dbf1024

talagluck reviewed Oct 18, 2021

View reviewed changes

Nathan Farmer and others added 12 commits October 19, 2021 11:29

[DOCS] Clarify that the data we are connecting to is known as a DataA…

c43f569

…sset (#3533)

Merge branch 'docs/DEVREL-213/everything-dataconnectors' of github.co…

d31261c

…m:great-expectations/great_expectations into docs/DEVREL-213/everything-dataconnectors

[DOCS] Cleanup working and a typo (#3533)

979d581

[DOCS] Rearrange this list into the order of the examples below (#3533)

c9ca799

[DOCS] Typo (#3533)

bc05b3c

Typo

44985cb

Clarify that the data is what we are referring to as a DataAsset

5806b67

Clean up

32d2769

Basic working integration tests

5953dd8

Cleanup

98112be

Merge branch 'develop' into docs/DEVREL-213/everything-dataconnectors

b33d342

Test datasets

9d92240

NathanFarmer changed the title ~~[DOCS] Choosing and configuring Data Connectors~~ [DOCS] Choosing and configuring DataConnectors Oct 20, 2021

Nathan Farmer added 5 commits October 20, 2021 11:08

Rearrange test set directory structure and get basic tests functioning

c0051f6

YAML config example alongside Python

bdf1b05

Assert that yaml and python configs are equivalent

2dde0ec

Steps 1 and 2 with tabs

4408446

Remove test sets that are no longer needed

53616a2

alexsherstinsky reviewed Oct 26, 2021

View reviewed changes

docs/guides/connecting_to_your_data/how_to_configure_an_inferredassetdataconnector.md Show resolved Hide resolved

alexsherstinsky suggested changes Oct 26, 2021

View reviewed changes

talagluck reviewed Oct 26, 2021

View reviewed changes

Nathan Farmer added 6 commits October 27, 2021 09:18

Minor revisions

5b74dae

Example for data_references

8cc7e7e

Update glob_directive docstrings

b9accbc

Explain what glob_directive does in this context

4ea85cc

Better explanation for ConfiguredAssetDataConnector

c977227

Also add clarifiaction for what we mean by Data Asset to how_to_confi…

3630de2

…gure_a_configuredassetdataconnector.md

NathanFarmer added the core-team label Nov 1, 2021

Nathan Farmer and others added 6 commits November 2, 2021 09:19

Example for loading a specific batch with batch_identifiers

d21bfe8

Linting

2b1086c

Re-align line numbers after lint

0b79069

Merge branch 'develop' into docs/DEVREL-213/everything-dataconnectors

499fc83

Batching core concepts link

648f64d

Merge branch 'docs/DEVREL-213/everything-dataconnectors' of github.co…

8130542

…m:great-expectations/great_expectations into docs/DEVREL-213/everything-dataconnectors

NathanFarmer requested review from talagluck and alexsherstinsky November 2, 2021 13:49

Clean up

b780691

talagluck approved these changes Nov 2, 2021

View reviewed changes

docs/guides/connecting_to_your_data/how_to_configure_an_inferredassetdataconnector.md Show resolved Hide resolved

talagluck approved these changes Nov 2, 2021

View reviewed changes

alexsherstinsky approved these changes Nov 2, 2021

View reviewed changes

NathanFarmer enabled auto-merge (squash) November 2, 2021 15:24

Shinnnyshinshin approved these changes Nov 2, 2021

View reviewed changes

Merge branch 'develop' into docs/DEVREL-213/everything-dataconnectors

4033abc

donaldheppner approved these changes Nov 2, 2021

View reviewed changes

NathanFarmer merged commit 948d6ed into develop Nov 2, 2021

NathanFarmer deleted the docs/DEVREL-213/everything-dataconnectors branch November 2, 2021 17:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DOCS] Choosing and configuring DataConnectors #3533

[DOCS] Choosing and configuring DataConnectors #3533

NathanFarmer commented Oct 12, 2021 •

edited

Loading

netlify bot commented Oct 12, 2021 •

edited

Loading

github-actions bot commented Oct 12, 2021

talagluck left a comment

alexsherstinsky left a comment

talagluck left a comment

talagluck Oct 26, 2021

talagluck Oct 26, 2021

NathanFarmer commented Oct 27, 2021 •

edited

Loading

talagluck left a comment

talagluck left a comment

alexsherstinsky left a comment

Shinnnyshinshin left a comment


		The configuration would also work with a regex capturing the entire filename (e.g. `pattern: (.*)\.csv`). However, capturing the month on its own allows for `batch_identifiers` to be used to retrieve a specific Batch of the Data Asset. For more information about capture groups, refer to the Python documentation on [regular expressions](https://docs.python.org/3/library/re.html#re.Match.group).

		Later on we could retrieve the data in `yellow_tripdata_2019-02.csv` of `yellow_tripdata` as its own batch using `context.get_validator()` by specifying `{"month": "2019-02"}` as the `batch_identifier`.

[DOCS] Choosing and configuring DataConnectors #3533

[DOCS] Choosing and configuring DataConnectors #3533

Conversation

NathanFarmer commented Oct 12, 2021 • edited Loading

Definition of Done

netlify bot commented Oct 12, 2021 • edited Loading

github-actions bot commented Oct 12, 2021

HOWDY! This is your friendly 🤖 CHANGELOG bot 🤖

talagluck left a comment

Choose a reason for hiding this comment

alexsherstinsky left a comment

Choose a reason for hiding this comment

talagluck left a comment

Choose a reason for hiding this comment

talagluck Oct 26, 2021

Choose a reason for hiding this comment

talagluck Oct 26, 2021

Choose a reason for hiding this comment

NathanFarmer commented Oct 27, 2021 • edited Loading

talagluck left a comment

Choose a reason for hiding this comment

talagluck left a comment

Choose a reason for hiding this comment

alexsherstinsky left a comment

Choose a reason for hiding this comment

Shinnnyshinshin left a comment

Choose a reason for hiding this comment

NathanFarmer commented Oct 12, 2021 •

edited

Loading

netlify bot commented Oct 12, 2021 •

edited

Loading

NathanFarmer commented Oct 27, 2021 •

edited

Loading