[MAINTENANCE] Add force_reuse_spark_context to DatasourceConfigSchema

[pasmavie]

With PR 2733 a parameter was added to SparkDFDatasource, to make GE's context reuse an existing Spark Context.

This was extremely useful.

However, when using a dynamic Data Context configuration (e.g. in EMR) like

data_context_config = DataContextConfig(
    config_version=2,
    plugins_directory=None,
    config_variables_file_path=None,
    datasources={
        "my_data_source": {
            "class_name": "SparkDFDatasource",
            "spark_config": dict(spark.sparkContext.getConf().getAll()),
            "force_reuse_spark_context": True,
            "module_name": "great_expectations.datasource",
            "batch_kwargs_generators": {},
        }
    },
    ...

I've found that the spark_config and force_reuse_spark_context parameters weren't actually passed to geat_expectations.core.util.get_or_create_spark_application().

In fact, the parameters are lost when the DataContextConfig object is dumped into a dictionary, because the DatasourceConfigSchema (elem of the DataContextConfigSchema) doesn't list these parameters.

Changes proposed in this pull request:

• Add spark_config and force_reuse_spark_context to the DatasourceConfigSchema

Definition of Done

Please delete options that are not relevant.

• My code follows the Great Expectations style guide
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added unit tests where applicable and made sure that new and existing tests are passing.
• I have run any local integration tests and made sure that nothing is broken.

[netlify]

✔️ Deploy Preview for knoxpod ready!

🔨 Explore the source changes: 227511e

🔍 Inspect the deploy log: https://app.netlify.com/sites/knoxpod/deploys/60ed73cafe66250007930236

😎 Browse the preview: https://deploy-preview-2968--knoxpod.netlify.app

[gx-cla-bot]

A new contributor, HUZZAH! Welcome and thanks for joining our community. In order to accept a pull request we require that all contributors sign our Contributor License Agreement. We have two different CLAs, depending on whether you are contributing to GE in a personal or professional capacity. Please sign the one that is applicable to your situation so that we may accept your contribution:

Individual Contributor License Agreement v1.0
Software Grant and Corporate Contributor License Agreement v1.0

Once you have signed the CLA, you can add a comment with the text @cla-bot check and the bot will update the PR status!

Please reach out to @kyleaton, on our Slack if you have any questions or if you have already signed the CLA and are receiving this message in error.

Users missing a CLA: gio@gio

[pasmavie]

@cla-bot check

[talagluck]

Hi @gipaetusb, thank you so much for the contribution - this is awesome! Just a few things:

• It looks like we have a CLA record for the username gipaetusb, but there were a few commits made as gio or gio@gio and we are missing a CLA for that username. Are you able to fix this? Please let me know if you have any questions.
• It looks like a number of unit tests are failing as a result, and they will need to be updated, but I would expect the changes will be pretty small.
• Do you think this PR would benefit from a unit test for the additional functionality?
  Please let me know if you have any questions or need additional help. Thanks again!

[pasmavie]

@cla-bot check

[pasmavie]

Hi @talagluck, thanka for taking a look:

• I've amended the commits so that the username is gipaetusb for all. I hope it's good now.
• Regarding the failing unit tests, I don't have time to take a look now but I can do in the future :)
• I'll also add a unit test for this feat

[talagluck]

Hi @gipaetusb - thanks for the follow-up! It looks like the CLA verification is still failing, and I still see the user gio when I look at the commit history. Any ideas?

Sounds good about the tests - I'll check in in in a week or so to see how things are going. Thanks again!

[talagluck]

Hi @gipaetusb ! I wanted to check in here. It looks like we are still having CLA issues. I also wanted to check in to see if you had time to fix the failing tests, or if you needed some help?

[pasmavie]

Hi @talagluck thank you for the follow up.
I'm pretty busy lately so I didn't try to fix the CLA nor the unit test. Dedicated 30min this morning but amending the old commit and rebasing this branch with all the changes that happened after has become too cumbersome.

Much easier to open a new, cleaner PR right? Here you go: #3126

If someone can help with the unit tests, that would be great :) !

[talagluck]

Thanks for the follow-up, @gipaetusb ! Someone from our team will jump in next week to review and work on the unit tests.