[BUGFIX] Be able to use spark execution engine with spark reuse flag

[fep2]

Changes proposed in this pull request:

• adding force_reuse_spark_context to Execution Engine Schema

• My code follows the Great Expectations style guide

• I have performed a self-review of my own code

• I have added unit tests where applicable and made sure that new and existing tests are passing.

• I have run any local integration tests and made sure that nothing is broken.

Thank you for submitting!

[netlify]

👷 Deploy request for niobium-lead-7998 pending review.
Visit the deploys page to approve it

🔨 Explore the source changes: afc1e1d

[talagluck]

Thanks so much for opening this PR, @fep2! We will review and be in touch in the next couple of days. In the meantime, it appears that some tests are failing. Would you mind taking a look and seeing if you can resolve? Please let me know if you need any help with that.

[talagluck]

Hi @fep2 - just wanted to check in, would you be up for seeing which tests are failing and fixing them? Thanks!

[fep2]

@talagluck I found the issue for why it breaks last week, I need to update some of the class in order for the test to pass. I'm making progress
great_expectations/core/util.py:604: if not force_reuse_spark_context and spark_restart_required( great_expectations/data_context/types/base.py:672: force_reuse_spark_context = fields.Boolean(required=False, missing=False) great_expectations/data_context/types/base.py:792: force_reuse_spark_context = fields.Bool(required=False, allow_none=True) great_expectations/datasource/sparkdf_datasource.py:70: force_reuse_spark_context=False, great_expectations/datasource/sparkdf_datasource.py:103: "force_reuse_spark_context": force_reuse_spark_context, great_expectations/datasource/sparkdf_datasource.py:119: force_reuse_spark_context=False, great_expectations/datasource/sparkdf_datasource.py:136: force_reuse_spark_context, great_expectations/datasource/sparkdf_datasource.py:155: force_reuse_spark_context=force_reuse_spark_context, great_expectations/execution_engine/sparkdf_execution_engine.py:154: force_reuse_spark_context=False, great_expectations/execution_engine/sparkdf_execution_engine.py:165: force_reuse_spark_context=force_reuse_spark_context,

those path need to be updated, and that should hopefully fix the test.

[fep2]

@talagluck I was able to fix the test woo hoo!

[fep2]

Now some are failing boo, but they don't seem to be related to the MR.

[talagluck]

Amazing - thanks so much for following up, and thanks for the contribution! LGTM!