fix: Fix materialization when running on Spark cluster.

[ckarwicki]

What this PR does / why we need it:
When running materialization and have Spark offline store configured to use cluster (spark.master pointing to actual Spark master node) self.to_spark_df().write.parquet(temp_dir, mode="overwrite") will create parquet file in worker node but return pq.read_table(temp_dir) is executed on driver node and it can't read from worker. Proposed fix makes materialization work when run on Spark cluster.

Which issue(s) this PR fixes:

Fixes #

[achals]

Thanks for the PR @ckarwicki , have you been able to test this out yourself?

[achals]

/ok-to-test

[codecov-commenter]

Codecov Report

Base: 66.85% // Head: 75.69% // Increases project coverage by +8.84% 🎉

Coverage data is based on head (b9d36c1) compared to base (d7b0c52).
Patch coverage: 0.00% of modified lines in pull request are covered.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3166      +/-   ##
==========================================
+ Coverage   66.85%   75.69%   +8.84%     
==========================================
  Files         175      211      +36     
  Lines       15848    17923    +2075     
==========================================
+ Hits        10595    13567    +2972     
+ Misses       5253     4356     -897     

Flag	Coverage Δ
integrationtests	66.85% <ø> (-0.01%)	⬇️
unittests	57.90% <0.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...ffline_stores/contrib/spark_offline_store/spark.py	7.72% <0.00%> (ø)
...on/feast/infra/materialization/snowflake_engine.py	92.09% <0.00%> (-0.05%)	⬇️
sdk/python/feast/infra/provider.py	77.94% <0.00%> (ø)
sdk/python/feast/infra/passthrough_provider.py	96.29% <0.00%> (ø)
.../python/feast/infra/feature_servers/base_config.py	100.00% <0.00%> (ø)
...n/feast/infra/feature_servers/aws_lambda/config.py	100.00% <0.00%> (ø)
..._stores/contrib/postgres_offline_store/postgres.py	34.14% <0.00%> (ø)
...ine_stores/contrib/cassandra_repo_configuration.py	100.00% <0.00%> (ø)
...offline_stores/contrib/spark_repo_configuration.py	20.00% <0.00%> (ø)
...thon/feast/infra/utils/postgres/postgres_config.py	100.00% <0.00%> (ø)
... and 113 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

[ckarwicki]

@achals Yes, this has been tested on Spark cluster and local mode.

[adchia]

Hey! @ckarwicki can you also sign your commits?

[niklasvm]

This may break some of the unit tests that test for data types being preserved between arrow and spark. I previously changed this method to resolve failing unit tests.

[ckarwicki]

@niklasvm Unit test are all fine - you can see unit test checks on this PR - all are greed. This change preserves types and only removes code to create parquet files - which fails when run on Spark cluster. Instead of creating parquet file and then reading it to create Arrow table we are directly creating Arrow table from Pandas df. Besides toPandas() internally is also using Arrow.

[kevjumba]

@ckarwicki can you sign your commits, the DCO check details should give you a command to sign them

[ckarwicki]

@kevjumba signed commits.

[kevjumba]

/lgtm

[feast-ci-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: adchia, ckarwicki, kevjumba

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [adchia,kevjumba]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment