Refactor Environment class and DataSourceCreator API, and use fixtures for datasets and data sources

[achals]

Signed-off-by: Achal Shah achals@gmail.com

What this PR does / why we need it:

This PR introduces a number of changes for speed and customizability:

• Removed ds and data_source from the Environment class, since they were used in an older version of the Environment.
• Added a DataSourceCache fixture to re-use staged data for speedups.
• Split out construct_universal_test_environment and construct_test_environment for both happy path integration tests and non-universal-repo integration tests.
• Removed the get_prefixed_table_name from the DataSourceCreator since it wasn't needed.

Additionally,

• Fixed schema inference and added test coverage on all sources.

Which issue(s) this PR fixes:

Fixes #1797

Does this PR introduce a user-facing change?:

none

[codecov-commenter]

Codecov Report

Merging #1790 (ff1fb81) into master (3c39261) will decrease coverage by 0.67%.
The diff coverage is 93.45%.

@@            Coverage Diff             @@
##           master    #1790      +/-   ##
==========================================
- Coverage   85.32%   84.64%   -0.68%     
==========================================
  Files          93       93              
  Lines        6950     6812     -138     
==========================================
- Hits         5930     5766     -164     
- Misses       1020     1046      +26     

Flag	Coverage Δ
integrationtests	84.57% <93.45%> (-0.69%)	⬇️
unittests	63.78% <42.90%> (+0.19%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
sdk/python/feast/infra/offline_stores/bigquery.py	80.30% <ø> (ø)
sdk/python/feast/infra/utils/aws_utils.py	88.34% <ø> (-2.92%)	⬇️
...ion/feature_repos/universal/data_source_creator.py	80.00% <ø> (+2.22%)	⬆️
...alization/test_offline_online_store_consistency.py	86.98% <ø> (-13.02%)	⬇️
...tegration/offline_store/test_s3_custom_endpoint.py	24.00% <25.00%> (-5.42%)	⬇️
sdk/python/feast/feature_view.py	86.30% <57.14%> (-0.75%)	⬇️
sdk/python/feast/type_map.py	67.46% <71.42%> (+2.46%)	⬆️
sdk/python/tests/conftest.py	96.29% <91.30%> (-3.71%)	⬇️
...ation/feature_repos/universal/data_sources/file.py	66.66% <93.33%> (+2.15%)	⬆️
...n/feature_repos/universal/data_sources/bigquery.py	97.61% <94.11%> (-2.39%)	⬇️
... and 19 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3c39261...ff1fb81. Read the comment docs.

[adchia]

/lgtm

[woop]

Apologies if this has been mentioned before, but can you remind me why we use a string reference to the class instead of a code reference?

[achals]

Mainly to not have the repo_configuration file not have any code dependency on any specific offline store, similar to the online/offline store config.

[woop]

Is there any downside to having that code dependency?

[achals]

Oops missed this. For tests, presumably not. However I can imagine we'll want this to be extensible to support the test suite for when users bring their own offline stores with a custom DataSourceCreator, so it may be needed then. Happy to change it for YAGNI, or keep it for forward looking flexibility

[woop]

My personal preference is to YAGNI this baby until we need it, just because I feel like we might run into bugs with these strings.

[achals]

Aight sg

[woop]

/lgtm

[feast-ci-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: achals, woop

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [achals,woop]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment