Fix BQ historical retrieval with rows that got backfilled

[MattDelac]

What this PR does / why we need it:
There is bug about finding the "latest" values when the rows have been backfilled (when created_timestamp and event_timestamp don't follow the same order).

The problem is that the ANY_VALUE strategy does not work here because it will take the first occurrence it sees.

For example, the following example would fail

| driver_id | value | event_timestamp | created_at |
|-----------|-------|-----------------|------------|
| 1         | 10    | today           | tomorrow   |
| 1         | 20    | tomorrow        | today      |

The JOIN happening later would consist of the entity_dataframe with the __latest CTE where created_at and event_timestamp got mixed

| driver_id | value | event_timestamp | created_at |
|-----------|-------|-----------------|------------|
| 1         | 20    | tomorrow        | tomorrow   |

Because the JOIN operation happens on USING(driver_id, event_timestamp, created_at), it cannot find a match and will return NULL for the desired feature.

The goal of this PR is to make this step work and deterministic by using a Window function. Then we preserve the information between event_timestamp and created_at on a given row

Tophat

# Logs of the example I ran (on billions of records)
Job started at 2021-08-05 11:07:28
Job started at 2021-08-05 11:42:00 # Thus it took 35 minutes

Metrics (compared to current SQL template)
Elapsed time: 24 min 49 sec (vs 26 min 35 sec)
Slot time consumed: 21 days 6 hr (vs 26 days 0 hr)
Bytes shuffled: 23.11 TB (vs 22.86 TB)
Bytes spilled to disk: 7.87 TB (10.09 TB)

I also confirm that it fixes the bug we had on a specific feature view

The new test properly catches the problem encountered with the backfill rows (as the feature of the second driver is Null instead of 40)

Which issue(s) this PR fixes:

Fixes #

Does this PR introduce a user-facing change?:

[Bug] Fixed how we retrieve the latest value of a feature in the BigQuery SQL template

[feast-ci-bot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[codecov-commenter]

Codecov Report

Merging #1744 (0fb046e) into master (7977a53) will increase coverage by 0.08%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #1744      +/-   ##
==========================================
+ Coverage   84.65%   84.73%   +0.08%     
==========================================
  Files          85       85              
  Lines        6268     6297      +29     
==========================================
+ Hits         5306     5336      +30     
+ Misses        962      961       -1     

Flag	Coverage Δ
integrationtests	84.65% <100.00%> (+0.08%)	⬆️
unittests	64.25% <6.89%> (-0.27%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
sdk/python/feast/infra/offline_stores/bigquery.py	80.30% <ø> (ø)
...gration/offline_store/test_historical_retrieval.py	99.05% <100.00%> (+0.09%)	⬆️
...python/feast/infra/offline_stores/offline_utils.py	92.22% <0.00%> (+1.11%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7977a53...0fb046e. Read the comment docs.

[feast-ci-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: MattDelac, woop

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [woop]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[woop]

/lgtm