Does BigQuery template query take advantage of Partitions?

[judahrand]

I'm unclear whether due to the dynamic nature of these WHERE clauses this query will take advantage of table partitions on the event timestamp?

This could be solved by predetermining these min and max values and templating them in directly.

https://cloud.google.com/bigquery/docs/querying-partitioned-tables#pruning_limiting_partitions

feast/sdk/python/feast/infra/offline_stores/bigquery.py

Lines 489 to 491 in b5d8a53

	WHERE {{ featureview.event_timestamp_column }} <= (SELECT MAX(entity_timestamp) FROM entity_dataframe)
	{% if featureview.ttl == 0 %}{% else %}
	AND {{ featureview.event_timestamp_column }} >= Timestamp_sub((SELECT MIN(entity_timestamp) FROM entity_dataframe), interval {{ featureview.ttl }} second)

[adchia]

@MattDelac was looking into taking advantage of partitioning more so my guess would be no?

@mavysavydav might be interested too as an fyi

[MattDelac]

@MattDelac was looking into taking advantage of partitioning more so my guess would be no?

I don't think Feast does a good job with leveraging Partitions everywhere. The problem might not just be here.
We should definitely have a whole RFC about "Handling partitions end-to-end workflow"

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[adchia]

Closing this to funnel more discussions in #2530