Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: Add documentation for custom offline stores #2430

Merged
merged 4 commits into from
Mar 22, 2022
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions docs/how-to-guides/adding-a-new-offline-store.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,8 @@ The OfflineStore class contains a couple of methods to read features from the of
There are two methods that deal with reading data from the offline stores`get_historical_features`and `pull_latest_from_table_or_query`.

* `pull_latest_from_table_or_query` is invoked when running materialization (using the `feast materialize` or `feast materialize-incremental` commands, or the corresponding `FeatureStore.materialize()` method. This method pull data from the offline store, and the `FeatureStore` class takes care of writing this data into the online store.
* `get_historical_features `is invoked when reading values from the offline store using the `FeatureStore.get_historica_features()` method. Typically, this method is used to retrieve features when training ML models.
* `get_historical_features` is invoked when reading values from the offline store using the `FeatureStore.get_historical_features()` method. Typically, this method is used to retrieve features when training ML models.
* `pull_all_from_table_or_query` is a method that pulls all the data from an offline store from a specified start date to a specified end date.

{% code title="feast_custom_offline_store/file.py" %}
```python
Expand Down Expand Up @@ -66,6 +67,7 @@ There are two methods that deal with reading data from the offline stores`get_hi
created_timestamp_column,
start_date,
end_date)
)
kevjumba marked this conversation as resolved.
Show resolved Hide resolved
```
{% endcode %}

Expand Down Expand Up @@ -223,7 +225,7 @@ To use our custom file offline store, we can use the following `feature_store.ya
project: test_custom
registry: data/registry.db
provider: local
offline_store:
offline_store:
type: feast_custom_offline_store.file.CustomFileOfflineStore
```
{% endcode %}
Expand Down
24 changes: 24 additions & 0 deletions sdk/python/feast/infra/offline_stores/offline_store.py
Original file line number Diff line number Diff line change
Expand Up @@ -179,6 +179,19 @@ def pull_latest_from_table_or_query(
end_date: datetime,
) -> RetrievalJob:
"""
This method is invoked when running materialization (using the `feast materialize` or `feast
kevjumba marked this conversation as resolved.
Show resolved Hide resolved
materialize-incremental` commands, or the corresponding FeatureStore.materialize() method.
This method pulls data from the offline store, and the FeatureStore class is used to write this
data into the online store.
Args:
kevjumba marked this conversation as resolved.
Show resolved Hide resolved
config: Repo configuration object
data_source: Data source to pull all of the columns from
join_key_columns: Columns of the join keys
feature_name_columns: Columns of the feature names needed
event_timestamp_column: Timestamp column
start_date: Starting date of query
end_date: Ending date of query

Note that join_key_columns, feature_name_columns, event_timestamp_column, and created_timestamp_column
have all already been mapped to column names of the source table and those column names are the values passed
into this function.
Expand Down Expand Up @@ -210,6 +223,17 @@ def pull_all_from_table_or_query(
end_date: datetime,
) -> RetrievalJob:
"""
Returns a Retrieval Job for all join key columns, feature name columns, and the event timestamp columns that occur between the start_date and end_date.

Args:
config: Repo configuration object
data_source: Data source to pull all of the columns from
join_key_columns: Columns of the join keys
feature_name_columns: Columns of the feature names needed
event_timestamp_column: Timestamp column
start_date: Starting date of query
end_date: Ending date of query

Note that join_key_columns, feature_name_columns, event_timestamp_column, and created_timestamp_column
have all already been mapped to column names of the source table and those column names are the values passed
into this function.
Expand Down