Skip to content

Commit

Permalink
docs: Improve quickstart to include push features, batch scoring, odfv (
Browse files Browse the repository at this point in the history
#3034)

* docs: Improve quickstart + local template to include push features, on demand features, and batch scoring

Signed-off-by: Danny Chiao <danny@tecton.ai>
  • Loading branch information
adchia authored Aug 18, 2022
1 parent d674a95 commit 83c5efb
Show file tree
Hide file tree
Showing 81 changed files with 2,729 additions and 1,193 deletions.
2 changes: 2 additions & 0 deletions docs/getting-started/concepts/feature-retrieval.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ Each of these retrieval mechanisms accept:
* some way of specifying entities (to fetch features for)
* some way to specify the features to fetch (either via [feature services](feature-retrieval.md#feature-services), which group features needed for a model version, or [feature references](feature-retrieval.md#feature-references))

Before beginning, you need to instantiate a local `FeatureStore` object that knows how to parse the registry (see [more details](https://docs.feast.dev/getting-started/concepts/registry))

<details>

<summary>How to: generate training data</summary>
Expand Down
51 changes: 45 additions & 6 deletions docs/getting-started/concepts/registry.md
Original file line number Diff line number Diff line change
@@ -1,30 +1,69 @@
# Registry

Feast uses a registry to store all applied Feast objects (e.g. Feature views, entities, etc). The registry exposes methods to apply, list, retrieve and delete these objects, and is an abstraction with multiple implementations.
Feast uses a registry to store all applied Feast objects (e.g. Feature views, entities, etc). The registry exposes
methods to apply, list, retrieve and delete these objects, and is an abstraction with multiple implementations.

### Options for registry implementations

By default, the registry Feast uses a file-based registry implementation, which stores the protobuf representation of the registry as a serialized file. This registry file can be stored in a local file system, or in cloud storage (in, say, S3 or GCS).
#### File-based registry
By default, Feast uses a file-based registry implementation, which stores the protobuf representation of the registry as
a serialized file. This registry file can be stored in a local file system, or in cloud storage (in, say, S3 or GCS).

However, there's inherent limitations with a file-based registry, since changing a single field in the registry requires re-writing the whole registry file. With multiple concurrent writers, this presents a risk of data loss, or bottlenecks writes to the registry since all changes have to be serialized (e.g. when running materialization for multiple feature views or time ranges concurrently).
The quickstart guides that use `feast init` will use a registry on a local file system. To allow Feast to configure
a remote file registry, you need to create a GCS / S3 bucket that Feast can understand:
{% tabs %}
{% tab title="Example S3 file registry" %}
```yaml
project: feast_demo_aws
provider: aws
registry: s3://[YOUR BUCKET YOU CREATED]/registry.pb
online_store: null
offline_store:
type: file
```
{% endtab %}
{% tab title="Example GCS file registry" %}
```yaml
project: feast_demo_gcp
provider: gcp
registry: gs://[YOUR BUCKET YOU CREATED]/registry.pb
online_store: null
offline_store:
type: file
```
{% endtab %}
{% endtabs %}
However, there are inherent limitations with a file-based registry, since changing a single field in the registry
requires re-writing the whole registry file. With multiple concurrent writers, this presents a risk of data loss, or
bottlenecks writes to the registry since all changes have to be serialized (e.g. when running materialization for
multiple feature views or time ranges concurrently).
#### SQL Registry
Alternatively, a [SQL Registry](../../tutorials/using-scalable-registry.md) can be used for a more scalable registry.
This supports any SQLAlchemy compatible database as a backend. The exact schema can be seen in [sql.py](https://github.com/feast-dev/feast/blob/master/sdk/python/feast/infra/registry/sql.py)
### Updating the registry
We recommend users store their Feast feature definitions in a version controlled repository, which then via CI/CD automatically stays synced with the registry. Users will often also want multiple registries to correspond to different environments (e.g. dev vs staging vs prod), with staging and production registries with locked down write access since they can impact real user traffic. See [Running Feast in Production](../../how-to-guides/running-feast-in-production.md#1.-automatically-deploying-changes-to-your-feature-definitions) for details on how to set this up.
We recommend users store their Feast feature definitions in a version controlled repository, which then via CI/CD
automatically stays synced with the registry. Users will often also want multiple registries to correspond to
different environments (e.g. dev vs staging vs prod), with staging and production registries with locked down write
access since they can impact real user traffic. See [Running Feast in Production](../../how-to-guides/running-feast-in-production.md#1.-automatically-deploying-changes-to-your-feature-definitions) for details on how to set this up.
### Accessing the registry from clients
Users can specify the registry through a `feature_store.yaml` config file, or programmatically. We often see teams preferring the programmatic approach because it makes notebook driven development very easy:
Users can specify the registry through a `feature_store.yaml` config file, or programmatically. We often see teams
preferring the programmatic approach because it makes notebook driven development very easy:

#### Option 1: programmatically specifying the registry

```python
repo_config = RepoConfig(
registry=RegistryConfig(path="gs://feast-test-gcs-bucket/registry.pb"),
project="feast_demo_gcp",
provider="gcp",thon
provider="gcp",
offline_store="file", # Could also be the OfflineStoreConfig e.g. FileOfflineStoreConfig
online_store="null", # Could also be the OnlineStoreConfig e.g. RedisOnlineStoreConfig
)
Expand Down
Loading

0 comments on commit 83c5efb

Please sign in to comment.