Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Feast/IKV online store documentation #4146

Merged
merged 4 commits into from
Apr 25, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -175,6 +175,7 @@ The list below contains the functionality that contributors are planning to deve
* [x] [Bigtable](https://docs.feast.dev/reference/online-stores/bigtable)
* [x] [SQLite](https://docs.feast.dev/reference/online-stores/sqlite)
* [x] [Dragonfly](https://docs.feast.dev/reference/online-stores/dragonfly)
* [x] [IKV - Inlined Key Value Store](https://docs.feast.dev/reference/online-stores/ikv)
* [x] [Azure Cache for Redis (community plugin)](https://github.com/Azure/feast-azure)
* [x] [Postgres (contrib plugin)](https://docs.feast.dev/reference/online-stores/postgres)
* [x] [Cassandra / AstraDB (contrib plugin)](https://docs.feast.dev/reference/online-stores/cassandra)
Expand Down
1 change: 1 addition & 0 deletions docs/SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,7 @@
* [Snowflake](reference/online-stores/snowflake.md)
* [Redis](reference/online-stores/redis.md)
* [Dragonfly](reference/online-stores/dragonfly.md)
* [IKV](reference/online-stores/ikv.md)
* [Datastore](reference/online-stores/datastore.md)
* [DynamoDB](reference/online-stores/dynamodb.md)
* [Bigtable](reference/online-stores/bigtable.md)
Expand Down
4 changes: 4 additions & 0 deletions docs/reference/online-stores/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,10 @@ Please see [Online Store](../../getting-started/architecture-and-components/onli
[dragonfly.md](dragonfly.md)
{% endcontent-ref %}

{% content-ref url="ikv.md" %}
[ikv.md](ikv.md)
{% endcontent-ref %}

{% content-ref url="datastore.md" %}
[datastore.md](datastore.md)
{% endcontent-ref %}
Expand Down
73 changes: 73 additions & 0 deletions docs/reference/online-stores/ikv.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
# IKV (Inlined Key-Value Store) online store

## Description

[IKV](https://github.com/inlinedio/ikv-store) is a fully-managed embedded key-value store, primarily designed for storing ML features. Most key-value stores (think Redis or Cassandra) need a remote database cluster, whereas IKV allows you to utilize your existing application infrastructure to store data (cost efficient) and access it without any network calls (better performance).

For provisioning API keys for using it as an online-store in Feast, go to [https://inlined.io](https://inlined.io) or email onboarding[at]inlined.io

## Getting started
Make sure you have Python and `pip` installed.

Install the Feast SDK and CLI: `pip install feast`

In order to use this online store, you'll need to install the IKV extra (along with the dependency needed for the offline store of choice). E.g.
- `pip install 'feast[gcp, ikv]'`
- `pip install 'feast[snowflake, ikv]'`
- `pip install 'feast[aws, ikv]'`
- `pip install 'feast[azure, ikv]'`

You can get started by using any of the other templates (e.g. `feast init -t gcp` or `feast init -t snowflake` or `feast init -t aws`), and then swapping in IKV as the online store as seen below in the examples.

### 1. Provision an IKV store
Go to [https://inlined.io](https://inlined.io) or email onboarding[at]inlined.io

### 2. Configure

Update `my_feature_repo/feature_store.yaml` with the below contents:

{% code title="feature_store.yaml" %}
```yaml
project: my_feature_repo
registry: data/registry.db
provider: local
online_store:
type: ikv
account_id: secret
account_passkey: secret
store_name: your-store-name
mount_directory: /absolute/path/on/disk/for/ikv/embedded/index
```
{% endcode %}

After provision an IKV account/store, you should the required id, passkey and store-name.

Additionally you must specify a mount-directory - where IKV will pull/update (maintain) a copy of the index for online reads (IKV is an embedded database). It can be skipped only if you don't plan to read any data from this container. The mount directory path usually points to a location on local/remote disk.

The full set of configuration options is available in IKVOnlineStoreConfig at `sdk/python/feast/infra/online_stores/contrib/ikv_online_store/ikv.py`

## Functionality Matrix

The set of functionality supported by online stores is described in detail [here](overview.md#functionality).
Below is a matrix indicating which functionality is supported by the IKV online store.

| | IKV |
| :-------------------------------------------------------- | :---- |
| write feature values to the online store | yes |
| read feature values from the online store | yes |
| update infrastructure (e.g. tables) in the online store | yes |
| teardown infrastructure (e.g. tables) in the online store | yes |
| generate a plan of infrastructure changes | no |
| support for on-demand transforms | yes |
| readable by Python SDK | yes |
| readable by Java | no |
| readable by Go | no |
| support for entityless feature views | yes |
| support for concurrent writing to the same key | yes |
| support for ttl (time to live) at retrieval | no |
| support for deleting expired data | no |
| collocated by feature view | no |
| collocated by feature service | no |
| collocated by entity key | yes |

To compare this set of functionality against other online stores, please see the full [functionality matrix](overview.md#functionality-matrix).
38 changes: 19 additions & 19 deletions docs/reference/online-stores/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,26 +29,26 @@ See this [issue](https://github.com/feast-dev/feast/issues/2254) for a discussio
## Functionality Matrix

There are currently five core online store implementations: `SqliteOnlineStore`, `RedisOnlineStore`, `DynamoDBOnlineStore`, `SnowflakeOnlineStore`, and `DatastoreOnlineStore`.
There are several additional implementations contributed by the Feast community (`PostgreSQLOnlineStore`, `HbaseOnlineStore`, and `CassandraOnlineStore`), which are not guaranteed to be stable or to match the functionality of the core implementations.
There are several additional implementations contributed by the Feast community (`PostgreSQLOnlineStore`, `HbaseOnlineStore`, `CassandraOnlineStore` and `IKVOnlineStore`), which are not guaranteed to be stable or to match the functionality of the core implementations.
Details for each specific online store, such as how to configure it in a `feature_store.yaml`, can be found [here](README.md).

Below is a matrix indicating which online stores support what functionality.

| | Sqlite | Redis | DynamoDB | Snowflake | Datastore | Postgres | Hbase | [[Cassandra](https://cassandra.apache.org/_/index.html) / [Astra DB](https://www.datastax.com/products/datastax-astra?utm_source=feast)] |
| :-------------------------------------------------------- | :-- | :-- | :-- | :-- | :-- | :-- | :-- | :-- |
| write feature values to the online store | yes | yes | yes | yes | yes | yes | yes | yes |
| read feature values from the online store | yes | yes | yes | yes | yes | yes | yes | yes |
| update infrastructure (e.g. tables) in the online store | yes | yes | yes | yes | yes | yes | yes | yes |
| teardown infrastructure (e.g. tables) in the online store | yes | yes | yes | yes | yes | yes | yes | yes |
| generate a plan of infrastructure changes | yes | no | no | no | no | no | no | yes |
| support for on-demand transforms | yes | yes | yes | yes | yes | yes | yes | yes |
| readable by Python SDK | yes | yes | yes | yes | yes | yes | yes | yes |
| readable by Java | no | yes | no | no | no | no | no | no |
| readable by Go | yes | yes | no | no | no | no | no | no |
| support for entityless feature views | yes | yes | yes | yes | yes | yes | yes | yes |
| support for concurrent writing to the same key | no | yes | no | no | no | no | no | no |
| support for ttl (time to live) at retrieval | no | yes | no | no | no | no | no | no |
| support for deleting expired data | no | yes | no | no | no | no | no | no |
| collocated by feature view | yes | no | yes | yes | yes | yes | yes | yes |
| collocated by feature service | no | no | no | no | no | no | no | no |
| collocated by entity key | no | yes | no | no | no | no | no | no |
| | Sqlite | Redis | DynamoDB | Snowflake | Datastore | Postgres | Hbase | [[Cassandra](https://cassandra.apache.org/_/index.html) / [Astra DB](https://www.datastax.com/products/datastax-astra?utm_source=feast)] | [IKV](https://inlined.io) |
| :-------------------------------------------------------- | :-- | :-- | :-- | :-- | :-- | :-- | :-- | :-- | :-- |
| write feature values to the online store | yes | yes | yes | yes | yes | yes | yes | yes | yes |
| read feature values from the online store | yes | yes | yes | yes | yes | yes | yes | yes | yes |
| update infrastructure (e.g. tables) in the online store | yes | yes | yes | yes | yes | yes | yes | yes | yes |
| teardown infrastructure (e.g. tables) in the online store | yes | yes | yes | yes | yes | yes | yes | yes | yes |
| generate a plan of infrastructure changes | yes | no | no | no | no | no | no | yes | no |
| support for on-demand transforms | yes | yes | yes | yes | yes | yes | yes | yes | yes |
| readable by Python SDK | yes | yes | yes | yes | yes | yes | yes | yes | yes |
| readable by Java | no | yes | no | no | no | no | no | no | no |
| readable by Go | yes | yes | no | no | no | no | no | no | no |
| support for entityless feature views | yes | yes | yes | yes | yes | yes | yes | yes | yes |
| support for concurrent writing to the same key | no | yes | no | no | no | no | no | no | yes |
| support for ttl (time to live) at retrieval | no | yes | no | no | no | no | no | no | no |
| support for deleting expired data | no | yes | no | no | no | no | no | no | no |
| collocated by feature view | yes | no | yes | yes | yes | yes | yes | yes | no |
| collocated by feature service | no | no | no | no | no | no | no | no | no |
| collocated by entity key | no | yes | no | no | no | no | no | no | yes |
1 change: 1 addition & 0 deletions docs/roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ The list below contains the functionality that contributors are planning to deve
* [x] [Bigtable](https://docs.feast.dev/reference/online-stores/bigtable)
* [x] [SQLite](https://docs.feast.dev/reference/online-stores/sqlite)
* [x] [Dragonfly](https://docs.feast.dev/reference/online-stores/dragonfly)
* [x] [IKV - Inlined Key Value Store](https://docs.feast.dev/reference/online-stores/ikv)
* [x] [Azure Cache for Redis (community plugin)](https://github.com/Azure/feast-azure)
* [x] [Postgres (contrib plugin)](https://docs.feast.dev/reference/online-stores/postgres)
* [x] [Cassandra / AstraDB (contrib plugin)](https://docs.feast.dev/reference/online-stores/cassandra)
Expand Down
Loading