GitBook: [master] 33 pages modified

feast-dev · Jun 25, 2020 · ff95d68 · ff95d68
1 parent 7db6ad6
commit ff95d68
Show file tree

Hide file tree

Showing 5 changed files with 84 additions and 24 deletions.
diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md
@@ -35,7 +35,7 @@
 ## Administration
 
 * [Troubleshooting](administration/troubleshooting.md)
-* [Feast Upgrading](administration/upgrading.md)
+* [Upgrading Feast](administration/upgrading.md)
 
 ## Reference
 
@@ -47,6 +47,7 @@
   * [Go SDK](https://godoc.org/github.com/feast-dev/feast/sdk/go)
   * [Java SDK](https://javadoc.io/doc/dev.feast/feast-sdk)
   * [Python SDK](https://api.docs.feast.dev/python/)
+* [Limitations](reference/limitations.md)
 
 ## Contributing
 
@@ -55,4 +56,5 @@
 * [Style Guide](contributing/style-guide.md)
 * [Release Process](contributing/release-process.md)
 * [Adding a New Store](contributing/adding-a-new-store.md)
+* [Design Decisions](contributing/design-decisions.md)
 
diff --git a/docs/administration/upgrading.md b/docs/administration/upgrading.md
@@ -1,18 +1,16 @@
 # Upgrading Feast
 
+## Migration 0.5 -&gt; 0.6
 
-## Migration 0.5 -> 0.6
 ### Database schema
-In Release 0.6 we introduced [Flyway](https://flywaydb.org/) to handle schema migrations in PostgreSQL.
-Flyway is integrated into `core` and for now on all migrations will be run automatically on
-`core` start. It uses table `flyway_schema_history` in the same database (also created automatically) to keep track of already applied migrations.
-So no specific maintenance should be needed.
+
+In Release 0.6 we introduced [Flyway](https://flywaydb.org/) to handle schema migrations in PostgreSQL. Flyway is integrated into `core` and for now on all migrations will be run automatically on `core` start. It uses table `flyway_schema_history` in the same database \(also created automatically\) to keep track of already applied migrations. So no specific maintenance should be needed.
 
 If you already have existing deployment of feast 0.5 - Flyway will detect existing tables and omit first baseline migration.
 
-After `core` started you should have `flyway_schema_history` look
-like this
- ```
+After `core` started you should have `flyway_schema_history` look like this
+
+```text
 >> select version, description, script, checksum from flyway_schema_history
 
 version |              description                |                          script         |  checksum
@@ -24,34 +22,44 @@ version |              description                |                          scr
 In this release next major schema changes were done:
 
 * Source is not shared between FeatureSets anymore. It's changed to 1:1 relation
-and source's primary key is now auto-incremented number.
+
+  and source's primary key is now auto-incremented number.
+
 * Due to generalization of Source `sources.topics` & `sources.bootstrap_servers` columns were deprecated.
-They will be replaced with `sources.config`. Data migration handled by code when respected Source is used.
-`topics` and `bootstrap_servers` will be deleted in the next release.
-* Job (table `jobs`) is no longer connected to `Source` (table `sources`) since it uses consolidated source for optimization purposes.
-All data required by Job would be embedded in its table.
 
-New Models (tables):
-* feature_statistics
+  They will be replaced with `sources.config`. Data migration handled by code when respected Source is used.
+
+  `topics` and `bootstrap_servers` will be deleted in the next release.
+
+* Job \(table `jobs`\) is no longer connected to `Source` \(table `sources`\) since it uses consolidated source for optimization purposes.
+
+  All data required by Job would be embedded in its table.
+
+New Models \(tables\):
+
+* feature\_statistics
 
 Minor changes:
-* FeatureSet has new column version (see [proto](https://github.com/feast-dev/feast/blob/master/protos/feast/core/FeatureSet.proto) for details)
+
+* FeatureSet has new column version \(see [proto](https://github.com/feast-dev/feast/blob/master/protos/feast/core/FeatureSet.proto) for details\)
 * Connecting table `jobs_feature_sets` in many-to-many relation between jobs & feature sets
-has now `version` and `delivery_status`.
 
-## Migration 0.4 -> 0.6
+  has now `version` and `delivery_status`.
+
+## Migration 0.4 -&gt; 0.6
 
 ### Database
-For all versions earlier than 0.5 seamless migration is not feasible due to earlier breaking changes and
-creation of new database will be required.
 
-Since database will be empty - first (baseline) migration would be applied:
+For all versions earlier than 0.5 seamless migration is not feasible due to earlier breaking changes and creation of new database will be required.
+
+Since database will be empty - first \(baseline\) migration would be applied:
 
- ```
+```text
 >> select version, description, script, checksum from flyway_schema_history
 
 version |              description                |                          script         |  checksum
 --------+-----------------------------------------+-----------------------------------------+------------
  1       | Baseline                                | V1__Baseline.sql                        | 1091472110
  2       | RELEASE 0.6 Generalizing Source AND ... | V2__RELEASE_0.6_Generalizing_Source_... | 1537500232
 ```
+
diff --git a/docs/contributing/design-decisions.md b/docs/contributing/design-decisions.md
@@ -0,0 +1,8 @@
+# Design Decisions
+
+## Job Management
+
+* We currently support [optional consolidation](https://github.com/feast-dev/feast/pull/825/files#diff-11e36b0d5cfc6742aa51adf2812b664eR85) of jobs by source. This allows one job to populate many stores if all the store share a source. The upside is we need to provision less jobs. The downside is we need to managing the ingestion of data into stores independently becomes a lot more complicated. For the time being we will allow a flag to switch between both consolidation strategies, but we may remove consolidation in the future if it introduces too much complexity in the overall design.
+
+
+
diff --git a/docs/reference/limitations.md b/docs/reference/limitations.md
@@ -0,0 +1,43 @@
+# Limitations
+
+## Feast API
+
+<table>
+  <thead>
+    <tr>
+      <th style="text-align:left">Limitation</th>
+      <th style="text-align:left">Motivation</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style="text-align:left">Features names and entity names cannot overlap in feature set specifications</td>
+      <td
+      style="text-align:left">Features and entities become columns in historical stores which may cause
+        conflicts</td>
+    </tr>
+    <tr>
+      <td style="text-align:left">
+        <p>The following field names are reserved in feature sets and FeatureRow
+          messages</p>
+        <ul>
+          <li><code>event_timestamp</code>
+          </li>
+          <li><code>datetime</code>
+          </li>
+          <li><code>created_timestamp</code>
+          </li>
+          <li><code>ingestion_id</code>
+          </li>
+          <li><code>job_id</code>
+          </li>
+        </ul>
+      </td>
+      <td style="text-align:left">These key words are used for column names when persisting metadata in
+        historical stores</td>
+    </tr>
+  </tbody>
+</table>
+
+
+
diff --git a/docs/user-guide/feature-retrieval.md b/docs/user-guide/feature-retrieval.md
@@ -138,7 +138,6 @@ for feature_set in feature_sets:
 anomalies = tfdv.validate_statistics(statistics=stats, schema=schema)
 ```
 
-
 ## Online feature retrieval
 
 Online feature retrieval works in much the same way as batch retrieval, with one important distinction: Online stores only maintain the current state of features. No historical data is served.