-
Notifications
You must be signed in to change notification settings - Fork 1.7k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
feat(vertexai): add google_vertex_ai_index for Vertex AI Matching Eng…
…ine (#6728) * feat: add google_vertex_ai_index for Vertex AI Matching Engine * fix: increase timeouts to 60 min because 20 wasn't enough for creation * fix: change coe to make name computed instead of an input * fix: use costom flatten code to ignore_read a nested property's field * fix: add skip_import_test: true to the auto-gen test * feat: add a test with a manually updated ImportStateVerifyIgnore * Apply suggestions from code review [ci skip] Update descriptions based on the suggestions Co-authored-by: Stephen Lewis (Burrows) <stephen.r.burrows@gmail.com> * refactor: use ignore_read_extra instead of a manual test * fix: use an empty object for bruteForceConfig * feat: define additional fields to api.yaml * feat: add an example to increase test coverage * feat: deal with contentsDeltaUri as an updatable field * fix: fix the error because the cosine distance type only supports unit l2 norm type This is the error message from the endpoint: "Index with `COSINE_DISTANCE` distanceMeasureType currently only supports `UNIT_L2_NORM` featureNormType." * feat: remove approximate_neighbors_count from an example with brute_force_config approximate_neighbors_count is required if tree-AH algorithm is used. from https://cloud.google.com/vertex-ai/docs/matching-engine/configuring-indexes#brute-force-config * test: add a handwritten test for patch * fix: add update_mask: true to use the mask as a url param * refactor: put 'input: true' on the fields patch couldn't update * feat: use custom pre update code for a nested object * fix: update the handwritten test accordingly * feat: add custom flatten code for is_complete_overwrite Co-authored-by: Stephen Lewis (Burrows) <stephen.r.burrows@gmail.com>
- Loading branch information
Showing
8 changed files
with
554 additions
and
1 deletion.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
18 changes: 18 additions & 0 deletions
18
mmv1/templates/terraform/custom_flatten/vertex_ai_index_ignore_contents_delta_uri.go.erb
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| <%# The license inside this block applies to this file. | ||
| # Copyright 2022 Google Inc. | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
| -%> | ||
| func flatten<%= prefix -%><%= titlelize_property(property) -%>(v interface{}, d *schema.ResourceData, config *Config) interface{} { | ||
| // We want to ignore read on this field, but cannot because it is nested | ||
| return d.Get("metadata.0.contents_delta_uri") | ||
| } |
18 changes: 18 additions & 0 deletions
18
mmv1/templates/terraform/custom_flatten/vertex_ai_index_ignore_is_complete_overwrite.go.erb
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| <%# The license inside this block applies to this file. | ||
| # Copyright 2022 Google Inc. | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
| -%> | ||
| func flatten<%= prefix -%><%= titlelize_property(property) -%>(v interface{}, d *schema.ResourceData, config *Config) interface{} { | ||
| // We want to ignore read on this field, but cannot because it is nested | ||
| return d.Get("metadata.0.is_complete_overwrite") | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,40 @@ | ||
| resource "google_storage_bucket" "bucket" { | ||
| name = "<%= ctx[:test_env_vars]['project'] %>-<%= ctx[:vars]['bucket_name'] %>" # Every bucket name must be globally unique | ||
| location = "us-central1" | ||
| uniform_bucket_level_access = true | ||
| } | ||
|
|
||
| # The sample data comes from the following link: | ||
| # https://cloud.google.com/vertex-ai/docs/matching-engine/filtering#specify-namespaces-tokens | ||
| resource "google_storage_bucket_object" "data" { | ||
| name = "contents/data.json" | ||
| bucket = google_storage_bucket.bucket.name | ||
| content = <<EOF | ||
| {"id": "42", "embedding": [0.5, 1.0], "restricts": [{"namespace": "class", "allow": ["cat", "pet"]},{"namespace": "category", "allow": ["feline"]}]} | ||
| {"id": "43", "embedding": [0.6, 1.0], "restricts": [{"namespace": "class", "allow": ["dog", "pet"]},{"namespace": "category", "allow": ["canine"]}]} | ||
| EOF | ||
| } | ||
|
|
||
| resource "google_vertex_ai_index" "index" { | ||
| labels = { | ||
| foo = "bar" | ||
| } | ||
| region = "us-central1" | ||
| display_name = "<%= ctx[:vars]['display_name'] %>" | ||
| description = "index for test" | ||
| metadata { | ||
| contents_delta_uri = "gs://${google_storage_bucket.bucket.name}/contents" | ||
| config { | ||
| dimensions = 2 | ||
| approximate_neighbors_count = 150 | ||
| distance_measure_type = "DOT_PRODUCT_DISTANCE" | ||
| algorithm_config { | ||
| tree_ah_config { | ||
| leaf_node_embedding_count = 500 | ||
| leaf_nodes_to_search_percent = 7 | ||
| } | ||
| } | ||
| } | ||
| } | ||
| index_update_method = "BATCH_UPDATE" | ||
| } |
37 changes: 37 additions & 0 deletions
37
mmv1/templates/terraform/examples/vertex_ai_index_streaming.tf.erb
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,37 @@ | ||
| resource "google_storage_bucket" "bucket" { | ||
| name = "<%= ctx[:test_env_vars]['project'] %>-<%= ctx[:vars]['bucket_name'] %>" # Every bucket name must be globally unique | ||
| location = "us-central1" | ||
| uniform_bucket_level_access = true | ||
| } | ||
|
|
||
| # The sample data comes from the following link: | ||
| # https://cloud.google.com/vertex-ai/docs/matching-engine/filtering#specify-namespaces-tokens | ||
| resource "google_storage_bucket_object" "data" { | ||
| name = "contents/data.json" | ||
| bucket = google_storage_bucket.bucket.name | ||
| content = <<EOF | ||
| {"id": "42", "embedding": [0.5, 1.0], "restricts": [{"namespace": "class", "allow": ["cat", "pet"]},{"namespace": "category", "allow": ["feline"]}]} | ||
| {"id": "43", "embedding": [0.6, 1.0], "restricts": [{"namespace": "class", "allow": ["dog", "pet"]},{"namespace": "category", "allow": ["canine"]}]} | ||
| EOF | ||
| } | ||
|
|
||
| resource "google_vertex_ai_index" "index" { | ||
| labels = { | ||
| foo = "bar" | ||
| } | ||
| region = "us-central1" | ||
| display_name = "<%= ctx[:vars]['display_name'] %>" | ||
| description = "index for test" | ||
| metadata { | ||
| contents_delta_uri = "gs://${google_storage_bucket.bucket.name}/contents" | ||
| config { | ||
| dimensions = 2 | ||
| distance_measure_type = "COSINE_DISTANCE" | ||
| feature_norm_type = "UNIT_L2_NORM" | ||
| algorithm_config { | ||
| brute_force_config {} | ||
| } | ||
| } | ||
| } | ||
| index_update_method = "STREAM_UPDATE" | ||
| } |
22 changes: 22 additions & 0 deletions
22
mmv1/templates/terraform/pre_update/vertex_ai_index.go.erb
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| newUpdateMask := []string{} | ||
|
|
||
| if d.HasChange("metadata.0.contents_delta_uri") { | ||
| // Use the current value of isCompleteOverwrite when updating contentsDeltaUri | ||
| newUpdateMask = append(newUpdateMask, "metadata.contentsDeltaUri") | ||
| newUpdateMask = append(newUpdateMask, "metadata.isCompleteOverwrite") | ||
| } | ||
|
|
||
| for _, mask := range updateMask { | ||
| // Use granular update masks instead of 'metadata' to avoid the following error: | ||
| // 'If `contents_delta_gcs_uri` is set as part of `index.metadata`, then no other Index fields can be also updated as part of the same update call.' | ||
| if mask == "metadata" { | ||
| continue | ||
| } | ||
| newUpdateMask = append(newUpdateMask, mask) | ||
| } | ||
|
|
||
| // Refreshing updateMask after adding extra schema entries | ||
| url, err = addQueryParams(url, map[string]string{"updateMask": strings.Join(newUpdateMask, ",")}) | ||
| if err != nil { | ||
| return err | ||
| } |
Oops, something went wrong.