-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support for shardSize in google_vertex_ai_index #14935
Add support for shardSize in google_vertex_ai_index #14935
Comments
@i2sarroyo Hello, I was not aware that shard_size is available when creating a PR for google_vertex_ai_index: GoogleCloudPlatform/magic-modules#6728. Indeed, the shard_size appears in the example json below. Detailsfrom https://cloud.google.com/vertex-ai/docs/matching-engine/create-manage-index#index-metadata-fileHowever, it's not explained in the yaml specification and this official doc either. Based on the response and the following doc, I believe the available values are those three values. And the default is "SHARD_SIZE_MEDIUM". Have you tried either small or large shard size before?
Detailsfrom https://cloud.google.com/vertex-ai/docs/matching-engine/create-manage-index#create-index@rileykarson I'd appreciate it if you could tell us how to report the missing explanation to GCP. |
I could confirm Details-----------------------------------------------------
2023/07/08 12:42:41 [DEBUG] Google API Response Details:
---[ RESPONSE ]--------------------------------------
HTTP/2.0 200 OK
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Cache-Control: private
Content-Type: application/json; charset=UTF-8
Cross-Origin-Opener-Policy-Report-Only: same-origin; report-to="gfe-default_product_name"
Date: Sat, 08 Jul 2023 19:42:41 GMT
Report-To: {"group":"gfe-default_product_name","max_age":2592000,"endpoints":[{"url":"https://csp.withgoogle.com/csp/report-to/default_product_name"}]}
Server: ESF
Vary: Origin
Vary: X-Origin
Vary: Referer
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 0
{
"name": "projects/xxxxx/locations/us-central1/indexes/5220296490726981632",
"displayName": "tf-test-test-indexq0ihszz7z4",
"description": "index for test",
"metadataSchemaUri": "gs://google-cloud-aiplatform/schema/matchingengine/metadata/nearest_neighbor_search_1.0.0.yaml",
"metadata": {
"config": {
"dimensions": 2,
"distanceMeasureType": "COSINE_DISTANCE",
"featureNormType": "UNIT_L2_NORM",
"algorithmConfig": {
"bruteForceConfig": {}
},
"shardSize": "SHARD_SIZE_LARGE"
}
},
"etag": "AMEw9yORUpqz6mKEHUPrDJa3NspQbVBCfuT2TeN46W3UDCAB6VCohJ3q_xmnzl3nSIip",
"labels": {
"foo": "bar"
},
"createTime": "2023-07-08T19:04:32.204099Z",
"updateTime": "2023-07-08T19:42:35.799633Z",
"indexStats": {
"vectorsCount": "2",
"shardsCount": 1
},
"indexUpdateMethod": "STREAM_UPDATE"
} -----------------------------------------------------
2023/07/08 11:18:26 [DEBUG] Google API Response Details:
---[ RESPONSE ]--------------------------------------
HTTP/2.0 200 OK
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Cache-Control: private
Content-Type: application/json; charset=UTF-8
Cross-Origin-Opener-Policy-Report-Only: same-origin; report-to="gfe-default_product_name"
Date: Sat, 08 Jul 2023 18:18:26 GMT
Report-To: {"group":"gfe-default_product_name","max_age":2592000,"endpoints":[{"url":"https://csp.withgoogle.com/csp/report-to/default_product_name"}]}
Server: ESF
Vary: Origin
Vary: X-Origin
Vary: Referer
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 0
{
"name": "projects/xxxxxx/locations/us-central1/indexes/3721160764766027776",
"displayName": "tf-test-test-indexz199pyw7xj",
"description": "index for test",
"metadataSchemaUri": "gs://google-cloud-aiplatform/schema/matchingengine/metadata/nearest_neighbor_search_1.0.0.yaml",
"metadata": {
"config": {
"dimensions": 2,
"approximateNeighborsCount": 150,
"distanceMeasureType": "DOT_PRODUCT_DISTANCE",
"featureNormType": "NONE",
"algorithmConfig": {
"treeAhConfig": {
"leafNodeEmbeddingCount": "500",
"leafNodesToSearchPercent": 7
}
},
"shardSize": "SHARD_SIZE_SMALL"
}
},
"etag": "AMEw9yO_A6lg7upig0EnvkMxb96_Bf8a0dH37JT1a3IOKAXmmoRBEVKUidJUD7mVM2k=",
"labels": {
"foo": "bar"
},
"createTime": "2023-07-08T17:38:55.912414Z",
"updateTime": "2023-07-08T18:18:23.036365Z",
"indexStats": {
"vectorsCount": "2",
"shardsCount": 1
},
"indexUpdateMethod": "BATCH_UPDATE"
} |
Great news @shotarok. Thank you so much for the effort |
A "report an issue" button on cloud.google.com or the product's public issue tracker, such as this template for vertex is probably the best way to surface that kind of feedback with their product team. |
Thank you! I created this issue: https://issuetracker.google.com/issues/290565133 |
I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. |
Community Note
Description
The API supports the definition of the index deployed instances with the shardSize field in metadata config but the actual terraform config does not. By default it creates the index with the option SHARD_SIZE_MEDIUM which tenfold the memory of the minimun possible VM created for it and it impact the costs.
New or Affected Resource(s)
google_vertex_ai_index
Potential Terraform Configuration
References
The text was updated successfully, but these errors were encountered: