Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for shardSize in google_vertex_ai_index #14935

Assignees
Milestone

Comments

@i2sarroyo
Copy link

i2sarroyo commented Jun 20, 2023

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment. If the issue is assigned to the "modular-magician" user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If the issue is assigned to a user, that user is claiming responsibility for the issue. If the issue is assigned to "hashibot", a community member has claimed the issue already.

Description

The API supports the definition of the index deployed instances with the shardSize field in metadata config but the actual terraform config does not. By default it creates the index with the option SHARD_SIZE_MEDIUM which tenfold the memory of the minimun possible VM created for it and it impact the costs.

New or Affected Resource(s)

google_vertex_ai_index

Potential Terraform Configuration

resource "google_vertex_ai_index" "index" {
  region   = "us-central1"
  display_name = "test-index"
  description = "index for test"
  metadata {
    contents_delta_uri = "gs://${google_storage_bucket.bucket.name}/contents"
    config {
      dimensions = 2
      approximate_neighbors_count = 150
      distance_measure_type = "DOT_PRODUCT_DISTANCE"
      shard_size = "SHARD_SIZE_MEDIUM"
      algorithm_config {
        tree_ah_config {
          leaf_node_embedding_count = 500
          leaf_nodes_to_search_percent = 7
        }
      }
    }
  }
}

References

@shotarok
Copy link
Contributor

shotarok commented Jul 7, 2023

@i2sarroyo Hello, I was not aware that shard_size is available when creating a PR for google_vertex_ai_index: GoogleCloudPlatform/magic-modules#6728.

Indeed, the shard_size appears in the example json below.

Details image from https://cloud.google.com/vertex-ai/docs/matching-engine/create-manage-index#index-metadata-file

However, it's not explained in the yaml specification and this official doc either.

Based on the response and the following doc, I believe the available values are those three values. And the default is "SHARD_SIZE_MEDIUM". Have you tried either small or large shard size before?

  • SHARD_SIZE_SMALL
  • SHARD_SIZE_MEDIUM
  • SHARD_SIZE_LARGE
Details image from https://cloud.google.com/vertex-ai/docs/matching-engine/create-manage-index#create-index

@rileykarson I'd appreciate it if you could tell us how to report the missing explanation to GCP.

@shotarok
Copy link
Contributor

shotarok commented Jul 9, 2023

I could confirm SHARD_SIZE_SMALL and SHARD_SIZE_LARGE were available by running acceptance tests.

Details
-----------------------------------------------------
2023/07/08 12:42:41 [DEBUG] Google API Response Details:
---[ RESPONSE ]--------------------------------------
HTTP/2.0 200 OK
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Cache-Control: private
Content-Type: application/json; charset=UTF-8
Cross-Origin-Opener-Policy-Report-Only: same-origin; report-to="gfe-default_product_name"
Date: Sat, 08 Jul 2023 19:42:41 GMT
Report-To: {"group":"gfe-default_product_name","max_age":2592000,"endpoints":[{"url":"https://csp.withgoogle.com/csp/report-to/default_product_name"}]}
Server: ESF
Vary: Origin
Vary: X-Origin
Vary: Referer
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 0

{
  "name": "projects/xxxxx/locations/us-central1/indexes/5220296490726981632",
  "displayName": "tf-test-test-indexq0ihszz7z4",
  "description": "index for test",
  "metadataSchemaUri": "gs://google-cloud-aiplatform/schema/matchingengine/metadata/nearest_neighbor_search_1.0.0.yaml",
  "metadata": {
    "config": {
      "dimensions": 2,
      "distanceMeasureType": "COSINE_DISTANCE",
      "featureNormType": "UNIT_L2_NORM",
      "algorithmConfig": {
        "bruteForceConfig": {}
      },
      "shardSize": "SHARD_SIZE_LARGE"
    }
  },
  "etag": "AMEw9yORUpqz6mKEHUPrDJa3NspQbVBCfuT2TeN46W3UDCAB6VCohJ3q_xmnzl3nSIip",
  "labels": {
    "foo": "bar"
  },
  "createTime": "2023-07-08T19:04:32.204099Z",
  "updateTime": "2023-07-08T19:42:35.799633Z",
  "indexStats": {
    "vectorsCount": "2",
    "shardsCount": 1
  },
  "indexUpdateMethod": "STREAM_UPDATE"
}
-----------------------------------------------------
2023/07/08 11:18:26 [DEBUG] Google API Response Details:
---[ RESPONSE ]--------------------------------------
HTTP/2.0 200 OK
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Cache-Control: private
Content-Type: application/json; charset=UTF-8
Cross-Origin-Opener-Policy-Report-Only: same-origin; report-to="gfe-default_product_name"
Date: Sat, 08 Jul 2023 18:18:26 GMT
Report-To: {"group":"gfe-default_product_name","max_age":2592000,"endpoints":[{"url":"https://csp.withgoogle.com/csp/report-to/default_product_name"}]}
Server: ESF
Vary: Origin
Vary: X-Origin
Vary: Referer
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 0

{
  "name": "projects/xxxxxx/locations/us-central1/indexes/3721160764766027776",
  "displayName": "tf-test-test-indexz199pyw7xj",
  "description": "index for test",
  "metadataSchemaUri": "gs://google-cloud-aiplatform/schema/matchingengine/metadata/nearest_neighbor_search_1.0.0.yaml",
  "metadata": {
    "config": {
      "dimensions": 2,
      "approximateNeighborsCount": 150,
      "distanceMeasureType": "DOT_PRODUCT_DISTANCE",
      "featureNormType": "NONE",
      "algorithmConfig": {
        "treeAhConfig": {
          "leafNodeEmbeddingCount": "500",
          "leafNodesToSearchPercent": 7
        }
      },
      "shardSize": "SHARD_SIZE_SMALL"
    }
  },
  "etag": "AMEw9yO_A6lg7upig0EnvkMxb96_Bf8a0dH37JT1a3IOKAXmmoRBEVKUidJUD7mVM2k=",
  "labels": {
    "foo": "bar"
  },
  "createTime": "2023-07-08T17:38:55.912414Z",
  "updateTime": "2023-07-08T18:18:23.036365Z",
  "indexStats": {
    "vectorsCount": "2",
    "shardsCount": 1
  },
  "indexUpdateMethod": "BATCH_UPDATE"
}

@i2sarroyo
Copy link
Author

Great news @shotarok. Thank you so much for the effort

@rileykarson
Copy link
Collaborator

A "report an issue" button on cloud.google.com or the product's public issue tracker, such as this template for vertex is probably the best way to surface that kind of feedback with their product team.

@shotarok
Copy link
Contributor

Thank you! I created this issue: https://issuetracker.google.com/issues/290565133

@github-actions
Copy link

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Aug 11, 2023
@github-actions github-actions bot added forward/review In review; remove label to forward service/aiplatform-matching-engine labels Jan 14, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.