[BUG] Retreive knn_vector value raising an error. What's the alternative?

[igaloly]

What is the bug?
When I try to retrieve a value from a knn_vector, I get this error: knn vector does not support this operation.

I do not want to use _source to get the vector, because it's slow.
What are my options to save vector with the ability to retrieve it values?

Mapping:

{
  "test-1": {
    "mappings": {
      "properties": {
        "vector": {
          "type": "knn_vector",
          "dimension": 768
        }
      }
    }
  }
}

Query:

{
  "size": 0,
  "aggs": {
    "vector_iter": {
      "scripted_metric": {
        ...
        "map_script": "
            for(int i = 0; i < doc['vector'].length; i++) {
                state.sumVector[i] = state.sumVector[i] + doc['vector'][i]; <---- Here's the problem
            }
        ",
        ...
        "params": {
            "vectorLength": 768
        }
      }
    }
  }
}

How can one reproduce the bug?

1. Create an index
2. Post some documents with knn_field
3. Try to reach field value in some index in painless

What is the expected behavior?
To get the value of the specific index

What is your host/environment?

• Amazon OpenSearch Service
• OpenSearch v1.2

[VijayanB]

@igaloly You can use value member to get float [], like below

GET /myindex1/_search?pretty
{
  "size": 3,
  "query": {
    "script_score": {
      "query" : {
        "match_all": {}
       },
      "script": {
        "source" : """
           float sum = 0;
           float []v = doc['vector'].value;
           for (int i=0; i< v.length; i++) { 
             sum += v[i];
           }
           return sum;
        """
      }
    }
  }
}

[igaloly]

@VijayanB When performing this query, as you suggested, it raises the error: Illegal list shortcut value [value].

Query:

{
  "size": 0,
  "aggs": {
    "vector_iter": {
      "scripted_metric": {
        ...
        "map_script": "
            float []vector = doc['vector'].value; <---- Here's the problem
            for(int i = 0; i < vector.length; i++) {
                state.sumVector[i] = state.sumVector[i] + vector[i];
            }
        ",
        ...
        "params": {
            "vectorLength": 768
        }
      }
    }
  }
}

Also, Is it slow as params._source['vector']?

[VijayanB]

@igaloly doc values are faster if you are loading few fields from documents instead of whole document. Let me try to reproduce this issue and will get back to you soon.

[VijayanB]

@igaloly I am able to reproduce this issue and figured out that KNNScriptDocValue is only accessible inside script_score context at this moment. We will work on including additional context to make it accessible.

[igaloly]

@VijayanB Sounds good.
Do you have any (even rough) time estimate on when it will be available to Managed Opensearch users?

[VijayanB]

@igaloly We are targeting for OpenSearch 2.0.1 ( OpenSource) release. Will confirm in few days on when will this be available for managed OpenSearch users.

[igaloly]

Hi @VijayanB
Is there any update?

[VijayanB]

@igaloly Sorry for late response. We released this feature in OpenSearch 2.0.1 ( Open source). Unfortunately, i don't have any information on when will this be available for managed users at this moment.

[vamshin]

@igaloly As of today managed service is couple of months behind the OpenSearch version. Stay tuned to managed service release page

[igaloly]

@VijayanB @vamshin
Thank you very much!

[vamshin]

closing this thread. Feel free to reopen if you see issues.