Error when using opensearch or aurora workspaces and hybrid search

[massi-ang]

Description of the issue

The following error is thrown when running a chat with a Opensearch or Aurora workspace configured for Hybrid Search.

"message":"All records failed processing. 1 individual errors logged separately below.\n\nTraceback (most recent call last):\n  File \"/opt/python/aws_lambda_powertools/utilities/batch/base.py\", line 500, in _process_record\n    result = self.handler(record=data)\n             ^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/opt/python/aws_lambda_powertools/tracing/tracer.py\", line 678, in decorate\n    response = method(*args, **kwargs)\n               ^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/var/task/index.py\", line 122, in record_handler\n    handle_run(detail)\n  File \"/var/task/index.py\", line 95, in handle_run\n    response = model.run(\n               ^^^^^^^^^^\n  File \"/var/task/adapters/base/base.py\", line 163, in run\n    return self.run_with_chain(prompt, workspace_id)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/var/task/adapters/base/base.py\", line 100, in run_with_chain\n    result = conversation({\"question\": user_prompt})\n             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/opt/python/langchain/chains/base.py\", line 310, in __call__\n    raise e\n  File \"/opt/python/langchain/chains/base.py\", line 304, in __call__\n    self._call(inputs, run_manager=run_manager)\n  File \"/opt/python/langchain/chains/conversational_retrieval/base.py\", line 148, in _call\n    docs = self._get_docs(new_question, inputs, run_manager=_run_manager)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/opt/python/langchain/chains/conversational_retrieval/base.py\", line 305, in _get_docs\n    docs = self.retriever.get_relevant_documents(\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/opt/python/langchain/schema/retriever.py\", line 211, in get_relevant_documents\n    raise e\n  File \"/opt/python/langchain/schema/retriever.py\", line 204, in get_relevant_documents\n    result = self._get_relevant_documents(\n             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/opt/python/genai_core/langchain/workspace_retriever.py\", line 13, in _get_relevant_documents\n    result = genai_core.semantic_search.semantic_search(\n             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/opt/python/genai_core/semantic_search.py\", line 25, in semantic_search\n    return query_workspace_open_search(\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/opt/python/genai_core/opensearch/query.py\", line 129, in query_workspace_open_search\n    unique_items = sorted(\n                   ^^^^^^^\nTypeError: '<' not supported between instances of 'NoneType' and 'float'\n",

Probable cause

The issue is due to records only returned by keyword search not having a semantic search score.