Improve recall speed

[miltonhultgren]

In #173710 and #172164 we extended the capabilities of recall by making it include data from Search Connectors and asking the LLM to score the documents.
From our experience, this gives better relevance to the Knowledge base hits included but comes at the cost of performance (time it takes to perform a recall).

We would like to see if we can strike a good balance between time and accuracy. The idea is that perhaps by asking the LLM to evaluate the documents in parallel we can make that part faster, there may be other things we could do like ask the LLM to only report back the IDs above a certain score to minimize the tokens being generated.

UPDATE: We should also try to ask the LLM to return only the index of the document, not it's actual ID, to see if we can save on a little bit of tokens that way.

AC

• There exists some benchmarking information about how long each step in a recall takes (ELSER search on knowledge base, ELSER search on search-* + field_caps call, passing top 20 hits to LLM, LLM generating it's scores and output)
• Some amount of parallelisation is used when asking the LLM to score hits (between one request per hit and splitting the list of hits in two)
• The accuracy of the scoring should remain close to what it is today

[elasticmachine]

Pinging @elastic/obs-knowledge-team (Team:obs-knowledge)

[miltonhultgren]

Steps to set up search connector

• set up a search connector using a conf file and Docker: https://www.elastic.co/guide/en/enterprise-search/current/connectors-github.html. It doesn’t need to be running yet.
• in Kibana, navigate to Search > Overview
• Scroll to Connectors and click Create a Connector
• Select a type
• Give the index a name and click Create Index
• Click generate API key
• click on the copy button in the Deploy Connector section to copy the settings to clipboard
• in a directory create a file called config.yml
• Paste in the contents and save it. If you're running a CCS setup, change the elasticsearch.host value from localhost:9200 to whatever the url is of your CCS deployment
• If you haven’t before, create the elastic docker network: docker network create elastic
• Then start the connector:

docker run \
-v ~/<directory_where_you_put_the_config_file>:/config \
--network "elastic" \
--tty \
--rm \
docker.elastic.co/enterprise-search/elastic-connectors:8.13.0.0-SNAPSHOT \
/app/bin/elastic-ingest \
-c /config/config.yml

• Go back to Kibana. In section 4 the connector should show up.

• Provided you chose Github as a connector type, fill in the form:

• Github Cloud

• fill in gh token

• repo type: org

• organisation name: elastic

• list of repositories: a repo which has .md files, i.e.:elastic/observability-aiops

• Save configuration

• Before you hit "Sync", make sure ELSER is enabled: click the Pipelines tab, scroll down to Machine Learning Inference Pipelines and click "Deploy". Make sure it's running (sometimes it says its only deployed and it also needs to be started - well, obviously). When it's deployed and started, run a full sync.

[dgieselaar]

Done in #176428