Skip to content

Commit

Permalink
[completions and embeddings] Add support for Ollama (local LLMs) (#711)
Browse files Browse the repository at this point in the history
  • Loading branch information
eolivelli authored Nov 9, 2023
1 parent 5df7e67 commit b03f6d5
Show file tree
Hide file tree
Showing 17 changed files with 888 additions and 6 deletions.
1 change: 1 addition & 0 deletions examples/applications/ollama-chatbot/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
java/lib/*
39 changes: 39 additions & 0 deletions examples/applications/ollama-chatbot/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# Running your own Chat bot using Ollama.ai

This sample application shows how to build a chat bot over the content of a website.
In this case you are going to crawl the LangStream.ai documentation website.

The Chat bot will be able to help you with LangStream.

In this example we are using [HerdDB](ps://github.com/diennea/herddb) as a vector database using the JDBC driver,
but you can use any Vector databases.

As LLM we are using [Ollama](https://ollama.ai), that is a service that runs on your machine.


## Install Ollama


Follow the instructions on the Ollama.ai website to install Ollama.

Then start Ollama with the llama2 model

```
ollama run llama2
```


## Deploy the LangStream application in docker

The default docker runner starts Minio, Kafka and HerdDB, so you can run the application locally.

```
./bin/langstream docker run test -app examples/applications/ollama-chatbot -s examples/secrets/secrets.yaml
```

## Talk with the Chat bot using the CLI
Since the application opens a gateway, we can use the gateway API to send and consume messages.

```
./bin/langstream gateway chat test -cg bot-output -pg user-input -p sessionId=$(uuidgen)
```
33 changes: 33 additions & 0 deletions examples/applications/ollama-chatbot/assets.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
#
# Copyright DataStax, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

assets:
- name: "documents-table"
asset-type: "jdbc-table"
creation-mode: create-if-not-exists
config:
table-name: "documents"
datasource: "JdbcDatasource"
create-statements:
- |
CREATE TABLE documents (
filename TEXT,
chunk_id int,
num_tokens int,
lang TEXT,
text TEXT,
embeddings_vector FLOATA,
PRIMARY KEY (filename, chunk_id));
102 changes: 102 additions & 0 deletions examples/applications/ollama-chatbot/chatbot.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
#
# Copyright DataStax, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

topics:
- name: "questions-topic"
creation-mode: create-if-not-exists
- name: "answers-topic"
creation-mode: create-if-not-exists
- name: "log-topic"
creation-mode: create-if-not-exists
errors:
on-failure: "skip"
pipeline:
- name: "convert-to-structure"
type: "document-to-json"
input: "questions-topic"
configuration:
text-field: "question"
- name: "compute-embeddings"
type: "compute-ai-embeddings"
configuration:
model: "${secrets.ollama.model}"
embeddings-field: "value.question_embeddings"
text: "{{ value.question }}"
flush-interval: 0
- name: "lookup-related-documents"
type: "query-vector-db"
configuration:
datasource: "JdbcDatasource"
query: "SELECT text,embeddings_vector FROM documents ORDER BY cosine_similarity(embeddings_vector, CAST(? as FLOAT ARRAY)) DESC LIMIT 20"
fields:
- "value.question_embeddings"
output-field: "value.related_documents"
- name: "re-rank documents with MMR"
type: "re-rank"
configuration:
max: 5 # keep only the top 5 documents, because we have an hard limit on the prompt size
field: "value.related_documents"
query-text: "value.question"
query-embeddings: "value.question_embeddings"
output-field: "value.related_documents"
text-field: "record.text"
embeddings-field: "record.embeddings_vector"
algorithm: "MMR"
lambda: 0.5
k1: 1.2
b: 0.75
- name: "ai-chat-completions"
type: "ai-chat-completions"

configuration:
model: "${secrets.ollama.model}"
# on the log-topic we add a field with the answer
completion-field: "value.answer"
# we are also logging the prompt we sent to the LLM
log-field: "value.prompt"
# here we configure the streaming behavior
# as soon as the LLM answers with a chunk we send it to the answers-topic
stream-to-topic: "answers-topic"
# on the streaming answer we send the answer as whole message
# the 'value' syntax is used to refer to the whole value of the message
stream-response-completion-field: "value"
# we want to stream the answer as soon as we have 20 chunks
# in order to reduce latency for the first message the agent sends the first message
# with 1 chunk, then with 2 chunks....up to the min-chunks-per-message value
# eventually we want to send bigger messages to reduce the overhead of each message on the topic
min-chunks-per-message: 20
messages:
- role: system
content: |
An user is going to perform a questions, The documents below may help you in answering to their questions.
Please try to leverage them in your answer as much as possible.
Take into consideration that the user is always asking questions about the LangStream project.
If you provide code or YAML snippets, please explicitly state that they are examples.
Do not provide information that is not related to the LangStream project.
Documents:
{{# value.related_documents}}
{{ text}}
{{/ value.related_documents}}
- role: user
content: "{{ value.question}}"
- name: "cleanup-response"
type: "drop-fields"
output: "log-topic"
configuration:
fields:
- "question_embeddings"
- "related_documents"
36 changes: 36 additions & 0 deletions examples/applications/ollama-chatbot/configuration.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
#
#
# Copyright DataStax, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

configuration:
resources:
- type: "datasource"
name: "JdbcDatasource"
configuration:
service: "jdbc"
driverClass: "herddb.jdbc.Driver"
url: "${secrets.herddb.url}"
user: "${secrets.herddb.user}"
password: "${secrets.herddb.password}"
- type: "ollama-configuration"
name: "ollama"
configuration:
url: "${secrets.ollama.url}"
dependencies:
- name: "HerdDB.org JDBC Driver"
url: "https://repo1.maven.org/maven2/org/herddb/herddb-jdbc/0.28.0/herddb-jdbc-0.28.0-thin.jar"
sha512sum: "d8ea8fbb12eada8f860ed660cbc63d66659ab3506bc165c85c420889aa8a1dac53dab7906ef61c4415a038c5a034f0d75900543dd0013bdae50feafd46f51c8e"
type: "java-library"
115 changes: 115 additions & 0 deletions examples/applications/ollama-chatbot/crawler.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,115 @@
#
# Copyright DataStax, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

name: "Crawl a website"
resources:
size: 2
pipeline:
- name: "Crawl the WebSite"
type: "webcrawler-source"
configuration:
seed-urls: ["https://docs.langstream.ai/"]
allowed-domains: ["https://docs.langstream.ai"]
forbidden-paths: []
min-time-between-requests: 500
max-error-count: 5
max-urls: 1000
max-depth: 50
handle-robots-file: true
scan-html-documents: true
state-storage: disk
- name: "Extract text"
type: "text-extractor"
- name: "Normalise text"
type: "text-normaliser"
configuration:
make-lowercase: true
trim-spaces: true
- name: "Detect language"
type: "language-detector"
configuration:
allowedLanguages: ["en", "fr"]
property: "language"
- name: "Split into chunks"
type: "text-splitter"
configuration:
splitter_type: "RecursiveCharacterTextSplitter"
chunk_size: 400
separators: ["\n\n", "\n", " ", ""]
keep_separator: false
chunk_overlap: 100
length_function: "cl100k_base"
- name: "Convert to structured data"
type: "document-to-json"
configuration:
text-field: text
copy-properties: true
- name: "prepare-structure"
type: "compute"
configuration:
fields:
- name: "value.filename"
expression: "properties.url"
type: STRING
- name: "value.chunk_id"
expression: "properties.chunk_id"
type: STRING
- name: "value.language"
expression: "properties.language"
type: STRING
- name: "value.chunk_num_tokens"
expression: "properties.chunk_num_tokens"
type: STRING
- name: "compute-embeddings"
id: "step1"
type: "compute-ai-embeddings"
configuration:
model: "${secrets.ollama.model}"
embeddings-field: "value.embeddings_vector"
text: "{{ value.text }}"
batch-size: 10
flush-interval: 500
- name: "Delete stale chunks"
type: "query"
configuration:
datasource: "JdbcDatasource"
when: "fn:toInt(properties.text_num_chunks) == (fn:toInt(properties.chunk_id) + 1)"
mode: "execute"
query: "DELETE FROM documents WHERE filename = ? AND chunk_id > ?"
output-field: "value.delete-results"
fields:
- "value.filename"
- "fn:toInt(value.chunk_id)"
- name: "Write"
type: "vector-db-sink"
configuration:
datasource: "JdbcDatasource"
table-name: "documents"
fields:
- name: "filename"
expression: "value.filename"
primary-key: true
- name: "chunk_id"
expression: "value.chunk_id"
primary-key: true
- name: "embeddings_vector"
expression: "fn:toListOfFloat(value.embeddings_vector)"
- name: "lang"
expression: "value.language"
- name: "text"
expression: "value.text"
- name: "num_tokens"
expression: "value.chunk_num_tokens"
43 changes: 43 additions & 0 deletions examples/applications/ollama-chatbot/gateways.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
#
#
# Copyright DataStax, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

gateways:
- id: "user-input"
type: produce
topic: "questions-topic"
parameters:
- sessionId
produceOptions:
headers:
- key: langstream-client-session-id
valueFromParameters: sessionId

- id: "bot-output"
type: consume
topic: "answers-topic"
parameters:
- sessionId
consumeOptions:
filters:
headers:
- key: langstream-client-session-id
valueFromParameters: sessionId


- id: "llm-debug"
type: consume
topic: "log-topic"
5 changes: 5 additions & 0 deletions examples/secrets/secrets.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -144,6 +144,11 @@ secrets:
secret-key: "${BEDROCK_SECRET_KEY}"
region: "${REGION:-us-east-1}"
completions-model: "${BEDROCK_COMPLETIONS_MODEL}"
- name: ollama
id: ollama
data:
url: "${OLLAMA_URL:-http://host.docker.internal:11434}"
model: "${OLLAMA_MODEL:-mistral}"
- name: camel-github-source
id: camel-github-source
data:
Expand Down
Loading

0 comments on commit b03f6d5

Please sign in to comment.