-
Notifications
You must be signed in to change notification settings - Fork 2.3k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
feat(community): add libsql vectorstore (#6904)
Co-authored-by: jacoblee93 <jacoblee93@gmail.com>
- Loading branch information
1 parent
37745b0
commit f9f1b7f
Showing
7 changed files
with
566 additions
and
8 deletions.
There are no files selected for viewing
164 changes: 164 additions & 0 deletions
164
docs/core_docs/docs/integrations/vectorstores/libsql.mdx
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,164 @@ | ||
| # libSQL | ||
|
|
||
| [Turso](https://turso.tech) is a SQLite-compatible database built on [libSQL](https://docs.turso.tech/libsql), the Open Contribution fork of SQLite. Vector Similiarity Search is built into Turso and libSQL as a native datatype, enabling you to store and query vectors directly in the database. | ||
|
|
||
| LangChain.js supports using a local libSQL, or remote Turso database as a vector store, and provides a simple API to interact with it. | ||
|
|
||
| This guide provides a quick overview for getting started with libSQL vector stores. For detailed documentation of all libSQL features and configurations head to the API reference. | ||
|
|
||
| ## Overview | ||
|
|
||
| ## Integration details | ||
|
|
||
| | Class | Package | JS support | Package latest | | ||
| | ------------------- | ---------------------- | ---------- | ----------------------------------------------------------------- | | ||
| | `LibSQLVectorStore` | `@langchain/community` | ✅ |  | | ||
|
|
||
| ## Setup | ||
|
|
||
| To use libSQL vector stores, you'll need to create a Turso account or set up a local SQLite database, and install the `@langchain/community` integration package. | ||
|
|
||
| This guide will also use OpenAI embeddings, which require you to install the `@langchain/openai` integration package. You can also use other supported embeddings models if you wish. | ||
|
|
||
| You can use local SQLite when working with the libSQL vector store, or use a hosted Turso Database. | ||
|
|
||
| import IntegrationInstallTooltip from "@mdx_components/integration_install_tooltip.mdx"; | ||
|
|
||
| <IntegrationInstallTooltip></IntegrationInstallTooltip> | ||
|
|
||
| ```bash npm2yarn | ||
| npm install @libsql/client @langchain/openai @langchain/community | ||
| ``` | ||
|
|
||
| Now it's time to create a database. You can create one locally, or use a hosted Turso database. | ||
|
|
||
| ### Local libSQL | ||
|
|
||
| Create a new local SQLite file and connect to the shell: | ||
|
|
||
| ```bash | ||
| sqlite3 file.db | ||
| ``` | ||
|
|
||
| ### Hosted Turso | ||
|
|
||
| Visit [sqlite.new](https://sqlite.new) to create a new database, give it a name, and create a database auth token. | ||
|
|
||
| Make sure to copy the database auth token, and the database URL, it should look something like: | ||
|
|
||
| ```text | ||
| libsql://[database-name]-[your-username].turso.io | ||
| ``` | ||
|
|
||
| ### Setup the table and index | ||
|
|
||
| Execute the following SQL command to create a new table or add the embedding column to an existing table. | ||
|
|
||
| Make sure to mopdify the following parts of the SQL: | ||
|
|
||
| - `TABLE_NAME` is the name of the table you want to create. | ||
| - `content` is used to store the `Document.pageContent` values. | ||
| - `metadata` is used to store the `Document.metadata` object. | ||
| - `EMBEDDING_COLUMN` is used to store the vector values, use the dimensions size used by the model you plan to use (1536 for OpenAI). | ||
|
|
||
| ```sql | ||
| CREATE TABLE IF NOT EXISTS TABLE_NAME ( | ||
| id INTEGER PRIMARY KEY AUTOINCREMENT, | ||
| content TEXT, | ||
| metadata TEXT, | ||
| EMBEDDING_COLUMN F32_BLOB(1536) -- 1536-dimensional f32 vector for OpenAI | ||
| ); | ||
| ``` | ||
|
|
||
| Now create an index on the `EMBEDDING_COLUMN` column: | ||
|
|
||
| ```sql | ||
| CREATE INDEX IF NOT EXISTS idx_TABLE_NAME_EMBEDDING_COLUMN ON TABLE_NAME(libsql_vector_idx(EMBEDDING_COLUMN)); | ||
| ``` | ||
|
|
||
| Make sure to replace the `TABLE_NAME` and `EMBEDDING_COLUMN` with the values you used in the previous step. | ||
|
|
||
| ## Instantiation | ||
|
|
||
| To initialize a new `LibSQL` vector store, you need to provide the database URL and Auth Token when working remotely, or by passing the filename for a local SQLite. | ||
|
|
||
| ```typescript | ||
| import { LibSQLVectorStore } from "@langchain/community/vectorstores/libsql"; | ||
| import { OpenAIEmbeddings } from "@langchain/openai"; | ||
| import { createClient } from "@libsql/client"; | ||
|
|
||
| const embeddings = new OpenAIEmbeddings({ | ||
| model: "text-embedding-3-small", | ||
| }); | ||
|
|
||
| const libsqlClient = createClient({ | ||
| url: "libsql://[database-name]-[your-username].turso.io", | ||
| authToken: "...", | ||
| }); | ||
|
|
||
| // Local instantiation | ||
| // const libsqlClient = createClient({ | ||
| // url: "file:./dev.db", | ||
| // }); | ||
|
|
||
| const vectorStore = new LibSQLVectorStore(embeddings, { | ||
| db: libsqlClient, | ||
| tableName: "TABLE_NAME", | ||
| embeddingColumn: "EMBEDDING_COLUMN", | ||
| dimensions: 1536, | ||
| }); | ||
| ``` | ||
|
|
||
| ## Manage vector store | ||
|
|
||
| ### Add items to vector store | ||
|
|
||
| ```typescript | ||
| import type { Document } from "@langchain/core/documents"; | ||
|
|
||
| const documents: Document[] = [ | ||
| { pageContent: "Hello", metadata: { topic: "greeting" } }, | ||
| { pageContent: "Bye bye", metadata: { topic: "greeting" } }, | ||
| ]; | ||
|
|
||
| await vectorStore.addDocuments(documents); | ||
| ``` | ||
|
|
||
| ### Delete items from vector store | ||
|
|
||
| ```typescript | ||
| await vectorStore.deleteDocuments({ ids: [1, 2] }); | ||
| ``` | ||
|
|
||
| ## Query vector store | ||
|
|
||
| Once you have inserted the documents, you can query the vector store. | ||
|
|
||
| ### Query directly | ||
|
|
||
| Performing a simple similarity search can be done as follows: | ||
|
|
||
| ```typescript | ||
| const resultOne = await vectorStore.similaritySearch("hola", 1); | ||
|
|
||
| for (const doc of similaritySearchResults) { | ||
| console.log(`${doc.pageContent} [${JSON.stringify(doc.metadata, null)}]`); | ||
| } | ||
| ``` | ||
|
|
||
| For similarity search with scores: | ||
|
|
||
| ```typescript | ||
| const similaritySearchWithScoreResults = | ||
| await vectorStore.similaritySearchWithScore("hola", 1); | ||
|
|
||
| for (const [doc, score] of similaritySearchWithScoreResults) { | ||
| console.log( | ||
| `${score.toFixed(3)} ${doc.pageContent} [${JSON.stringify(doc.metadata)}` | ||
| ); | ||
| } | ||
| ``` | ||
|
|
||
| ## API reference | ||
|
|
||
| For detailed documentation of all `LibSQLVectorStore` features and configurations head to the API reference. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.