This is a Multimodal Search demo built with Weaviate, ImageBind/ Vertex AI and Next.js

👨🏾‍🍳 Blogpost - Building Multimodal AI in TypeScript

🐥 Getting Started

First, clone the project with the command below

git clone https://github.com/weaviate-tutorials/next-multimodal-search-demo

The repository lets you do three things

1. Run the Next.js Web App.
2. Run an instance of Weaviate OR create a Weaviate Sandbox
3. Import images, audio and videos into your Weaviate database.

🏗️ Running Weaviate (using ImageBind)

Note that the first time you run it, Docker will download ~4.8GB multi2vec-bind Weaviate module, which contains the ImageBind model.

To start the Weaviate instance, run the following command, which will use the docker-compose.yml file.

docker compose up -d

🏗️ Create a Weaviate Instance (Using Vertex AI)

Create a Weaviate instance on Weaviate Cloud Services as described in this guide

🦿 Create a .env file and add the following keys

• your Google Vertex API key as GOOGLE_API_KEY (you can get this in your Vertex AI settings)
• your Weaviate API key as WEAVIATE_ADMIN_KEY (you can get this in your Weaviate dashboard under sandbox details)
• your Weaviate host URL as WEAVIATE_HOST_URL (you can get this in your Weaviate dashboard under sandbox details)

📩 Importing Data

Before you can import data, add any files to their respective media type in the public/ folder.

With your data in the right folder, run yarn install to install all project dependencies and to import your data into Weaviate and initialize a collection, run:

yarn run import

this may take a minute or two.

🚀 Running your Next.js Application.

Make sure you have your Weaviate instance running with data imported before starting your Next.js Web App.

To run the Web App

yarn dev

... and you can search away!!

📚 Resources

Learn more about multimodal applications

• Multimodal Retrieval Augmented Generation(RAG)
• Multimodal Embedding Models

🤷🏾‍♂️ Troubleshooting

• Check out the Weaviate Docs
• Open an Issue

Some credit goes to Steven for his Spirals template