Explore the AI Singapore SEA-LION model with Chainlit and Ollama

Overview

• Getting Started
• Getting Started with Docker
• Default Model
• Customisations
• Acknowledgements

Note

This project is designed for local environments. Do not run it in production.

Meet the Cast

• AI Singapore SEA-LION
  • Model: https://ollama.com/aisingapore/gemma2-9b-cpt-sea-lionv3-instruct:q4_k_m
• Chainlit
• Ollama

Getting Started

Prerequisites

• Python 3.8 or newer
• Ollama

Run the app

• Install Ollama, if it is not already installed.
• Pull the model.

  ollama pull aisingapore/gemma2-9b-cpt-sea-lionv3-instruct:q4_k_m

• In the project directory, create a virtual environment.

  python -m venv venv

• Activate the virtual environment.

  source venv/bin/activate

• Copy .env and update the values, if necessary:

  cp .env.example .env

• Install the packages.

  pip install -r requirements.txt
  

• Run the app.

  chainlit run src/main.py -w

• Navigate to http://localhost:8000 to access the chatbot.

Getting Started with Docker

Note

At the time of writing, GPU support in Docker Desktop is only available on Windows with the WSL2 backend.

Prerequisites

• Docker
  • For the default model, set the memory limit to 6GB or more.
  • If a larger model is used, or if there are other active Docker containers in the environment, increase the memory limit further to take into account their memory requirements.

Run the app with Docker

• Copy .env and update the values, if necessary:

  cp .env.example .env

• Start the services:

  docker compose up

• Pull the SEA-LION model with Ollama:

  docker compose exec ollama ollama pull aisingapore/gemma2-9b-cpt-sea-lionv3-instruct:q4_k_m

• Navigate to http://localhost:8000 to access the chatbot.

Default Model

• The default model is gemma2-9b-cpt-sea-lionv3-instruct:q4_k_m.
• If you would like to test the other models, choose the model in https://ollama.com/aisingapore/gemma2-9b-cpt-sea-lionv3-instruct.
  • Check that there is sufficient disk storage and memory. For example, gemma2-9b-cpt-sea-lionv3-instruct:q8_0 requires at least 10GB of disk storage and 12GB of available memory in Docker.
  • Pull the model with Ollama.

    docker compose exec ollama ollama pull aisingapore/gemma2-9b-cpt-sea-lionv3-instruct:q8_0

  • Update the model name in .env.

    LLM_MODEL=aisingapore/gemma2-9b-cpt-sea-lionv3-instruct:q8_0
    

Customisations

• Please feel free to fork this repo and customise it.
• Examples:
  • OAuth
  • Data Persistence
  • Integrations with LangChain or other inference servers

Acknowledgements

• Kudos to the AI Singapore Team for their good work!