PyGPT4All

Official Python CPU inference for GPT4All language models based on llama.cpp and ggml

• Installation
• Tutorial
  • Model instantiation
  • Simple generation
  • Interactive Dialogue
• API reference
• License

Installation

pip install pygpt4all

Tutorial

You will need first to download the model weights, you can find and download all the supported models from here.

Model instantiation

Once the weights are downloaded, you can instantiate the models as follows:

• GPT4All model

from pygpt4all import GPT4All

model = GPT4All('path/to/ggml-gpt4all-l13b-snoozy.bin')

• GPT4All-J model

from pygpt4all import GPT4All_J

model = GPT4All_J('path/to/ggml-gpt4all-j-v1.3-groovy.bin')

Simple generation

The generate function is used to generate new tokens from the prompt given as input:

for token in model.generate("Tell me a joke ?\n"):
    print(token, end='', flush=True)

Interactive Dialogue

You can set up an interactive dialogue by simply keeping the model variable alive:

while True:
    try:
        prompt = input("You: ", flush=True)
        if prompt == '':
            continue
        print(f"AI:", end='')
        for token in model.generate(prompt):
            print(f"{token}", end='', flush=True)
        print()
    except KeyboardInterrupt:
        break

API reference

You can check the API reference documentation for more details.

License

This project is licensed under the MIT License.