Python GGML model creation library

[iamlemec]

Hi folks! Just wanted to show off a new Python-based GGML model creation library I'm working on. It's called gadget and you can check it out here: iamlemec/gadget. If you've heard of Equinox from JAX land, it's kind of like a GGML version of that. You can define models that call ggml_* functions through Python bindings, as in the very simple

class LinearModel(GgmlModel):
    weight: Tensor('F32', ('input_dim', 'output_dim'))
    inputs: Tensor('F32', ('input_dim', 'batch_size'))

    def forward(self):
        ctx = self.ctx_graph
        w, x = self.tensors['weight', 'inputs']
        return ggml_mul_mat(ctx, w, x)

and then just call them like functions with numpy or torch inputs. It takes care of tensor creation and graph (re)compilation. The library comes with working versions of both Llama (+kvcache) and BERT, which you can load directly from existing GGUFs. It also has general text generation and chat interfaces that you can use them with.

Though adding in Python as a dependency is a deal-breaker for some projects, a few advantages of gadget are

• tokenization is hell, let Huggingface handle it!
• can do rapid prototyping and experimentation without having to compile anything
• no need for round-trips to and from the GPU (could be important for embeddings?)
• easier to rapidly integrate things like novel sampling methods

As to the last point, I just added in support for the entropix (aka Shrek) sampler, and it was only a few lines of code. Anyway, comments, suggestions, and PRs are very welcome! Thanks for your time.

[ggerganov]

The link does not open - maybe the repo is private?

[iamlemec]

Ah, you're right! Should be visible now. Thanks.