How to debug and develop?

[qianlong0502]

Hi there,

First off, I'd like to express my admiration for this repository; it's truly a brilliant tool.

I'm curious about the development process behind this tool. Personally, when I code—particularly in Python—I tend to rely heavily on the debugger in VSCode, meticulously crafting code line by line. I'm beginning to wonder if this approach stems from my relative unfamiliarity with the program.

I'd really appreciate insights into your typical coding process. Additionally, I've been unable to find any documentation on debugging exllamav2. Could you provide some examples or guidance on this? Your assistance would be greatly appreciated.

Thanks in advance!

[turboderp]

I wish I had more time to write documentation, but things are moving way too fast.

For development I mostly use PyCharm which has a very competent Python debugger that I couldn't really live without. Stepping through functions and inspecting variables along the way is by far the best way to get familiar with any project, I find.

A lot of the CUDA parts are written separately (in various bespoke ways) before being integrated, and there isn't really any method to that. I've tried setting up CUDA debugging but haven't had much luck so far (it's really hard!) though I do have Visual Studio set up on a Windows PC which is handy as a dev environment for the non-CUDA C++ parts.

There isn't much of a process to it overall, though. I'm not that organized (:. But I usually start with some objective, say, supporting a new architecture, and then the "process" is along the lines of:

• look through the (HF Transformers) reference implementation first of all
• set up an inference example and a little environment where I can run that in a debugger
• step through a whole forward pass to get a sense of what's going on
• get the basics up and running, making sure tokenization works and the model weights will load
• start implementing features in ExLlama one by one, or layer by layer, adding whatever needs to be added until the model can complete a forward pass in FP16 precision (usually helps to run two instances of PyCharm here, and debugging the two implementations side by side)
• fight bugs and pull my hair out for some hours/days
• start working on quantization, leaning on the debugger again and stepping through the quant process layer by layer, making adjustments as needed
• do a bunch of testing to verify that everything is working
• wait for bug reports to start flooding in

I also do a lot of profiling with line-profiler, Nsight Systems and Nsight Compute. Couldn't manage without those tools, I think. But again it's not so much a process as a new investigation every time, followed by a bunch of googling and consulting with GPT4 about how to improvise a solution to each individual bottleneck.

Idk. I hope that helps. 🤷

[qianlong0502]

Thanks a lot for your insights and guidance—it's really helpful!