Make loading weights 10-100x faster

[jart]

This is a breaking change that's going to give us three benefits:

1. Your inference commands should load 100x faster
2. You may be able to safely load models 2x larger
3. You can run many concurrent inference processes

This was accomplished by changing the file format so we can mmap()
weights directly into memory without having to read() or copy them
thereby ensuring the kernel can make its file cache pages directly
accessible to our inference processes; and secondly, that the file
cache pages are much less likely to get evicted (which would force
loads to hit disk) because they're no longer competing with memory
pages that were needlessly created by gigabytes of standard i/o.

The new file format supports single-file models like LLaMA 7b, and
it also supports multi-file models like LLaMA 13B. Our Python tool
now merges the foo.1, foo.2, etc. files back into a single file so
that the C++ code which maps it doesn't need to reshape data every
time. That's made llama.cpp so much simpler. Much of its load code
has now been deleted.

Furthermore, this change ensures that tensors are aligned properly
on a 32-byte boundary. That opens the door to seeing if we can get
additional performance gains on some microprocessors, by using ops
that require memory alignment.

Lastly note that both POSIX and the Windows platform are supported

The issue this PR solves is #91

This PR was written in collaboration with @slaren. This PR is also rebased on
PR #586 so please do not squash merge! Use either merge or rebase.

[luminalle]

Should the other converters also be rewritten to handle this new format?

[jart]

Yes indeed. I just fixed the quantize program. Now I'm hunting down all the tests.

[jart]

All tests look green except for a CMake test. For example: https://github.com/ggerganov/llama.cpp/actions/runs/4559537462/jobs/8043597142?pr=613 I'm stumped on this error. I can't figure out where the file models/ggml-vocab.bin comes from. Does anyone know? Could it be a stale cache?

[FNsi]

All tests look green except for a CMake test. For example: https://github.com/ggerganov/llama.cpp/actions/runs/4559537462/jobs/8043597142?pr=613 I'm stumped on this error. I can't figure out where the file models/ggml-vocab.bin comes from. Does anyone know? Could it be a stale cache?

#355 mentioned "Added ./models/ggml-vocab.bin containing just LLaMA vocab data (used for tests)"

[bakkot]

Nit: why change the magic rather than the version? I assumed the plan was to keep the magic constant forever. If you bump the version instead, old executables will recognize new model files and give a more useful error message. And it's nice to distinguish between "this is definitely a model file for this project, but it's the wrong version" vs "this is some random junk we don't know anything about".

(This PR is a very neat bit of engineering; please don't let my nitpick distract from that.)

[Green-Sky]

not a nitpick but a real change request :)

[Green-Sky]

(nvm)

[ggerganov]

@jart
The models/ggml-vocab.bin is generated by convert-pth-to-ggml.py by providing an extra arg.

I had the expectation that mmap support would be much more intrusive, but in fact it turned out to be very compact. llama.cpp is much more simpler now. Good stuff

Regarding the version comment - yes, the plan was to bump versions and no the magic. But I'm ok to change the magic to commemorate the significance of this update. In fact, maybe we can make this a thing and everybody who makes a significant contribution to the project will get their initials appended to the version. What do you think? 😄

Let me play with this tonight before merging. We have to make special care that all the other ggml model files floating around (Alpaca, GPT4All, Chinese LLaMA, etc.) have a nice way to convert to this new format and update the instructions in the README.

Also, maybe some synchronisation with #545 would be needed

[jart]

File updated. A lot more tests are green now. No idea what's up with the sanitizer.

I thought so too! I too was pleasantly surprised by how well it worked out. Glad we took a few weeks to think.

I'm honored to hear you say that. I can roundup the magic to 64 bytes if you like, so there's room to hand out kudos without breaking backwards compatibility in the future. Since my initials also act as a stamp of approval, I'm going to be sending a follow-up change after this, that'll harden the loading code, so that folks will be able to trade model files for this format on HuggingFace with maximum safety and confidence.

#545 is an ambitious unification. I've done my best to comment my changes to make the merge less painful for the author. I've sought to update the other scripts too, but don't know how to run them. One thing you could also consider with this project is having a contrib/ folder, where folks can merge as much of their own stuff as they want, under the expectation that the ones who need it are the ones who maintain it.

[mqy]

1. Is it more safe to use mmap64 for 4GB+ files?
2. It seems mmap, mmap64 and MapViewOfFile support mapping from given offset. Is it possible to map from header_len (as offset)? If we can do this, no need to align model file, right?

[jart]

1. The right thing to do on 32-bit platforms is to have your build system define -D_FILE_OFFSET_BITS=64 which will cause your system header files to automatically #define mmap mmap64
2. File offsets passed to mmap() need to be page size aligned, so I don't think so.

[pgoodman]

@jart Is it possible to ensure the file size is a multiple of the hugepage size (e.g. using ftruncate), to benefit from fewer TLB lookups when the model data is accessed? (corresponding mmap hints or other system-specific APIs, e.g. needed for macOS, might need to be used)

[jart]

It doesn't matter with mmap() if the file length isn't page size aligned, even with smaller pages. You should be good to go if you modify the mmap() code in llama.cpp by hand and actually manage to get huge pages to work without nuking your machine :-)

[pgoodman]

TIL!

[jart]

@ggerganov This change now includes a migration tool named migrate-ggml-2023-03-30-pr613.py. This will ensure that users of the old GGML file format who've deleted the original .pth files, will be able to convert their ggml+ggmf files to the new ggml+ggjt format. Please take a look.

[x02Sylvie]

Having issue migrating alpaca model ggml-alpaca-13b-q4.bin, python script seems to think that model has two n_parts rather than one, adding --n_parts argument to conversion script to manually specify --n_parts 1 just like when running alpaca models on llama.cpp might resolve the issue?

[jart]

@x02Sylvie I don't have access to the Alpaca model. Could send a pull request fixing that after this gets merged?

[x02Sylvie]

I don't really know python, so I'd rather leave pull request to someone smarter than me,

I did however manage to get alpaca 13b model converted by manually setting n_parts to 1 in .py conversion script . I'm unsure if it's proper place to set n_parts though

def get_n_parts(dim):
    mappings = {4096: 1, 5120: 2, 6656: 4, 8192: 8}

    n_parts = mappings.get(dim)

    if n_parts is None:
        print(f"Invalid dim: {dim}")
        sys.exit(1)
    print(f"n_parts = {n_parts}\n")
    return n_parts

to

def get_n_parts(dim):
    mappings = {4096: 1, 5120: 2, 6656: 4, 8192: 8}

    n_parts = 1

    if n_parts is None:
        print(f"Invalid dim: {dim}")
        sys.exit(1)
    print(f"n_parts = {n_parts}\n")
    return n_parts

Model does work however after conversion

[gaceladri]

Hello,

I can not load the gtp4all after converting it to the new ggml format using your script:
python3 convert-gpt4all-to-ggml.py models/gpt4all/gpt4all-lora-quantized.bin ./models/tokenizer.model

I have opened a new issue probably related to this: #655 (comment)

[gaceladri]

I could run it with the previous version https://github.com/ggerganov/llama.cpp/tree/master-ed3c680

Hello,

I can not load the gtp4all after converting it to the new ggml format using your script: python3 convert-gpt4all-to-ggml.py models/gpt4all/gpt4all-lora-quantized.bin ./models/tokenizer.model

I have opened a new issue probably related to this: #655 (comment)

[rabidcopy]

Hello,

I can not load the gtp4all after converting it to the new ggml format using your script: python3 convert-gpt4all-to-ggml.py models/gpt4all/gpt4all-lora-quantized.bin ./models/tokenizer.model

I have opened a new issue probably related to this: #655 (comment)

You need to also run the resulting file through migrate-ggml-2023-03-30-pr613.py as well.

gpt4all weights -> convert-gpt4all-to-ggml.py -> converted gpt4all weights -> migrate-ggml-2023-03-30-pr613.py -> gpt4all weights compatible with the latest version of llama.cpp

[gaceladri]

It worked. Thank you for your fast response!

[asklar]

great work @jart and @slaren ! <3