Refactor using ld.so hardware capabilities

[leuc]

Summary

Split ggml into shared libraries for each supported CPU hardware platform, that are dynamically loaded at runtime.

Background

ld.so has a feature to load shared libraries based on the executing CPU hardware platform at runtime/load-time (not compile time).

to quote man ld.so

Some shared objects are compiled using hardware-specific instructions which do not exist on every CPU. Such objects should be installed in directories whose names define the required hardware capabilities, such as /usr/lib/sse2/. The dynamic linker checks these directories against the hardware of the machine and selects the most suitable version of a given shared object. Hardware capability directories can be cascaded to combine CPU features. The list of supported hardware capability names depends on the CPU. [...]

In addition manual loading of shared libraries could be done via dlopen() for platforms that ld.so doesn't recognize, like CUDA.

Pros

• clear cut code path: all hardware specific code goes into separated libraries, split by AVX+AVX2, AVX512, ARM/NEON, CUDA, AMD, Intel GPUs, etc.
• less #ifdefhell
• easier hardware specific loops and batch operations

Cons

• Major refactoring required
• Changes to the C API?

This ld.so feature came to mind, given current efforts with CUDA and the creation of ggml-cuda.cu. Why not split up the code more cleanly with a generalized hardware based structure?

[sw]

Good thinking, but I fear this is going to be a headache on Windows.

I thought about defining the functions in ggml.c which would get SIMD optimizations as __attribute__((weak)), then you could link in ggml-simd.o which contains the actual optimized functions which override the weak ones in ggml.c. The advantage of this is that it doesn't require a huge refactoring. But I gave up on this idea when I realized that MSVC doesn't seem to support this at all.

[leuc]

On Windows it could build with https://github.com/dlfcn-win32/dlfcn-win32 for a posix dlopen() ?

[SlyEcho]

Windows has .dll files and you can dynamically load them at runtime. It shouldn't be any kind of problem at all.