docs(blog): a journey through OSS LLMs and data

[lostmygithubaccount]

Please describe the issue

write down my adventure through open source LLMs, generating structured data, using them with Ibis, and a bunch of shenanigans along the way

overview of LLM frameworks:

• llama.cpp (including grammars)
• ollama (llama.cpp wrapper)
• mlx (apple only)
• candle (Rust)
• ...a bunch more I won't touch on probably

overview of OSS LLM landscape:

• Mistral
• Mixtral
• Phi-2
• ...and a bunch more I might touch on

overview of structured output frameworks:

• grammars in llama.cpp
• Marvin
• instructor
• outlines
• ...a bunch more I won't touch on probably

issues:

• formats (gguf, npz or whatever MLX uses, safetensors, etc)
• converting between formats (sometimes working convert.py scripts)
• hard-to-debug hanging
• using mac GPU
• slow
• bad accuracy

Code of Conduct

• I agree to follow this project's Code of Conduct

[lostmygithubaccount]

this one is pretty close and I might finish soon-ish but closing as not planned