Update on Protobuf // Buf research

[dave-gray101]

@mudler ,

As I mentioned a few weeks ago in #4012, I'm interested in doing a cleanup pass of our Protobuf layer. Lately, I have been investigating the Buf tool and ecosystem to see if it would be a useful asset to us.

This discussion ticket will serve as a place to store some of my findings so far - since I'd like your opinions as well.

---

.proto files themselves

Initially, I encountered the tool when looking for something capable of reviewing our existing protobuf file and getting a sense of how compliant we were with standard practices. While I've used protobuf a few times and am familiar with the dos and don'ts guide, I've somehow missed the companion document API best practices in the past!

To put it bluntly, buf lint is extremely unhappy with our backend.proto file. We violate a number of guidelines in that document.

My proposal is to migrate away from the backend.proto file to an improved "Backends v2" with a few important changes (potentially more to come?)

• Change the package name to something that properly identifies as LocalAI
• Split the RPC services to small bundles:
  • Mandatory Backend service with rpc methods to get name, check health, load model, and return a list of other capabilities
  • Actual backend functionality is split to its own service, such as TTS being seperated from ImageGeneration
  • Each specific "backend implementation binary" implements at least the main service, and any number of relevant optional services (no more need to have non-implemented methods on every backend)
  • Also allows LocalAI master node to query all connected backends in a p2p swarm to know who and what can service in the next PRs
• Split the messages up
  • RPC request / response messages should be specific to the rpc, apparently. Easily fixed.
  • Common fields go into can go into shared structures
• Go multi-file -- apparently, it's considered pretty abnormal to use a single .proto file and we will have better tooling with "one concept per file", which is usually the case with code-like-files anyway...

---

Generation

Buf is an open source tool that provides additional tooling around protobuf, like the linting I mentioned above. It can check for breaking changes - which may be useful in the future. It can also directly handle codegen - it handles some of the preprocessing, and seems to call out to the usual protogen family of tools to spit out the actual code. The organization behind Buf provides some additional features through what they call "BSR" or Buf Schema Registry -- which seems to be free for small, open source users like us today... but we would need to be aware of the fact that they could take that away from us at some point. Depending on our level of tolerance of dependency on this outside resource (granted, we already have those for CI and hosting), I see a few paths forward in order of decreasing dependence:

1. If we fully trust Buf/BSR, they provide a fully managed codegen solution that would allow us to remove the costly gRPC compiling / protogen steps entirely from our builds. We could easily point go and python directly at generated modules, and they provide a CMake config that we'd be able to merge into llama.cpp as well.
2. If we are not willing to be fully dependent on the BSR, but are willing to leverage their free resources for as long as they last, they also provide "managed generation plugins" - I have done some prototyping with these and they are pretty neat. They seem to allow us to generate code for all major languages on a CI machine, without needing to setup the protogen toolchains ourselves, as they will handle that all based on a simple yaml file. I've already successfully tested this with golang and python -- but I'd need to do some research to make sure we understand if c++ linking will need us to compile gRPC on CI anyway, which would negate the benefits
3. buf generate also doesn't need to use managed mode. If none of the above are palatable, I think we can use our current methods of getting protogen in place, and then just let buf call it. Again, more research needed, but this may have some benefits over option four if my preliminary research is right?
4. We could ignore buf generate completely, and use our current generation code unmodified - even if we do modify the underlying protobuf files significantly, and use buf lint in our pipeline! It's worth noting that this section and the previous header are essentially independent