feat(llama.cpp): Totally decentralized, private, distributed, p2p inference

[mudler]

TLDR;

This PR introduces a peer-to-peer distribution of the workload for inferencing models with llama.cpp (aka 'sharding' for some folks). There are now workers and server nodes, and workers are automatically discovered by the servers also behind different networks.

The work is spread not for each request to a worker, but the workers are actually contributing to the same request alltogether.

This was possible only because of the upstream work in llama.cpp (a big thank you from the LocalAI community!), so it supports only gguf models.

Here you can observe computation split in the same node, but discovery is done with mdns:

Description

As #2324 introduced distributed inferencing thanks to @rgerganov (thank you!) implementation in ggerganov/llama.cpp#6829 in upstream llama.cpp, thanks to that now it is possible to distribute the workload to remote llama.cpp gRPC servers.

To share the workload across nodes however, it would be needed to bring up the services manually and specify the IPs of the endpoints. The objective of this PR is to automatically setup a network of "workers" and the API server leveraging the workers that works in LANs but also behind different networks.

For this, we will split local-ai in two "entrypoints" now, the HTTP server, and the llama workers.

A shared token between the server and the workers is needed to let the communication happen via the p2p network. It supports both local network (with mdns discovery) and dht for communicating also behind different networks.

The token is generated automatically when starting the server with the --p2p flag, and can be used by starting the workers with local-ai worker p2p-llama-cpp-rpc by passing the token via environment variable (TOKEN) or with args (--token).

A network is established between the server and the workers with dht and mdns discovery protocols, the llama.cpp rpc server is automatically started and exposed to the underlying p2p network so the API server can connect on.

When the HTTP server is started, it will discover the workers in the network and automatically create the port-forwards to the service locally. Then llama.cpp is configured to use the services.

This feature is behind the "p2p" GO_FLAGS

Usage

1. Start the server with --p2p:

./local-ai run --p2p
# 1:02AM INF loading environment variables from file envFile=.env
# 1:02AM INF Setting logging to info
# 1:02AM INF P2P mode enabled
# 1:02AM INF No token provided, generating one
# 1:02AM INF Generated Token:
# XXXXXXXXXXX
# 1:02AM INF Press a button to proceed

A token is displayed, copy it and press enter.

You can re-use the same token later restarting the server with --p2ptoken (or P2P_TOKEN).

2. Start the workers. Now you can copy the local-ai binary in other hosts, and run as many workers with that token:

TOKEN=XXX ./local-ai  p2p-llama-cpp-rpc
# 1:06AM INF loading environment variables from file envFile=.env
# 1:06AM INF Setting logging to info
# {"level":"INFO","time":"2024-05-19T01:06:01.794+0200","caller":"config/config.go:288","message":"connmanager disabled\n"}
# {"level":"INFO","time":"2024-05-19T01:06:01.794+0200","caller":"config/config.go:295","message":" go-libp2p resource manager protection enabled"}
# {"level":"INFO","time":"2024-05-19T01:06:01.794+0200","caller":"config/config.go:409","message":"max connections: 100\n"}
# 1:06AM INF Starting llama-cpp-rpc-server on '127.0.0.1:34371'
# {"level":"INFO","time":"2024-05-19T01:06:01.794+0200","caller":"node/node.go:118","message":" Starting EdgeVPN network"}
# create_backend: using CPU backend
# Starting RPC server on 127.0.0.1:34371, backend memory: 31913 MB
# 2024/05/19 01:06:01 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). # See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details.
# {"level":"INFO","time":"2024-05-19T01:06:01.805+0200","caller":"node/node.go:172","message":" Node ID: 12D3KooWJ7WQAbCWKfJgjw2oMMGGss9diw3Sov5hVWi8t4DMgx92"}
# {"level":"INFO","time":"2024-05-19T01:06:01.806+0200","caller":"node/node.go:173","message":" Node Addresses: [/ip4/127.0.0.1/tcp/44931 /ip4/127.0.0.1/udp/33251/quic-v1/webtransport/certhash/uEiAWAhZ-W9yx2ZHnKQm3BE_ft5jjoc468z5-Rgr9XdfjeQ/certhash/uEiB8Uwn0M2TQBELaV2m4lqypIAY2S-2ZMf7lt_N5LS6ojw /ip4/127.0.0.1/udp/35660/quic-v1 /ip4/192.168.68.110/tcp/44931 /ip4/192.168.68.110/udp/33251/quic-v1/webtransport/certhash/uEiAWAhZ-W9yx2ZHnKQm3BE_ft5jjoc468z5-Rgr9XdfjeQ/certhash/uEiB8Uwn0M2TQBELaV2m4lqypIAY2S-2ZMf7lt_N5LS6ojw /ip4/192.168.68.110/udp/35660/quic-v1 /ip6/::1/tcp/41289 /ip6/::1/udp/33160/quic-v1/webtransport/certhash/uEiAWAhZ-W9yx2ZHnKQm3BE_ft5jjoc468z5-Rgr9XdfjeQ/certhash/uEiB8Uwn0M2TQBELaV2m4lqypIAY2S-2ZMf7lt_N5LS6ojw /ip6/::1/udp/35701/quic-v1]"}
# {"level":"INFO","time":"2024-05-19T01:06:01.806+0200","caller":"discovery/dht.go:104","message":" Bootstrapping DHT"}

(Note you can also supply the token via args)

At this point, you should see in the server logs messages stating that new workers are found

3. Now you can start doing inference as usual on the server (the node used on step 1)

Notes

• Only single model is supported for now
• There are many options at p2p level that can be tweaked, room for follow-ups
• Make sure that the server sees new workers after usage starts - currently, if you start the inference you can't add other workers later on.

Technical implementation

The PR leverages https://github.com/mudler/edgevpn to establish a secure, distributed network between the a set of nodes using a shared token. Edgevpn already implements a small protocol to decentralize and establish a peer-to-peer network between the nodes using libp2p. Edgevpn also offers a small shared ledger that is really useful for this purpose - as it allows to share automatically information about the workers and discover them automatically, as edgevpn is a library - which despite the name - does not necessarly is a VPN and can be used also only to establish a private p2p network.

Behind the scenes, this PR uses the services capabilities (https://github.com/mudler/edgevpn/blob/master/cmd/service.go) of Edgevpn to expose and connect to services in the private p2p network.

Edgevpn has a shared ledger between all the nodes of a p2p network (identified by the token) - the worker creates a tunnel to the p2p network to expose a service, which is identified by a UUID. The UUID is announced to the ledger so all the nodes knows about the service available.

The HTTP server (started with the --p2p flag) is tracking and discovering services UUIDs in the ledger periodically, and when it finds new service in the ledger, it port-forwards to it by creating a local tunnel, and populating the LLAMA_CPP_GRPC_SERVER environment variable as expected

[netlify]

✅ Deploy Preview for localai canceled.

Name	Link
🔨 Latest commit	6fefd3b
🔍 Latest deploy log	https://app.netlify.com/sites/localai/deploys/664b74604d44110008fec520

[mudler]

we need to build some safeguard to check if the rpc-server llama.cpp breaks CLI options, but there is lots of room for optimizations for a later batch of changes

[JackBekket]

we need to build some safeguard to check if the rpc-server llama.cpp breaks CLI options, but there is lots of room for optimizations for a later batch of changes

I've tried to attach local-ai-cpu worker to local-ai-gpu head and it failed because CUDA mismatch on worker side
Maybe there should be safeguard for that

Also I would like to ask what p2p discovery method you are using? Is it KademilaDHT peer discovery?
https://github.com/libp2p/specs/blob/master/kad-dht/README.md ?
https://github.com/libp2p/go-libp2p/tree/master/examples/chat-with-rendezvous ?

[JackBekket]

Are you using libp2p GossipSub for routing?

[mudler]

Are you using libp2p GossipSub for routing?

yes! it uses https://github.com/mudler/edgevpn behind the scenes, which in turn, uses libp2p

I've tried to attach local-ai-cpu worker to local-ai-gpu head and it failed because CUDA mismatch on worker side Maybe there should be safeguard for that

Good point - I'm not sure what are the plans in llama.cpp about that, but looks something we want to cover here

Also I would like to ask what p2p discovery method you are using? Is it KademilaDHT peer discovery? https://github.com/libp2p/specs/blob/master/kad-dht/README.md ? https://github.com/libp2p/go-libp2p/tree/master/examples/chat-with-rendezvous ?

it uses both DHT (the repository you linked) discovery and mDNS