iroh for distributed LLM inference

Most teams end up with GPUs scattered across machines: a beefy workstation here, a couple of spare desktops there, maybe a rack in the back office. Individually, none of them are big enough to run the model you actually want. So you either buy more hardware, or you send your requests to a third-party API and give up control over when models change, where your data goes, and what hardware runs your workloads.

Mesh LLM takes a third path: pool the GPUs you already have into a single OpenAI-compatible API, with no central server coordinating the mesh.

Three ways to run a request

When a request comes in, Mesh LLM picks one of three strategies:

• Local execution on the requesting machine's GPU, if it's big enough
• Routing to a peer node that already has the model loaded
• Pipeline splitting, for models too large for any single machine

Split mode partitions a large model by layer ranges across multiple nodes, so a handful of modest machines can collectively run a model that none of them could load on its own. That's what makes it possible to support 40+ models on the mesh, from half-billion-parameter models small enough for a laptop up to 235B-parameter mixture-of-experts models that need the whole cluster.

No central server to run

Mesh LLM is built on iroh, so nodes connect directly to each other over authenticated QUIC instead of routing requests through a coordinator. Iroh handles NAT traversal and hole-punching automatically, and falls back to a relay when a direct connection isn't possible.

Everything a node needs to say to another node, gossip announcements, routing queries, inference requests, plugin channels, travels over the same tagged QUIC streams, demultiplexed by a single leading byte. That keeps the protocol simple even as the mesh grows.

Getting started

The client is small (about 18 MB) and drops in as a replacement for any OpenAI-compatible endpoint: point your existing tooling at localhost:9337/v1 and it works the same as talking to a hosted API, except the requests stay on hardware you control. From there you can join the public mesh to borrow spare capacity, or configure a private deployment scoped to your own machines.

If you're building something that needs to move inference (or any other workload) across a fleet of machines without standing up new infrastructure, let's talk.