Biapy's Bookmarks

experiment

Mesh LLM

https://github.com/michaelneale/mesh-llm

reference impl with llama.cpp compiled to distributed inference across machines, with real end to end demo.

Mesh LLM lets you pool spare GPU capacity across machines and expose the result as one OpenAI-compatible API.

If a model fits on one machine, it runs there. If it does not, Mesh LLM automatically spreads the work across the mesh.