Live truecolor terminal dashboard for a locally running LLM: tok/s, TTFT, GPUs, context, MTP acceptance, layer and real expert routing (llama.cpp).

A live terminal dashboard for the LLM running on your machine.

It finds the inference servers you already have up (llama.cpp llama-server, ollama, vLLM, SGLang, Strata, …), reads their counters and NVIDIA, AMD or Intel GPU telemetry, and turns them into a truecolor picture of what the model is doing right now: tokens per second, time to first token, GPU load and memory, context fill, speculative-decoding acceptance, which layers are busy on which GPU, and, with a small server patch, exactly which experts a mixture-of-experts model routed the last token through.

A unified library of SOTA model optimization techniques like quantization, distillation, pruning, neural architecture search, speculative decoding, etc. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM, TensorRT, vLLM, etc. to optimize inference speed.

• Model Optimizer @ GitHub.

Make GPU Sharing Flexible and Easy . Virtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond.

kvcached (KV cache daemon) is a KV cache library for LLM serving/training on shared GPUs. By bringing OS-style virtual memory abstraction to LLM systems, it enables elastic and demand-driven KV cache allocation, improving GPU utilization under dynamic workloads.