Tag

vllm

Oldest Newest URL A-Z URL Z-A Title A-Z Title Z-A Random
Model Optimizer
https://nvidia.github.io/Model-Optimizer/

A unified library of SOTA model optimization techniques like quantization, distillation, pruning, neural architecture search, speculative decoding, etc. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM, TensorRT, vLLM, etc. to optimize inference speed.

 
apache2-licensed foss inference llm open-source optimization vllm
Added 2 hours ago
kvcached
https://github.com/ovg-project/kvcached

Make GPU Sharing Flexible and Easy . Virtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond.

kvcached (KV cache daemon) is a KV cache library for LLM serving/training on shared GPUs. By bringing OS-style virtual memory abstraction to LLM systems, it enables elastic and demand-driven KV cache allocation, improving GPU utilization under dynamic workloads.

 
apache2-licensed cache foss gpu key-value llm open-source self-hosted vllm
Added 8 hours ago