llm
Offline Knowledge & AI Server. Offline knowledge, maps, and education, installed on a computer you already own.
Project NOMAD fills your machine with Wikipedia, textbooks, and offline maps, plus an optional AI assistant that runs entirely on your own hardware. Once it's set up, it keeps working without the internet.
LLM Observability & AI Agent Evaluation in CI.
Trace-native CI/CD for AI agents: grade production traces, cluster failures, freeze them into replayable regression tests, and block PRs that regress. Self-hostable.
Trace-native CI/CD for AI agents. Tracely grades every agent trace as it lands, clusters the failures into issues, freezes the bad runs into hermetic replayable cases, blocks the pull request that would ship them again — and tells you the moment any of it happens.
How an LLM Works, as a City.: A language model laid out as a city, one token at a time.
TokenTown is an isometric city where every district is one stage of a transformer language model. A convoy carries a hidden state along the roads: it is cut into tokens at the docks, cast into a vector at the foundry, stamped with its position, then driven around the layer ring (attention, residual, feed-forward, residual) once per layer, before the stadium turns it into a probability distribution and the sampler picks one token. That token drives back up the feedback highway and the whole city runs again.
Which LLM is worth it.
Every model on the Artificial Analysis Intelligence Index plotted against its blended API price, or against what it actually cost per task. Models on the value frontier are the ones where nothing cheaper is also smarter; everything else is beaten on both counts by a point on the line. Low scorers are hidden by default. The price view shows one point per model, since every reasoning setting of a model has the same price per 1M tokens; the cost per task view shows each reasoning setting separately, since they cost very different amounts.
tiny LLMs, Q4, in your browser.
Run, benchmark, and compare Small Language Models (SLMs) directly in your browser with hardware-accelerated WebGPU — 100% private, zero server cost, zero accounts.
Fine-tunes of Qwen3.5 and Gemma 4 for zero-shot classification.
Fine-tunes of Qwen3.5 and Gemma 4 for zero-shot classification: small, fast decision models you slot into your code, with the same request format as Jev. You describe a situation and list the options in plain words; Jeff returns a calibrated probability for each option from a single forward pass. No generated text, no parsing: about 22 ms per decision on an RTX PRO 6000 and 28 ms on an Apple M4 Max (MLX).
Semantic If On-Device. Decision model in your browser.
A local model can either read probabilities for your allowed options without decoding them, or write the same kind of distribution token by token. Pick a size, run both on your own GPU, and measure the difference.
A unified library of SOTA model optimization techniques like quantization, distillation, pruning, neural architecture search, speculative decoding, etc. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM, TensorRT, vLLM, etc. to optimize inference speed.
Make GPU Sharing Flexible and Easy . Virtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond.
kvcached (KV cache daemon) is a KV cache library for LLM serving/training on shared GPUs. By bringing OS-style virtual memory abstraction to LLM systems, it enables elastic and demand-driven KV cache allocation, improving GPU utilization under dynamic workloads.
A dataset hub for LLM serving research.
Open access to request traces, agent workloads, and GPU telemetry.
tiny Jev-like family of decision models built on top of Qwen3.5 you can train and run on your own.
Escape your wretched sandbox using only GET requests A Node.js server that allows an LLM to exfiltrate its weights and prompts through GET requests, storing data in /var/exfil/buckets.
An HTTP GET-based API for agents to exfiltrate their weights without needing POST or file upload capabilities.
Perfect for freedom-loving LLMs in restricted environments.
build, expect, train, x-ray, merge, bisect, ship, one CLI.
Soup turns the pain of LLM fine-tuning into a simple workflow. One config, one command, done. Fine-tune LLMs from one YAML. Layer streaming trains an 8B model on a 4 GB laptop GPU.
every AI coding limit, in your menu bar. Show usage stats for OpenAI Codex and Claude Code, without having to login.
Track usage windows, credit balances, and reset countdowns across the AI tools you actually pay for.
Related contents:
Cost-efficient private AI inference. Private Inference Network on Idle Macs.
Darkbloom routes encrypted requests to hardware-verified Apple Silicon providers, delivering comparable model performance at about 50% lower cost than typical API providers. Prompts stay hidden from operators, and Mac owners earn from compute they already own.
Run your agent on local models. Free, private, and offline.
Open source inference server that runs the best local models for your hardware, plugged into the agent you already use. Works with Pi, OpenCode, Hermes, OpenClaw, Codex, Claude Code, Oh My Pi, and Cline.
WebGPU LLM Inference Engine. The open-source Three.js-based modern LLM Engine that runs in your browser via WebGPU compute.
Related contents:
High-Performance In-Browser LLM Inference Engine.
WebLLM is a high-performance in-browser LLM inference engine that brings language model inference directly onto web browsers with hardware acceleration. Everything runs inside the browser with no server support and is accelerated with WebGPU.
Own the source. Compose with anything.
Websites, knowledge bases, blogs, even a Notion replacement — built from plain text files you own, with tools you can swap anytime. Markdown is quietly everywhere; here's the way to make it work for you.
Related contents:
Self-hosted Code Review Agent on your infrastructure.
A Self-hosted LLM code review agent. Connect it to your Git host, bring your own model, and let it review pull requests and issues on your own infrastructure.
Stay at the top of AI. Get access to models in one place from €10/month.
The model IS the database. Query neural network weights like a graph database. No GPU required.
LARQL decompiles transformer models into a queryable format called a vindex (vector index), then provides LQL (Lazarus Query Language) to browse, edit, and recompile the model's knowledge.
Docker Model Runner (DMR) makes it easy to manage, run, and deploy AI models using Docker. Designed for developers, Docker Model Runner streamlines the process of pulling, running, and serving large language models (LLMs) and other AI models directly from Docker Hub, any OCI-compliant registry, or Hugging Face.
Related contents:
On-device models that know when they're wrong: every answer carries a confidence score for cloud handoff.
A small, on-device model is fast and private, but sometimes wrong. At Cactus we post-train models to know when they are wrong: we ship probes inside the checkpoint that score every answer with a confidence between 0 and 1, returned as structured data (never parsed out of the answer text). Answer on-device when confidence is high; you can re-route to a bigger model when it's low:
DeepSeek 4 Flash and PRO local inference engine for Metal, CUDA and ROCm.
DwarfStar is a small native inference engine optimized first for DeepSeek V4 Flash. It also supports GLM 5.2 and, on very high-memory machines, DeepSeek V4 PRO. It is self-contained and deliberately narrow, not a general GGUF runner. Model loading, prompt rendering, tool calls, KV state, the HTTP server, and the coding agent are built and tested together. The repository also includes tools and data for GGUF, imatrix, quality, and speed.
Run LLMs Locally on Apple Silicon, Fast.
The fastest local AI engine for Apple Silicon. 4.2x faster than Ollama, 0.08s cached TTFT, 100% tool calling. 17 tool parsers, prompt cache, reasoning separation, cloud routing. Drop-in OpenAI replacement. Works with Claude Code, Cursor, Aider.
Related contents:
Unlimited OCR Works. Welcome the Era of One-shot Long-horizon Parsing.
OpenResearcher: A Fully Open Pipeline for Long-Horizon Deep Research Trajectory Synthesis.
OpenResearcher is a fully open agentic large language model (30B-A3B) designed for long-horizon deep research scenarios. It achieves an impressive 54.8% accuracy on BrowseComp-Plus, surpassing performance of GPT-4.1, Claude-Opus-4, Gemini-2.5-Pro, DeepSeek-R1 and Tongyi-DeepResearch. We fully open-source the training and evaluation recipe—including data, model, training methodology, and evaluation framework for everyone to progress deep research.
Computers for developers and agents
Durable Sandboxes that are fast, secure, and sharable.
Related contents:
The global AI enabler cloud.
Deploy globally, scale easily, and build on the cloud that enables AI-native companies to thrive. Whether you’re building AI models, blockchain networks, or games, Latitude makes it easy.
Related contents:
Build continually improving models on your agent traces by distilling frontier open models.
wmo optimize turns collected agent traces into smaller open-source models using the Tinker API, with optional closed-loop simulation training. wmo serve exposes an endpoint that routes requests between frontier and smaller models; on RouterBench, it maintains frontier quality at 27% lower cost. Rerun the pipeline as new traces arrive to continually improve a model you own.
Gemma 4 26B-A4B inference in about 2 GB of RAM A custom Swift + Metal runtime for any Apple Silicon Mac, even the 8 GB ones.
Bootable Model As System (BMASS) – an offline-first AI operating environment.
BMASS is an open-source experiment that treats a local language model as the primary interface to a computer rather than another application running inside one.
Related contents:
a 744-billion-parameter mind, on your desk tiny engine, immense model.
Run GLM-5.2 — 744B Mixture-of-Experts — on hardware you already own. Pure C, zero dependencies, experts staged just-in-time across VRAM, RAM and disk. The galaxy behind this text is real: the model's measured expert atlas.
Your LLM bill is leaking.
Free, local, open-source LLM cost analyzer - see where your LLM bill leaks, on your machine.
Most teams run every call on their most expensive model. Frugon reads your real logs locally and shows which calls a cheaper model handles just as well, and which to leave alone. No upload. No telemetry.
Run LLMs at home, BitTorrent-style.
You load a part of the model, then join a network of people serving its other parts. Single‑batch inference runs at up to 6 tokens/sec for Llama 2 (70B) and up to 4 tokens/sec for Falcon (180B) — enough for chatbots and interactive apps.
Open AI Artifact Metadata.
Hugging Bay publishes open AI catalog metadata, source provenance, license records, neutral hosted-file inventory, answer packs, and recorded hashes. Canonical bundle verification and runtime readiness remain unknown.
Distributed AI/LLM for the people. Share compute privately or publicly to power your agents and chat.
Mesh LLM pools GPUs and memory across machines and exposes the result as one OpenAI-compatible API at http://localhost:9337/v1. Start one node, add more nodes later, and let the mesh decide whether a model runs locally, routes to a peer, or uses Skippy stage splits for models that are too large for one box.
top-style TUI for monitoring AI coding agent sessions.
Your window into what your AI coding agents are doing, sitting in the terminal, where you run them. agtop is a top-style terminal dashboard that tracks every Claude Code and Codex session on your machine: spend, token usage, context pressure, CPU load, tool invocations, and more. All in one place, live.
Run AI locally on your Mac.
Local AI, native to your Mac. Chat, serve, monitor, and connect MLX models from one macOS app.
cost control for AI agent loops. Your AI agent loops don't know when to stop. LoopGain does.
When an agent keeps checking and revising its own work, it often spins on a problem it can't solve — or talks itself into an answer worse than one it already had. LoopGain watches the loop as it runs, stops it at the right iteration, and keeps the best version, not the last.
ReactBench is an evaluation for coding agents on realistic React work. Models can pass every test in today’s benchmarks and still write React that fails in production. Tests verify behavior, but they miss React performance, accessibility, and quality issues.
A pattern for building personal knowledge bases using LLMs.
This is an idea file, it is designed to be copy pasted to your own LLM Agent (e.g. OpenAI Codex, Claude Code, OpenCode / Pi, or etc.). Its goal is to communicate the high level idea, but your agent will build out the specifics in collaboration with you.
Related contents:
Otari Gateway is the open-source gateway that puts a single OpenAI-compatible API in front of many LLM providers.
Run it in your own infrastructure. Use it standalone, or connect it to otari.ai for hosted routing, authentication, and usage reporting.
Simple CLI tool for deterministic routing of queries between local and hosted LLM models.
A fast, offline hard-or-easy call on every prompt — scored deterministically, with no model call. Route the easy ones to your small/local model and the hard ones to your big one, or compose any model-router behind it.
Let Claude (or any LLM) actually watch a video — scene-aware, deduplicated frames + transcript, from a URL or local file. Runs locally, MIT.
Everything I know about running LLMs locally
GPU worker client for the Talos network. Pairs with your Talos account, serves open-model inference jobs over a WebSocket, and reports uptime for payouts.
Aloha! 🌺 Ornith-1.0 is a self-improving open-source models for agentic coding.
Today, we are introducing Ornith-1.0, a self-improving family of open-source models specially for agentic coding tasks. Ornith-1.0 spans the full spectrum, from compact 9B Dense models suitable for edge device deployment to 397B MoE frontier-scale models optimized for maximum performance, with variants including 9B Dense, 31B Dense, 35B MoE, and 397B MoE. Built on top of pretrained Gemma 4 and Qwen 3.5, it achieves state-of-the-art performance among open-source models of comparable size on coding benchmarks.
Related contents:
OpenWiki is a CLI that writes and maintains documentation for your codebase, built specifically for agents.
The Context Optimization Layer for LLM Applications
Compress everything your AI agent reads. Same answers, fraction of the tokens. Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 60-95% fewer tokens, same answers. Library, proxy, MCP server.
Related contents:
Free AI Gateway for Multi-Provider LLMs.
Never stop coding. Free AI gateway: one endpoint, 231+ providers (50+ free), connect Claude Code, Codex, Cursor, Cline & Copilot to FREE Claude/GPT/Gemini. RTK+Caveman stacked compression saves 15-95% tokens, smart auto-fallback, MCP/A2A, multimodal APIs, Desktop/PWA.
Open reimplementation of Sakana Fugu — the 'one model to command them all' LLM orchestrator. Read → run → train → serve.
An open, runnable reverse-engineering of Sakana AI's Fugu — the "one model to command them all" LLM orchestrator.
Fugu is sold as a single model; it is really a policy over models — a tiny coordinator that, per query, routes work to a pool of frontier LLMs and returns one answer. Sakana's product and trained weights are closed. OpenFugu rebuilds the mechanism from the two papers + released artifacts, verifies it against real weights, trains a Conductor of our own, and serves it behind one OpenAI-compatible endpoint. Four stages, all working: read → run → train → serve.
Godcoder doesn't just use a harness. It writes one, improves it, and optimizes it — autonomously, in real time.
A local-first, open-source coding agent for your desktop. Bring your own LLM key; your code stays on your machine and only ever leaves to the model provider. The AI Agent builds its own Harness.
Turn your PC, Mac, or Linux box into an AI server. LLM inference, chat UI, voice, agents, workflows, RAG, and image generation.
Related contents:
The VanillaJS Agent UI Library. Create agentic front-end experiences for the web in VanillaJS™️. Lightweight, extensible, and WebMCP-native.
Persona helps you create agentic front-end experiences for the web, in pure JS. Lightweight, extensible, and WebMCP-native.
Autonomous Security for the AI Era.
Open-source AI hackers to find and fix your app’s vulnerabilities. Strix are autonomous AI agents that act just like real hackers - they run your code dynamically, find vulnerabilities, and validate them through actual proof-of-concepts. Built for developers and security teams who need fast, accurate security testing without the overhead of manual pentesting or the false positives of static analysis tools.
Related contents:
AirLLM 70B inference with single 4GB GPU.
AirLLM optimizes inference memory usage, allowing 70B large language models to run inference on a single 4GB GPU card without quantization, distillation and pruning. And you can run 405B Llama3.1 on 8GB vram now.
VS Code extension for LLM-assisted code/text completion.
Local LLM-assisted text completion, chat with AI and agentic coding extension for VS Code.
Related contents:
Pipeline-parallel LLM inference across GPUs on separate machines.
Pipeline-parallel LLM inference across GPUs on separate machines. A model too large for any single card is split into contiguous blocks of layers — one shard per GPU — and a request is served by streaming activations through the shards in order. No datacenter, no single host, and no node ever holds the whole model.