llm
build, expect, train, x-ray, merge, bisect, ship, one CLI.
Soup turns the pain of LLM fine-tuning into a simple workflow. One config, one command, done. Fine-tune LLMs from one YAML. Layer streaming trains an 8B model on a 4 GB laptop GPU.
every AI coding limit, in your menu bar. Show usage stats for OpenAI Codex and Claude Code, without having to login.
Track usage windows, credit balances, and reset countdowns across the AI tools you actually pay for.
Related contents:
Cost-efficient private AI inference. Private Inference Network on Idle Macs.
Darkbloom routes encrypted requests to hardware-verified Apple Silicon providers, delivering comparable model performance at about 50% lower cost than typical API providers. Prompts stay hidden from operators, and Mac owners earn from compute they already own.
Run your agent on local models. Free, private, and offline.
Open source inference server that runs the best local models for your hardware, plugged into the agent you already use. Works with Pi, OpenCode, Hermes, OpenClaw, Codex, Claude Code, Oh My Pi, and Cline.
WebGPU LLM Inference Engine. The open-source Three.js-based modern LLM Engine that runs in your browser via WebGPU compute.
Related contents:
High-Performance In-Browser LLM Inference Engine.
WebLLM is a high-performance in-browser LLM inference engine that brings language model inference directly onto web browsers with hardware acceleration. Everything runs inside the browser with no server support and is accelerated with WebGPU.
Own the source. Compose with anything.
Websites, knowledge bases, blogs, even a Notion replacement — built from plain text files you own, with tools you can swap anytime. Markdown is quietly everywhere; here's the way to make it work for you.
Related contents:
Self-hosted Code Review Agent on your infrastructure.
A Self-hosted LLM code review agent. Connect it to your Git host, bring your own model, and let it review pull requests and issues on your own infrastructure.
Stay at the top of AI. Get access to models in one place from €10/month.
The model IS the database. Query neural network weights like a graph database. No GPU required.
LARQL decompiles transformer models into a queryable format called a vindex (vector index), then provides LQL (Lazarus Query Language) to browse, edit, and recompile the model's knowledge.
Docker Model Runner (DMR) makes it easy to manage, run, and deploy AI models using Docker. Designed for developers, Docker Model Runner streamlines the process of pulling, running, and serving large language models (LLMs) and other AI models directly from Docker Hub, any OCI-compliant registry, or Hugging Face.
Related contents:
On-device models that know when they're wrong: every answer carries a confidence score for cloud handoff.
A small, on-device model is fast and private, but sometimes wrong. At Cactus we post-train models to know when they are wrong: we ship probes inside the checkpoint that score every answer with a confidence between 0 and 1, returned as structured data (never parsed out of the answer text). Answer on-device when confidence is high; you can re-route to a bigger model when it's low:
DeepSeek 4 Flash and PRO local inference engine for Metal, CUDA and ROCm.
DwarfStar is a small native inference engine optimized first for DeepSeek V4 Flash. It also supports GLM 5.2 and, on very high-memory machines, DeepSeek V4 PRO. It is self-contained and deliberately narrow, not a general GGUF runner. Model loading, prompt rendering, tool calls, KV state, the HTTP server, and the coding agent are built and tested together. The repository also includes tools and data for GGUF, imatrix, quality, and speed.
Run LLMs Locally on Apple Silicon, Fast.
The fastest local AI engine for Apple Silicon. 4.2x faster than Ollama, 0.08s cached TTFT, 100% tool calling. 17 tool parsers, prompt cache, reasoning separation, cloud routing. Drop-in OpenAI replacement. Works with Claude Code, Cursor, Aider.
Related contents:
Unlimited OCR Works. Welcome the Era of One-shot Long-horizon Parsing.
OpenResearcher: A Fully Open Pipeline for Long-Horizon Deep Research Trajectory Synthesis.
OpenResearcher is a fully open agentic large language model (30B-A3B) designed for long-horizon deep research scenarios. It achieves an impressive 54.8% accuracy on BrowseComp-Plus, surpassing performance of GPT-4.1, Claude-Opus-4, Gemini-2.5-Pro, DeepSeek-R1 and Tongyi-DeepResearch. We fully open-source the training and evaluation recipe—including data, model, training methodology, and evaluation framework for everyone to progress deep research.
Computers for developers and agents
Durable Sandboxes that are fast, secure, and sharable.
Related contents:
The global AI enabler cloud.
Deploy globally, scale easily, and build on the cloud that enables AI-native companies to thrive. Whether you’re building AI models, blockchain networks, or games, Latitude makes it easy.
Related contents:
Build continually improving models on your agent traces by distilling frontier open models.
wmo optimize turns collected agent traces into smaller open-source models using the Tinker API, with optional closed-loop simulation training. wmo serve exposes an endpoint that routes requests between frontier and smaller models; on RouterBench, it maintains frontier quality at 27% lower cost. Rerun the pipeline as new traces arrive to continually improve a model you own.
Gemma 4 26B-A4B inference in about 2 GB of RAM A custom Swift + Metal runtime for any Apple Silicon Mac, even the 8 GB ones.
Bootable Model As System (BMASS) – an offline-first AI operating environment.
BMASS is an open-source experiment that treats a local language model as the primary interface to a computer rather than another application running inside one.
Related contents:
a 744-billion-parameter mind, on your desk tiny engine, immense model.
Run GLM-5.2 — 744B Mixture-of-Experts — on hardware you already own. Pure C, zero dependencies, experts staged just-in-time across VRAM, RAM and disk. The galaxy behind this text is real: the model's measured expert atlas.
Your LLM bill is leaking.
Free, local, open-source LLM cost analyzer - see where your LLM bill leaks, on your machine.
Most teams run every call on their most expensive model. Frugon reads your real logs locally and shows which calls a cheaper model handles just as well, and which to leave alone. No upload. No telemetry.
Run LLMs at home, BitTorrent-style.
You load a part of the model, then join a network of people serving its other parts. Single‑batch inference runs at up to 6 tokens/sec for Llama 2 (70B) and up to 4 tokens/sec for Falcon (180B) — enough for chatbots and interactive apps.
Open AI Artifact Metadata.
Hugging Bay publishes open AI catalog metadata, source provenance, license records, neutral hosted-file inventory, answer packs, and recorded hashes. Canonical bundle verification and runtime readiness remain unknown.
Distributed AI/LLM for the people. Share compute privately or publicly to power your agents and chat.
Mesh LLM pools GPUs and memory across machines and exposes the result as one OpenAI-compatible API at http://localhost:9337/v1. Start one node, add more nodes later, and let the mesh decide whether a model runs locally, routes to a peer, or uses Skippy stage splits for models that are too large for one box.
top-style TUI for monitoring AI coding agent sessions.
Your window into what your AI coding agents are doing, sitting in the terminal, where you run them. agtop is a top-style terminal dashboard that tracks every Claude Code and Codex session on your machine: spend, token usage, context pressure, CPU load, tool invocations, and more. All in one place, live.
Run AI locally on your Mac.
Local AI, native to your Mac. Chat, serve, monitor, and connect MLX models from one macOS app.
cost control for AI agent loops. Your AI agent loops don't know when to stop. LoopGain does.
When an agent keeps checking and revising its own work, it often spins on a problem it can't solve — or talks itself into an answer worse than one it already had. LoopGain watches the loop as it runs, stops it at the right iteration, and keeps the best version, not the last.
ReactBench is an evaluation for coding agents on realistic React work. Models can pass every test in today’s benchmarks and still write React that fails in production. Tests verify behavior, but they miss React performance, accessibility, and quality issues.
A pattern for building personal knowledge bases using LLMs.
This is an idea file, it is designed to be copy pasted to your own LLM Agent (e.g. OpenAI Codex, Claude Code, OpenCode / Pi, or etc.). Its goal is to communicate the high level idea, but your agent will build out the specifics in collaboration with you.
Related contents:
Otari Gateway is the open-source gateway that puts a single OpenAI-compatible API in front of many LLM providers.
Run it in your own infrastructure. Use it standalone, or connect it to otari.ai for hosted routing, authentication, and usage reporting.
Simple CLI tool for deterministic routing of queries between local and hosted LLM models.
A fast, offline hard-or-easy call on every prompt — scored deterministically, with no model call. Route the easy ones to your small/local model and the hard ones to your big one, or compose any model-router behind it.
Let Claude (or any LLM) actually watch a video — scene-aware, deduplicated frames + transcript, from a URL or local file. Runs locally, MIT.
Everything I know about running LLMs locally
GPU worker client for the Talos network. Pairs with your Talos account, serves open-model inference jobs over a WebSocket, and reports uptime for payouts.
Aloha! 🌺 Ornith-1.0 is a self-improving open-source models for agentic coding.
Today, we are introducing Ornith-1.0, a self-improving family of open-source models specially for agentic coding tasks. Ornith-1.0 spans the full spectrum, from compact 9B Dense models suitable for edge device deployment to 397B MoE frontier-scale models optimized for maximum performance, with variants including 9B Dense, 31B Dense, 35B MoE, and 397B MoE. Built on top of pretrained Gemma 4 and Qwen 3.5, it achieves state-of-the-art performance among open-source models of comparable size on coding benchmarks.
Related contents:
OpenWiki is a CLI that writes and maintains documentation for your codebase, built specifically for agents.
The Context Optimization Layer for LLM Applications
Compress everything your AI agent reads. Same answers, fraction of the tokens. Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 60-95% fewer tokens, same answers. Library, proxy, MCP server.
Related contents:
Free AI Gateway for Multi-Provider LLMs.
Never stop coding. Free AI gateway: one endpoint, 231+ providers (50+ free), connect Claude Code, Codex, Cursor, Cline & Copilot to FREE Claude/GPT/Gemini. RTK+Caveman stacked compression saves 15-95% tokens, smart auto-fallback, MCP/A2A, multimodal APIs, Desktop/PWA.
Open reimplementation of Sakana Fugu — the 'one model to command them all' LLM orchestrator. Read → run → train → serve.
An open, runnable reverse-engineering of Sakana AI's Fugu — the "one model to command them all" LLM orchestrator.
Fugu is sold as a single model; it is really a policy over models — a tiny coordinator that, per query, routes work to a pool of frontier LLMs and returns one answer. Sakana's product and trained weights are closed. OpenFugu rebuilds the mechanism from the two papers + released artifacts, verifies it against real weights, trains a Conductor of our own, and serves it behind one OpenAI-compatible endpoint. Four stages, all working: read → run → train → serve.
Godcoder doesn't just use a harness. It writes one, improves it, and optimizes it — autonomously, in real time.
A local-first, open-source coding agent for your desktop. Bring your own LLM key; your code stays on your machine and only ever leaves to the model provider. The AI Agent builds its own Harness.
Turn your PC, Mac, or Linux box into an AI server. LLM inference, chat UI, voice, agents, workflows, RAG, and image generation.
Related contents:
The VanillaJS Agent UI Library. Create agentic front-end experiences for the web in VanillaJS™️. Lightweight, extensible, and WebMCP-native.
Persona helps you create agentic front-end experiences for the web, in pure JS. Lightweight, extensible, and WebMCP-native.
Autonomous Security for the AI Era.
Open-source AI hackers to find and fix your app’s vulnerabilities. Strix are autonomous AI agents that act just like real hackers - they run your code dynamically, find vulnerabilities, and validate them through actual proof-of-concepts. Built for developers and security teams who need fast, accurate security testing without the overhead of manual pentesting or the false positives of static analysis tools.
Related contents:
AirLLM 70B inference with single 4GB GPU.
AirLLM optimizes inference memory usage, allowing 70B large language models to run inference on a single 4GB GPU card without quantization, distillation and pruning. And you can run 405B Llama3.1 on 8GB vram now.
VS Code extension for LLM-assisted code/text completion.
Local LLM-assisted text completion, chat with AI and agentic coding extension for VS Code.
Related contents:
Pipeline-parallel LLM inference across GPUs on separate machines.
Pipeline-parallel LLM inference across GPUs on separate machines. A model too large for any single card is split into contiguous blocks of layers — one shard per GPU — and a request is served by streaming activations through the shards in order. No datacenter, no single host, and no node ever holds the whole model.
Agent substrate is a system built on top of Kubernetes which manages agent-like workloads to achieve higher scale and efficiency than Kubernetes alone can offer, with lower latency. It builds on top of Kubernetes features like Pods and Pod autoscaling, but takes the Kubernetes control-plane out of the critical path to achieve lower latency.
Related contents:
Build applications that make decisions (chatbots, agents, simulations, etc...). Monitor, trace, persist, and execute on your own infrastructure.
Apache Burr (incubating) makes it easy to develop applications that make decisions (chatbots, agents, simulations, etc...) from simple python building blocks.
Apache Burr works well for any application that uses LLMs, and can integrate with any of your favorite frameworks. Burr includes a UI that can track/monitor/trace your system in real time, along with pluggable persisters (e.g. for memory) to save & load application state.
Zero trace. Full answer. — Transparent Claude API proxy that anonymizes PII and sensitive data before it leaves your environment. From pentesters, to pentesters.
PromptZero applies Zero Trust principles to LLM interactions. A local, transparent proxy that detects and replaces sensitive data — identities, infrastructure, secrets, client material — in your prompts before they leave your environment, and restores the real values in the response. Never trust the API. Always verify what crosses the boundary. Your data stays home.
Related contents:
An experiment in using LLMs to teach you, rather than think for you.
Lathe generates hands-on, multi-part technical tutorials on demand, with skills tuned to make content approachable. Then you work through them yourself, by hand, in a local UI built from the ground up for pleasant learning. (Just like we did it in the stone age 😎)
feel LLM tokens-per-second.
tokenspeed is a tiny terminal toy that streams fake tokens at any rate you set, so you can see what those numbers actually look like.
Local-first AI memory layer for any LLM. Persistent knowledge graph, entity extraction, semantic retrieval. Works with Ollama, OpenAI, Anthropic, or any OpenAI-compatible backend.
mnemo is a sidecar service that watches every conversation you feed it, extracts named entities and relationships using an LLM, builds a persistent knowledge graph in SQLite, and injects relevant context back into future prompts — automatically, in under 50ms. It works with Ollama (fully local, free), OpenAI, Anthropic, or any OpenAI-compatible API. It ships as a single static binary with zero cloud dependency.
LLM inference, optimized for your Mac. Local AI, no more waiting on your Mac.
LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar.
macOS-native MLX server with smart caching. Claude Code, OpenClaw, and Cursor respond in 5 seconds, not 90.
Related contents:
A browser extension for AI agents. It clicks, fills, navigates, and extracts. Tedious work, on autopilot.
Related contents:
Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 60-95% fewer tokens, same answers. Library, proxy, MCP server.
Headroom is the context optimization layer for LLM applications. Compress tool outputs, DB results, file reads, and RAG results before they reach the model. Same answers, fraction of the tokens.
Related contents:
Research anything. Do anything.
Research at the speed of thought. The agentic research platform that plans, retrieves, and cites — so you can think faster.
Scira (Formerly MiniPerplx) is a minimalistic AI-powered search engine that helps you find information on the internet and cites it too. Powered by Vercel AI SDK!
Kubernetes for Local LLMs.
A Kubernetes operator for self-hosted LLM inference. vLLM, llama.cpp, TGI, NVIDIA, Apple Silicon.
Related contents:
Unified TUI for inventory, health-check, and track every API key issued across your AI providers.
A unified view of every API key across all your AI providers — with health scoring, snapshot diffs, and encrypted credential storage.
KeyLedger is an interactive terminal dashboard that gives you a unified view of every API key issued across your AI providers — with health scoring, snapshot diffs, and encrypted credential storage.