llm
a 744-billion-parameter mind, on your desk tiny engine, immense model.
Run GLM-5.2 — 744B Mixture-of-Experts — on hardware you already own. Pure C, zero dependencies, experts staged just-in-time across VRAM, RAM and disk. The galaxy behind this text is real: the model's measured expert atlas.
Your LLM bill is leaking.
Free, local, open-source LLM cost analyzer - see where your LLM bill leaks, on your machine.
Most teams run every call on their most expensive model. Frugon reads your real logs locally and shows which calls a cheaper model handles just as well, and which to leave alone. No upload. No telemetry.
Run LLMs at home, BitTorrent-style.
You load a part of the model, then join a network of people serving its other parts. Single‑batch inference runs at up to 6 tokens/sec for Llama 2 (70B) and up to 4 tokens/sec for Falcon (180B) — enough for chatbots and interactive apps.
Open AI Artifact Metadata.
Hugging Bay publishes open AI catalog metadata, source provenance, license records, neutral hosted-file inventory, answer packs, and recorded hashes. Canonical bundle verification and runtime readiness remain unknown.
Distributed AI/LLM for the people. Share compute privately or publicly to power your agents and chat.
Mesh LLM pools GPUs and memory across machines and exposes the result as one OpenAI-compatible API at http://localhost:9337/v1. Start one node, add more nodes later, and let the mesh decide whether a model runs locally, routes to a peer, or uses Skippy stage splits for models that are too large for one box.
top-style TUI for monitoring AI coding agent sessions.
Your window into what your AI coding agents are doing, sitting in the terminal, where you run them. agtop is a top-style terminal dashboard that tracks every Claude Code and Codex session on your machine: spend, token usage, context pressure, CPU load, tool invocations, and more. All in one place, live.
Run AI locally on your Mac.
Local AI, native to your Mac. Chat, serve, monitor, and connect MLX models from one macOS app.
cost control for AI agent loops. Your AI agent loops don't know when to stop. LoopGain does.
When an agent keeps checking and revising its own work, it often spins on a problem it can't solve — or talks itself into an answer worse than one it already had. LoopGain watches the loop as it runs, stops it at the right iteration, and keeps the best version, not the last.
ReactBench is an evaluation for coding agents on realistic React work. Models can pass every test in today’s benchmarks and still write React that fails in production. Tests verify behavior, but they miss React performance, accessibility, and quality issues.
A pattern for building personal knowledge bases using LLMs.
This is an idea file, it is designed to be copy pasted to your own LLM Agent (e.g. OpenAI Codex, Claude Code, OpenCode / Pi, or etc.). Its goal is to communicate the high level idea, but your agent will build out the specifics in collaboration with you.
Related contents:
Otari Gateway is the open-source gateway that puts a single OpenAI-compatible API in front of many LLM providers.
Run it in your own infrastructure. Use it standalone, or connect it to otari.ai for hosted routing, authentication, and usage reporting.
Simple CLI tool for deterministic routing of queries between local and hosted LLM models.
A fast, offline hard-or-easy call on every prompt — scored deterministically, with no model call. Route the easy ones to your small/local model and the hard ones to your big one, or compose any model-router behind it.
Let Claude (or any LLM) actually watch a video — scene-aware, deduplicated frames + transcript, from a URL or local file. Runs locally, MIT.
Everything I know about running LLMs locally
GPU worker client for the Talos network. Pairs with your Talos account, serves open-model inference jobs over a WebSocket, and reports uptime for payouts.
Aloha! 🌺 Ornith-1.0 is a self-improving open-source models for agentic coding.
Today, we are introducing Ornith-1.0, a self-improving family of open-source models specially for agentic coding tasks. Ornith-1.0 spans the full spectrum, from compact 9B Dense models suitable for edge device deployment to 397B MoE frontier-scale models optimized for maximum performance, with variants including 9B Dense, 31B Dense, 35B MoE, and 397B MoE. Built on top of pretrained Gemma 4 and Qwen 3.5, it achieves state-of-the-art performance among open-source models of comparable size on coding benchmarks.
Related contents:
OpenWiki is a CLI that writes and maintains documentation for your codebase, built specifically for agents.
The Context Optimization Layer for LLM Applications
Compress everything your AI agent reads. Same answers, fraction of the tokens. Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 60-95% fewer tokens, same answers. Library, proxy, MCP server.
Related contents:
Free AI Gateway for Multi-Provider LLMs.
Never stop coding. Free AI gateway: one endpoint, 231+ providers (50+ free), connect Claude Code, Codex, Cursor, Cline & Copilot to FREE Claude/GPT/Gemini. RTK+Caveman stacked compression saves 15-95% tokens, smart auto-fallback, MCP/A2A, multimodal APIs, Desktop/PWA.
Open reimplementation of Sakana Fugu — the 'one model to command them all' LLM orchestrator. Read → run → train → serve.
An open, runnable reverse-engineering of Sakana AI's Fugu — the "one model to command them all" LLM orchestrator.
Fugu is sold as a single model; it is really a policy over models — a tiny coordinator that, per query, routes work to a pool of frontier LLMs and returns one answer. Sakana's product and trained weights are closed. OpenFugu rebuilds the mechanism from the two papers + released artifacts, verifies it against real weights, trains a Conductor of our own, and serves it behind one OpenAI-compatible endpoint. Four stages, all working: read → run → train → serve.
Godcoder doesn't just use a harness. It writes one, improves it, and optimizes it — autonomously, in real time.
A local-first, open-source coding agent for your desktop. Bring your own LLM key; your code stays on your machine and only ever leaves to the model provider. The AI Agent builds its own Harness.
Turn your PC, Mac, or Linux box into an AI server. LLM inference, chat UI, voice, agents, workflows, RAG, and image generation.
Related contents:
The VanillaJS Agent UI Library. Create agentic front-end experiences for the web in VanillaJS™️. Lightweight, extensible, and WebMCP-native.
Persona helps you create agentic front-end experiences for the web, in pure JS. Lightweight, extensible, and WebMCP-native.
Autonomous Security for the AI Era.
Open-source AI hackers to find and fix your app’s vulnerabilities. Strix are autonomous AI agents that act just like real hackers - they run your code dynamically, find vulnerabilities, and validate them through actual proof-of-concepts. Built for developers and security teams who need fast, accurate security testing without the overhead of manual pentesting or the false positives of static analysis tools.
Related contents:
AirLLM 70B inference with single 4GB GPU.
AirLLM optimizes inference memory usage, allowing 70B large language models to run inference on a single 4GB GPU card without quantization, distillation and pruning. And you can run 405B Llama3.1 on 8GB vram now.
VS Code extension for LLM-assisted code/text completion.
Local LLM-assisted text completion, chat with AI and agentic coding extension for VS Code.
Related contents:
Pipeline-parallel LLM inference across GPUs on separate machines.
Pipeline-parallel LLM inference across GPUs on separate machines. A model too large for any single card is split into contiguous blocks of layers — one shard per GPU — and a request is served by streaming activations through the shards in order. No datacenter, no single host, and no node ever holds the whole model.
Agent substrate is a system built on top of Kubernetes which manages agent-like workloads to achieve higher scale and efficiency than Kubernetes alone can offer, with lower latency. It builds on top of Kubernetes features like Pods and Pod autoscaling, but takes the Kubernetes control-plane out of the critical path to achieve lower latency.
Related contents:
Build applications that make decisions (chatbots, agents, simulations, etc...). Monitor, trace, persist, and execute on your own infrastructure.
Apache Burr (incubating) makes it easy to develop applications that make decisions (chatbots, agents, simulations, etc...) from simple python building blocks.
Apache Burr works well for any application that uses LLMs, and can integrate with any of your favorite frameworks. Burr includes a UI that can track/monitor/trace your system in real time, along with pluggable persisters (e.g. for memory) to save & load application state.
Zero trace. Full answer. — Transparent Claude API proxy that anonymizes PII and sensitive data before it leaves your environment. From pentesters, to pentesters.
PromptZero applies Zero Trust principles to LLM interactions. A local, transparent proxy that detects and replaces sensitive data — identities, infrastructure, secrets, client material — in your prompts before they leave your environment, and restores the real values in the response. Never trust the API. Always verify what crosses the boundary. Your data stays home.
Related contents:
An experiment in using LLMs to teach you, rather than think for you.
Lathe generates hands-on, multi-part technical tutorials on demand, with skills tuned to make content approachable. Then you work through them yourself, by hand, in a local UI built from the ground up for pleasant learning. (Just like we did it in the stone age 😎)
feel LLM tokens-per-second.
tokenspeed is a tiny terminal toy that streams fake tokens at any rate you set, so you can see what those numbers actually look like.
Local-first AI memory layer for any LLM. Persistent knowledge graph, entity extraction, semantic retrieval. Works with Ollama, OpenAI, Anthropic, or any OpenAI-compatible backend.
mnemo is a sidecar service that watches every conversation you feed it, extracts named entities and relationships using an LLM, builds a persistent knowledge graph in SQLite, and injects relevant context back into future prompts — automatically, in under 50ms. It works with Ollama (fully local, free), OpenAI, Anthropic, or any OpenAI-compatible API. It ships as a single static binary with zero cloud dependency.
LLM inference, optimized for your Mac. Local AI, no more waiting on your Mac.
LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar.
macOS-native MLX server with smart caching. Claude Code, OpenClaw, and Cursor respond in 5 seconds, not 90.
Related contents:
A browser extension for AI agents. It clicks, fills, navigates, and extracts. Tedious work, on autopilot.
Related contents:
Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 60-95% fewer tokens, same answers. Library, proxy, MCP server.
Headroom is the context optimization layer for LLM applications. Compress tool outputs, DB results, file reads, and RAG results before they reach the model. Same answers, fraction of the tokens.
Related contents:
Research anything. Do anything.
Research at the speed of thought. The agentic research platform that plans, retrieves, and cites — so you can think faster.
Scira (Formerly MiniPerplx) is a minimalistic AI-powered search engine that helps you find information on the internet and cites it too. Powered by Vercel AI SDK!
Kubernetes for Local LLMs.
A Kubernetes operator for self-hosted LLM inference. vLLM, llama.cpp, TGI, NVIDIA, Apple Silicon.
Related contents:
Unified TUI for inventory, health-check, and track every API key issued across your AI providers.
A unified view of every API key across all your AI providers — with health scoring, snapshot diffs, and encrypted credential storage.
KeyLedger is an interactive terminal dashboard that gives you a unified view of every API key issued across your AI providers — with health scoring, snapshot diffs, and encrypted credential storage.
Find the best local LLM that actually runs on your hardware.
Auto-detects your GPU/CPU/RAM and ranks the top models from HuggingFace that fit your system.
Find the local LLM that actually runs and performs best on your hardware. Ranked by real, recency-aware benchmarks, not parameter count. One command, run it instantly.
Fast, lossless LLM inference via dual-view diffusion decoding. Memory-Efficient Parallel Token Generation via Dual-View Diffusion.
Official implementation and model checkpoints for Orthrus, a dual-architecture framework that unifies the exact generation fidelity of autoregressive Large Language Models (LLMs) with the high-speed parallel token generation of diffusion models.
Refreshingly simple local chat. Local AI for Text, Images, and Speech
The omni-modal alternative to cloud AI. Automatically optimized for your GPU and NPU. Open source, community driven, and private.
Related contents:
Local AI anywhere, for everyone — LLM inference, chat UI, voice, agents, workflows, RAG, and image generation. No cloud, no subscriptions.
Dream Server is the exit. A local-first AI stack — LLM inference, chat, voice, agents, workflows, RAG, image generation, and privacy tools — deployed on your hardware with a single command. No cloud required. No subscriptions required. No one watching. Cloud and hybrid API modes are optional when you choose them.
A comprehensive PHP Generative AI Framework using OpenAI GPT 4. Inspired by Langchain.
Related contents:
A complete walkthrough of how large language models like ChatGPT are built — from raw internet text to a conversational assistant. Based on Andrej Karpathy's technical deep dive.
Experimental Linux strace LLM agent.
pike-agent records and analyzes how programs behave on Linux. It traces a program's activity, indexes it into a database, and lets you chat with an LLM agent about it in a TUI.
Related contents:
Scan your website to see how ready it is for AI agents. We check multiple emerging standards — from robots.txt and Markdown negotiation to MCP, OAuth, Agent Skills and agentic commerce.
AI Model Security Database.
Track jailbreaks, prompt injections, and security incidents across all major AI models. The question is not if - it's when.
All-in-One RAG Framework.
RAG-Anything eliminates the need for multiple specialized tools. It provides seamless processing and querying across all content modalities within a single integrated framework. Unlike conventional RAG approaches that struggle with non-textual elements, our all-in-one system delivers comprehensive multimodal retrieval capabilities.
Users can query documents containing interleaved text, visual diagrams, structured tables, and mathematical formulations through one cohesive interface. This consolidated approach makes RAG-Anything particularly valuable for academic research, technical documentation, financial reports, and enterprise knowledge management where rich, mixed-content documents demand a unified processing framework.
GAIA Logo GAIA: AI Agent Framework for AMD Ryzen AI.
GAIA is AMD's open-source framework for building intelligent AI agents that run 100% locally on AMD Ryzen AI hardware. Keep your data private, eliminate cloud costs, and deploy in air-gapped environments—all with hardware-accelerated performance.
Related contents:
Datadog Static AI Security Testing (SAIST) tool.
Code Security scans your first-party code and open source libraries used in your applications in both your repositories and running services, providing end-to-end visibility from development to production.
Related contents:
A language designed for machines to write. A programming language designed for LLMs to write, not humans.
Vera (v-ERR-a) is a programming language designed for large language models to write. The name comes from the Latin veritas (truth). Programs compile to WebAssembly and run at the command line or in the browser.
reference impl with llama.cpp compiled to distributed inference across machines, with real end to end demo.
Mesh LLM lets you pool spare GPU capacity across machines and expose the result as one OpenAI-compatible API.
If a model fits on one machine, it runs there. If it does not, Mesh LLM automatically spreads the work across the mesh.
Related contents:
A ~9M parameter LLM that talks like a small fish.
This project exists to show that training your own language model is not magic. No PhD required. No massive GPU cluster. One Colab notebook, 5 minutes, and you have a working LLM that you built from scratch — data generation, tokenizer, model architecture, training loop, and inference. If you can run a notebook, you can train a language model.
The free AI already on your Mac.
macOS Tahoe ships with a 3B parameter LLM. apfel gives you CLI access with one brew install. No model downloads, no API keys, no configuration needed, just works.
Related contents:
llama.cpp fork with TurboQuant optimization
Related contents:
Gemma Gem runs Google's Gemma 4 model entirely on-device via WebGPU — no API keys, no cloud, no data leaving your machine.
Your personal AI assistant living right inside the browser. Gemma Gem runs Google's Gemma 4 model entirely on-device via WebGPU — no API keys, no cloud, no data leaving your machine. It can read pages, click buttons, fill forms, run JavaScript, and answer questions about any site you visit.
LiteRT-LM is Google's production-ready, high-performance, open-source inference framework for deploying Large Language Models on edge devices.
ZML is a production inference stack, purpose-built to decouple AI workloads from proprietary hardware.
Any model, many hardwares, one codebase, peak performance. Compiled directly to NVIDIA, AMD, TPU, Trainium for peak hardware performance on any accelerator. No rewriting.
Related contents:
Folo is the AI RSS Reader
AI RSS Reader for deep, noise-free reading with contextual AI.
The AI that reads the internet for you, cutting through noise to surface the knowledge you actually care about.