sql
The best dashboards are built with code. Create fast, beautiful data apps, dashboards, and reports from the command line. Write Markdown, JavaScript, SQL, Python, R… and any language you like. Free and open-source.
A static site generator for data apps, dashboards, reports, and more. Observable Framework combines JavaScript on the front-end for interactive graphics with any language on the back-end for data analysis.
csvkit is a suite of command-line tools for converting to and working with CSV, the king of tabular file formats.
It is inspired by pdftk, GDAL and the original csvcut tool by Joe Germuska and Aaron Bycoffe.
It provides in2csv to convert xlsx to csv, and csvsql to populate a database with a csv file contents.
Sirius is a GPU-native SQL engine. It plugs into existing databases such as DuckDB via the standard Substrait query format, requiring no query rewrites or major system changes. Sirius currently supports DuckDB and Doris (coming soon), other systems marked with * are on our roadmap.
Create custom SQLite functions in JavaScript. Extend your database with scalars, aggregates, window functions, and collations directly in JavaScript.
SQLite-JS is a powerful extension that brings JavaScript capabilities to SQLite. With this extension, you can create custom SQLite functions, aggregates, window functions, and collation sequences using JavaScript code, allowing for flexible and powerful data manipulation directly within your SQLite database.
LumoSQL is a modification (not a fork) of the SQLite embedded data storage library, which is among the most-deployed software. LumoSQL adds security, privacy, performance and measurement features to SQLite.
DuckLake is an integrated data lake and catalog format
DuckLake delivers advanced data lake features without traditional lakehouse complexity by using Parquet files and your SQL database. It's an open, standalone format from the DuckDB team.
DuckLake is an open Lakehouse format that is built on SQL and Parquet. DuckLake stores metadata in a catalog database, and stores data in Parquet files. The DuckLake extension allows DuckDB to directly read and write data from DuckLake.
Index your Gmail account to a SQLite DB and play with the data.
This is a script to download emails from Gmail and store them in a SQLite database for further analysis. I find it extremely useful to have all my emails in a database to run queries on them. For example, I can find out how many emails I received per sender, which emails take the most space, and which emails from which sender I never read.
Search input query parser and React component.
A powerful search query language parser with SQL output support, inspired by Elasticsearch and Tantivy query syntax.
All-in-One Desktop App to Analyze Data Locally.
TextQuery is an all-in-one desktop app to import, query, modify, and visualize your raw data with SQL.
A Doom-like game using DuckDB.
A 3D first-person shooter game implemented entirely in SQL using DuckDB-WASM.
SQLite is a C-language library that implements a small, fast, self-contained, high-reliability, full-featured, SQL database engine. SQLite is the most used database engine in the world. SQLite is built into all mobile phones and most computers and comes bundled inside countless other applications that people use every day. More Information...
Related contents:
- SQLite File Format Viewer.
- SQLite Transactions and Virtual Tables @ Misframe.
- A faster way to copy SQLite databases between computers @, alexwlchan.
- Scaling SQLite to 4M QPS on a single server (EC2 vs Bare Metal) @ Expensify.
- PSA: SQLite WAL checksums fail silently and may lose data @ blag.
- Lessons learned from building a sync-engine and reactivity system with SQLite @ Antoine's blog.
A collection of language tools and a Language Server Protocol (LSP) implementation for Postgres, focusing on developer experience and reliable SQL tooling.
Related contents:
A ledger implementation in PostgreSQL.
Related contents:
Cloud-native stream processing. Distributed stream processing engine in Rust.
Transform, filter, aggregate, and join data streams by writing SQL, with sub-second results. Scale from zero to millions of events per second.
Arroyo is a distributed stream processing engine written in Rust, designed to efficiently perform stateful computations on streams of data. Unlike traditional batch processing, streaming engines can operate on both bounded and unbounded sources, emitting results as soon as they are available.
Related contents:
The Headless CMS + Backend for Every Custom Build. Directus is a real-time API and App dashboard for managing SQL database content. The flexible backend for all your projects 🐰 Turn your DB into a headless CMS, admin panels, or apps with a custom UI, instant APIs, auth & more.
Built for developers who need more than just a CMS. Manage complex content structures, handle digital assets, and control user permissions – all through an intuitive Studio.
Related contents:
Open Source, Multi Cloud, Analytics, IaC, more. Provision. Query. Secure.
Query, provision and operate Cloud and SaaS resources and APIs using an extensible SQL based framework. Deploy, manage and query cloud resources and interact with APIs using SQL.
Related content:
SQL Indexing and Tuning e-Book for developers. Use The Index, Luke covers Oracle, MySQL, PostgreSQL, SQL Server, ...
A site explaining SQL indexing to developers—no crap about administration. SQL indexing is the most effective tuning method—yet it is often neglected during development. Use The Index, Luke explains SQL indexing from grounds up and doesn’t stop at ORM tools like Hibernate.
A Detective SQL Game
SQL Noir is an interactive mystery-solving game where you crack cases by writing SQL queries.
Efficient data transformation and modeling framework that is backwards compatible with dbt.
SQLMesh is a next-generation data transformation framework designed to ship data quickly, efficiently, and without error. Data teams can efficiently run and deploy data transformations written in SQL or Python with visibility and control at any size.
Related contents:
Visualizing and understanding PostgreSQL EXPLAIN plans made easy.
A VueJS component to show a graphical vizualization of a PostgreSQL execution plan.
review Postgres query plans quickly.
A simple yet powerful tool to help you speed up queries. pgMustard speeds up your journey from knowing which query is a problem to working out what can be done about it.
The lightweight, user-friendly, distributed relational database built on SQLite. Fault tolerance and high availability with zero hassle.
Related contents:
Easily ask questions about your Linux, Windows, and macOS infrastructure.
Related contents:
SQL-like Querying for Various Data Sources.
Musoq lets you use SQL-like queries on files, directories, images and other data sources without a database. It's designed to ease life for developers.
Musoq is a tool that lets developers and IT professionals query different data sources using SQL-like syntax, without needing to import data into a database first. It’s designed for scenarios where you need to analyze files, directories, archives, or other data sources quickly and efficiently.
Data Runs Better on SDF. Transform Data Better with SDF. SDF is the fastest way to build a scalable, reliable, and optimized data warehouse.
SDF is a developer platform for data that scales SQL understanding across an organization, empowering all data teams to unlock the full potential of their data.
SDF is a multi-dialect SQL compiler, transformation framework, and analytical database engine. It natively compiles SQL dialects, like Snowflake, and connects to their corresponding data warehouses to materialize models.
✨A CLI tool for analyzing queries and DB tables for MySQL.
The Database Toolkit for Python.
SQLAlchemy is the Python SQL toolkit and Object Relational Mapper that gives application developers the full power and flexibility of SQL.
It provides a full suite of well known enterprise-level persistence patterns, designed for efficient and high-performing database access, adapted into a simple and Pythonic domain language.
Distributed SQL Databases
Fastest serverless distributed SQL database for always available applications.
SQL language server written in Go.
sqls aims to provide advanced intelligence for you to edit sql in your own editor.
Open-source sQL AI Agent. Text2SQL made Easy!
Wren AI is an open-source SQL AI Agent that empowers data, product, and business teams to access insights through AI chat, built-in well designed intuitive UI and UX, integrating seamlessly with tools like Excel and Google Sheets.
Type safe queries using the Rust type system.
The goal of this library is to allow writing relational database queries using familiar Rust syntax. The library should guarantee that a query can not fail if it compiles. This already includes preventing use after free for row ids passed between queries and even database migrations!
The open table format for analytic datasets.
Iceberg is a high-performance format for huge analytic tables. Iceberg brings the reliability and simplicity of SQL tables to big data, while making it possible for engines like Spark, Trino, Flink, Presto, Hive and Impala to safely work with the same tables, at the same time.
Related contents:
The world's most advanced open source database.
Related contents:
- What I Wish Someone Told Me About Postgres @ ChallahScript.
- Logical replication in Postgres: Basics @ EDB.
- Database sharding case study: PostgreSQL @ The Polymathic Engineer.
- I Tried Every Open Source Database and Found Only One That is Actually Useful @ Hacker Noon.
- How direct TLS can speed up your connections @ Marc Bowes.
- PostgreSQL pour remplacer Redis @ Rodolphe Bréard :fr:.
- One more reason to choose Postgres over MySQL @ tanin.
- From 99.9% to 99.99%: Building PostgreSQL Resilience into Your Product Architecture @ Stormatics.
- Handling unique indexes on large data in PostgreSQL @ Volodymyr Potiichuk.
type safe SQL for TypeScript/JavaScript. Lightweight RDBMS agnostic TypeScript/JS abstraction for SQLite, PostgreSQL and MySQL.
SQL you know but type safe parameterized, expressive & portable.
litdb contains LINQ-like type-safe SQL builders for TypeScript/JavaScript for writing type-safe expressive SQL that's parameterized & portable across SQLite, MySQL & PostgreSQL.
The Query Builder Component for React.
React Query Builder is a fully customizable query builder component for React, along with a collection of utility functions for importing from, and exporting to, various query languages like SQL, MongoDB, and more.
A consistent code style guide for SQL to ensure legible and maintainable projects.
A web-based tool to easily generate customizable SQLMap commands for testing SQL injection vulnerabilities. Features include target configuration, connection options, detection levels, and various SQL injection techniques. Perfect for penetration testers and security enthusiasts.
Business Intelligence as Code. Build polished data products with SQL. Build fast, interactive data visualizations in pure SQL and markdown.
Evidence is a lightweight framework for building data apps. It's open source and free to get started.
Run SQL queries on CSV files directly in your browser. No data leaves your browser. Fast, private, and easy to use.
A (somewhat opinionated) list of SQL tips and tricks that I've picked up over the years.
There's so much you can you do with SQL but I've focused on what I find most useful in my day-to-day work as a data analyst and what I wish I had known when I first started writing SQL.
Use SQL for everything. Query anything with old-school cool SQL.
Anyquery is a CLI tool to run SQL queries on any data source, no matter if it's a file, an API, logs, or a local app. See the integrations for the full extent of what you can do.
🤖 Chat with your SQL database 📊. Accurate Text-to-SQL Generation via LLMs using RAG 🔄.
Personalized AI SQL Agent. Let Vanna.AI write your SQL for you
The fastest way to get actionable insights from your database just by asking questions.
wrangle data.
sq is a command line tool that provides jq-style access to structured data sources: SQL databases, or document formats like CSV or Excel. It is the lovechild of sql+jq.
Proof of SQL is a high performance zero knowledge (ZK) prover developed by the Space and Time team, which cryptographically guarantees SQL queries were computed accurately against untampered data. It targets online latencies while proving computations over entire chain histories, an order of magnitude faster than state-of-the art zkVMs and coprocessors.
The Cloud Operational Data Store. Use SQL to transform, deliver, and act on fast-changing data.
Materialize is a cloud-native data warehouse purpose-built for operational workloads where an analytical data warehouse would be too slow, and a stream processor would be too complicated.
Using SQL and common tools in the wider data ecosystem, Materialize allows you to build real-time automation, engaging customer experiences, and interactive data products that drive value for your business while reducing the cost of data freshness.
FerretDB allows you to use MongoDB drivers seamlessly with PostgreSQL as the database backend. Use all tools, drivers, UIs, and the same query language and stay open-source.
meilibridge is a robust package designed to seamlessly sync data from both SQL and NoSQL databases to Meilisearch, providing an efficient and unified search solution.
Upon first encountering SQL after two decades of Fortran, C, Java, and Python, I thought I had stumbled into hell. I quickly realized that was optimistic: after all, hell has rules.
I have since realized that SQL does too, and that they are no more confusing or contradictory than those of most other programming languages. They only appear so because it draws on a tradition unfamiliar to those of us raised with derivatives of C. To quote Terry Pratchett, it is not mad, just differently sane.
Welcome, then, to a world in which the strange will become familiar, and the familiar, strange. Welcome, thrice welcome, to SQL.
DataFusion is a very fast, extensible query engine for building high-quality data-centric systems in Rust, using the Apache Arrow in-memory format.
DataFusion is great for building projects such as domain specific query engines, new database platforms and data pipelines, query languages and more. It lets you start quickly from a fully working engine, and then customize those features specific to your use.
Build APIs in 5 minutes. Build NodeJS / GO APIs in 5 minutes not weeks. A GraphQL-to-SQL converting app.
Just write simple GraphQL queries to define the data you need and GraphJin will auto-magically convert them into efficient SQL queries and fetch the data you need.
📊 Cube — The Semantic Layer for Building Data Applications. The Universal Semantic Layer.
Build trust with a semantic layer. Connect siloed data, define consistent metrics, and power AI and analytics with context.
Cube is the semantic layer for building data applications. It helps data engineers and application developers access data from modern data stores, organize it into consistent definitions, and deliver it to every application.
Cube was designed to work with all SQL-enabled data sources, including cloud data warehouses like Snowflake or Google BigQuery, query engines like Presto or Amazon Athena, and application databases like Postgres. Cube has a built-in relational caching engine to provide sub-second latency and high concurrency for API requests.
an open source Distributed SQL Database.
YDB is a versatile open source Distributed SQL Database that combines high availability and scalability with strong consistency and ACID transactions. It accommodates transactional (OLTP), analytical (OLAP), and streaming workloads simultaneously.
AdminerEvo is a web-based database management interface, with a focus on security, user experience, performance, functionality and size.
AdminerEvo works out of the box with MySQL, MariaDB, PostgreSQL, SQLite, MS SQL, Oracle, Elasticsearch and MongoDB. In addition, there are plugins for SimpleDB, Firebird and ClickHouse.
Open Source SQL+noSQL Database Client. The Smartest SQL+noSQL Database Client.
Database manager for MySQL, PostgreSQL, SQL Server, MongoDB, SQLite and others. Runs under Windows, Linux, Mac or as web application
BigTesty: integration testing for BigQuery.
BigTesty is a framework that allows to create Integration Tests with BigQuery on a real and short-lived Infrastructure. Integration and End-to-End tests are a robust way to validate if SQL queries work as expected. There is no an emulator in this case and the queries are executed directly in the BigQuery Engine.
Dataherald is a natural language-to-SQL engine built for enterprise-level question answering over relational data. It allows you to set up an API from your database that can answer questions in plain English.
Transform Data in Your Warehouse. Build trusted data products faster.
Accelerate your data transformation process with dbt Cloud and start delivering data that you and your team can rely on. dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications. Analysts using dbt can transform their data by simply writing select statements, while dbt handles turning these statements into tables and views in a data warehouse.
Sources: