data-lake
A High-Performance Analytical Database.
The world's fastest open query engine for sub-second analytics both on and off the data lakehouse. With the flexibility to support nearly any scenario, StarRocks provides best-in-class performance for multi-dimensional analytics, real-time analytics, and ad-hoc queries. A Linux Foundation project.
Transactional Catalog for Data Lakes with Git-like semantics.
Nessie supports Iceberg Tables/Views. Additionally, Nessie is focused on working with the widest range of tools possible, which can be seen in the feature matrix.
Related contents:
Scalable Distributed Storage.
SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, xDC replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding. Enterprise version is at seaweedfs.com.
Metadata driven Spark Declarative Pipelines framework for bronze/silver pipelines.
DLT-META is a metadata-driven framework designed to work with Lakeflow Declarative Pipelines. This framework enables the automation of bronze and silver data pipelines by leveraging metadata recorded in an onboarding JSON file. This file, known as the Dataflowspec, serves as the data flow specification, detailing the source and target metadata required for the pipelines.
Related contents:
a data lakehouse for you and me. managed + real-time Iceberg.
🥮 is real-time + managed Apache Iceberg. bringing open analytical tables on object store to every team.
pg_mooncake is a ClickHouse alternative for real-time analytics built on Postgres. It turns Postgres into a real-time analytics database by adding:
Columnar storage (Apache Iceberg, via Moonlink)
Vectorized execution with DuckDB (via pg_duckdb).
Fast analytics queries require both columnar storage & vectorized execution, and previous Postgres analytics solutions only solved half the problem.
Postgres with Iceberg and data lake access.
pg_lake integrates Iceberg and data lake files into Postgres. With the pg_lake extensions, you can use Postgres as a stand-alone lakehouse system that supports transactions and fast queries on Iceberg tables, and can directly work with raw data files in object stores like S3.
Related contents:
Your data lakehouse, built like software.
Bauplan is a cloud-native lakehouse platform for engineering teams who treat data like software. Ship pipelines without managing infrastructure, using a specialized Python runtime, Git-for-Data built on Apache Iceberg, and just a few simple APIs.
Related contents:
DuckLake is an integrated data lake and catalog format
DuckLake delivers advanced data lake features without traditional lakehouse complexity by using Parquet files and your SQL database. It's an open, standalone format from the DuckDB team.
DuckLake is an open Lakehouse format that is built on SQL and Parquet. DuckLake stores metadata in a catalog database, and stores data in Parquet files. The DuckLake extension allows DuckDB to directly read and write data from DuckLake.
AI data platform.
From data warehouse to autonomous data and AI platform
BigQuery is the autonomous data to AI platform, automating the entire data life cycle, from ingestion to AI-driven insights, so you can go from data to AI to action faster.
Gemini in BigQuery features are now included in BigQuery pricing models.
Related contents:
Fastest way to Replicate your Database data in Data Lake. OLake makes data replication faster by parallelizing full loads, leveraging change streams for real-time sync, and pulling data in a database-native format for efficient ingestion.
Fastest open-source tool for replicating Databases to Apache Iceberg or Data Lakehouse. ⚡ Efficient, quick and scalable data ingestion for real-time analytics. Supporting Postgres, MongoDB and MySQL
Related contents:
A unified metadata lake across all your sources, formats, cloud providers, and regions in a federated architecture. World's most powerful open data catalog for building a high-performance, geo-distributed and federated metadata lake.
Apache Gravitino is a high-performance, geo-distributed, and federated metadata lake. It manages metadata directly in different sources, types, and regions, providing users with unified metadata access for data and AI assets.
An Open Source Data Lake Platform.
Apache Hudi is a transactional data lake platform that brings database and data warehouse capabilities to the data lake. Hudi reimagines slow old-school batch data processing with a powerful new incremental processing framework for low latency minute-level analytics.