The Open Lakehouse Format for Multimodal AI.

Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming.

• Lance @ GitHub.

Related contents:

• S3 is the perfect place to store data, until you try to search it @ Gordon Murray.

Modern columnar data format for ML and LLMs implemented in Rust. Convert from parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

Lance is a modern columnar data format optimized for machine learning and AI applications. It efficiently handles diverse multimodal data types while providing high-performance querying and versioning capabilities.

• Lance @ GitHub.

Related contents:

• Lance takes aim at Parquet in file format joust @ The Register.

A declarative, 🐻‍❄️-native data frame validation library.

Dataframely is a Python package to validate the schema and content of polars data frames. Its purpose is to make data pipelines more robust by ensuring that data meet expectations and more readable by adding schema information to data frame type hints.

• Dataframely @ GitHub.

DataFrames for the new era.

Dataframes powered by a multithreaded, vectorized query engine, written in Rust

Polars is a DataFrame interface on top of an OLAP Query Engine implemented in Rust using Apache Arrow Columnar Format as the memory model.

Polars is an open-source library for data manipulation, known for being one of the fastest data processing solutions on a single machine. It features a well-structured, typed API that is both expressive and easy to use.

• Polars @ GitHub.

Related contents:

• Polars at Decathlon: Ready to Play? @ Decathlon Digital.
• Decathlon Switches to Polars to Optimize Data Pipelines and Infrastructure Costs @ InfoQ.
• DuckDB beats Polars for 1TB of data @ Confessions of a Data Guy.