A modern open source language for analyzing, transforming, and modeling data.

Malloy is a modern open source language for describing data relationships and transformations. It is both a semantic modeling language and a query language that uses an existing SQL engine to execute queries. Malloy currently can connect to BigQuery, Snowflake, PostgreSQL, MySQL, Trino, or Presto, and natively supports DuckDB. We've built a Visual Studio Code extension to facilitate building Malloy data models, querying and transforming data, and creating simple visualizations and dashboards.

• Malloy @ GitHub.

Related contents:

• Lessons on building an AI data analyst @ Pedro Nascimento.

Chronon is a data platform for serving for AI/ML applications.

Chronon is a platform that abstracts away the complexity of data computation and serving for AI/ML applications. Users define features as transformation of raw data, then Chronon can perform batch and streaming computation, scalable backfills, low-latency serving, guaranteed correctness and consistency, as well as a host of observability and monitoring tools.

It allows you to utilize all of the data within your organization, from batch tables, event streams or services to power your AI/ML projects, without needing to worry about all the complex orchestration that this would usually entail.

Extract, Transform, Index Data. Easy and Fresh. CocoIndex is the world's first open-source engine that supports both custom transformation logic and incremental updates specialized for data indexing.

With CocoIndex, users declare the transformation, CocoIndex creates & maintains an index, and keeps the derived index up to date based on source update, with minimal computation and changes.

• CocoIndex @ GitHub.

🪄 Create rich visualizations with AI

Data Formulator is an application from Microsoft Research that uses large language models to transform data, expediting the practice of data visualization.

Data Formulator is an AI-powered tool for analysts to iteratively create rich visualizations. Unlike most chat-based AI tools where users need to describe everything in natural language, Data Formulator combines user interface interactions (UI) and natural language (NL) inputs for easier interaction. This blended approach makes it easier for users to describe their chart designs while delegating data transformation to AI.

Efficient data transformation and modeling framework that is backwards compatible with dbt.

SQLMesh is a next-generation data transformation framework designed to ship data quickly, efficiently, and without error. Data teams can efficiently run and deploy data transformations written in SQL or Python with visibility and control at any size.

• SQLMesh @ GitHub.

Related contents:

• Why SQLMesh Might be The Best dbt Alternative @ The Data Toolbox.

Data Runs Better on SDF. Transform Data Better with SDF. SDF is the fastest way to build a scalable, reliable, and optimized data warehouse.

SDF is a developer platform for data that scales SQL understanding across an organization, empowering all data teams to unlock the full potential of their data.

SDF is a multi-dialect SQL compiler, transformation framework, and analytical database engine. It natively compiles SQL dialects, like Snowflake, and connects to their corresponding data warehouses to materialize models.

• SDF Docs.
• SDF @ GitHub.

Source: Testing is Not Enough: Transforming Data Quality with Write, Audit, Publish using SDF Build @ SDF Blog.

Cross-Language Serialization for Relational Algebra. A cross platform way to express data transformation, relational algebra, standardized record expression and plans.

Substrait is a format for describing compute operations on structured data. It is designed for interoperability across different languages and systems.

• Substrait @ GitHub.
• Your November Dose of Data - November 2024 @ Data Council.

RSS-lambda transforms RSS feeds without RSS client lock-in.

There are RSS clients that can perform transformations on RSS feeds, e.g. only keep entries with certain keywords, or translate texts of the entries

However, using those features from the RSS clients will create RSS client lock-in that prevents you from moving to another RSS client if you desire

RSS-lambda is an application that perform transformations on the server-side instead so that you can freely move to another RSS client while keeping the transformations. It's also self-hostable so that you don't even need to rely on the official server instance!

• RSS-lambda @ GitHub.

Transform Data in Your Warehouse. Build trusted data products faster.

Accelerate your data transformation process with dbt Cloud and start delivering data that you and your team can rely on. dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications. Analysts using dbt can transform their data by simply writing select statements, while dbt handles turning these statements into tables and views in a data warehouse.

• dbt Core @ GitHub.
• dbt Developer Hub.

Sources:

• 268 - Résilience de la data - Sammy Teillet @ <ifttd> :fr:.
• Optimizing SQL queries for speed with dbt @ DataDuel.co.
• Test Driven Development (TDD) with dbt: Test First, SQL Later @ Xebia.
• Understanding dbt: basics and best practices @ Datadog.