Tag

apache-spark

Oldest Newest URL A-Z URL Z-A Title A-Z Title Z-A Random
DLT-META
https://databrickslabs.github.io/dlt-meta/

Metadata driven Spark Declarative Pipelines framework for bronze/silver pipelines.

DLT-META is a metadata-driven framework designed to work with Lakeflow Declarative Pipelines. This framework enables the automation of bronze and silver data pipelines by leveraging metadata recorded in an onboarding JSON file. This file, known as the Dataflowspec, serves as the data flow specification, detailing the source and target metadata required for the pipelines.

Related contents:

 
apache-spark dataflow data-lake data-pipeline declarative metadata source-available
Added 7 months ago
Apache Spark
https://spark.apache.org/

Unified Engine for large-scale data analytics.

Apache Spark™ is a multi-language engine for executing data engineering, data science, and machine learning on single-node machines or clusters.

Related contents:

 
apache2-licensed apache-spark big-data data-analytics data-science foss machine-learning open-source sql
Added 7 months ago