big-data
Open Source, Distributed, Big Data Enterprise Search Engine.
Datafari is an open source enterprise search solution enriched with AI. It is the perfect product for anyone who needs to search and analyze its corporate data and documents, both within the content and the metadata. Plus, with its genAI modules, it allows to easily leverage mistral, openai, or local LLMs for your company data.
AI data platform.
From data warehouse to autonomous data and AI platform
BigQuery is the autonomous data to AI platform, automating the entire data life cycle, from ingestion to AI-driven insights, so you can go from data to AI to action faster.
Gemini in BigQuery features are now included in BigQuery pricing models.
Related contents:
Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.
Related contents:
Related contents:
- The New Look and Feel of Apache Kafka 4.0 @ The New Stack.
- Kafka: The End of the Beginning @ Materialized View.
- Optimizing Kafka Tracing with OpenTelemetry: Boost Visibility & Performance @ New Relic.
- Introducing Apache Kafka® 4.1.0: What’s New and How to Upgrade @ Confluent.
- Testing Kafka-based Asynchronous Workflows Using OpenTelemetry @ Signadot.
A unified metadata lake across all your sources, formats, cloud providers, and regions in a federated architecture. World's most powerful open data catalog for building a high-performance, geo-distributed and federated metadata lake.
Apache Gravitino is a high-performance, geo-distributed, and federated metadata lake. It manages metadata directly in different sources, types, and regions, providing users with unified metadata access for data and AI assets.
Insights, Unlocked in Real Time.
Apache Pinot™: The real-time analytics open source platform for lightning-fast insights, effortless scaling, and cost-effective data-driven decisions.
Related contents:
The open table format for analytic datasets.
Iceberg is a high-performance format for huge analytic tables. Iceberg brings the reliability and simplicity of SQL tables to big data, while making it possible for engines like Spark, Trino, Flink, Presto, Hive and Impala to safely work with the same tables, at the same time.
Related contents:
- PyIceberg: Current State and Roadmap @ Ju Data Engineering Newsletter.
- The Equality Delete Problem in Apache Iceberg @ Data Engineer Things's Medium.
- How I Saved Millions by Restructuring Iceberg Metadata @ Gautham Gondi's Medium.
- High Throughput Ingestion with Iceberg @ Adobe Tech Blog's Medium.
- Scaling Iceberg Writes with Confidence: A Conflict-Free Distributed Architecture for Fast, Concurrent, Consistent Append-Only Writes @ e6data.
The best way of working with Protocol Buffers. Elastic, self-hosted Kafka with Advanced Semantic Intelligence Guarantee streaming data quality and slash cloud costs 10x with Bufstream, a drop-in replacement for Apache Kafka®.
Bufstream is a Kafka-compatible streaming system which stores records directly in an object storage service like S3.
Seamless multi-master sync, that scales from Big Data to Mobile, with an Intuitive HTTP/JSON API and designed for Reliability.
CouchDB is a database that completely embraces the web. Store your data with JSON documents. Access your documents with your web browser, via HTTP. Query, combine, and transform your documents with JavaScript. CouchDB works well with modern web and mobile apps. You can distribute your data, efficiently using CouchDB’s incremental replication. CouchDB supports master-master setups with automatic conflict detection.
Related contents:
Open source analytics infrastructure. Fast and scalable. No bloat. GDPR compliant.
A single production-ready Docker image built on ClickHouse, Kafka, and Node.js for tracking events, users, page views, and interactions.
The Apache® Hadoop® project develops open-source software for reliable, scalable, distributed computing.
The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-availability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of computers, each of which may be prone to failures.
Source Available Reinvented Kafka. 10x Cost Efficiency.
AutoMQ is a cloud-first alternative to Kafka by decoupling durability to S3 and EBS. 10x cost-effective. Autoscale in seconds. Single-digit ms latency.
AutoMQ is a stateless Kafka on S3. 10x Cost-Effective. No Cross-AZ Traffic Cost. Autoscale in seconds. Single-digit ms latency. Multi-AZ Availability.
Related contents:
An Open Source Data Lake Platform.
Apache Hudi is a transactional data lake platform that brings database and data warehouse capabilities to the data lake. Hudi reimagines slow old-school batch data processing with a powerful new incremental processing framework for low latency minute-level analytics.
Les technologies numériques sont incroyablement puissantes et redéfinissent le fonctionnement de notre société. Pour les acteurs qui œuvrent pour l'intérêt général, la technologie peut parfois être un levier démutiplicateur d'impacts positifs, cependant et malheureusement ces acteurs n'ont souvent pas les ressources technologiques ou humaines pour accélérer leur action citoyenne. Data for Good existe pour rétablir l'équilibre.
Interactive SQL. Analyze petabyte-scale data where it lives with ease and flexibility.
Amazon Athena is a serverless, interactive analytics service built on open-source frameworks, supporting open-table and file formats. Athena provides a simplified, flexible way to analyze petabytes of data where it lives. Analyze data or build applications from an Amazon Simple Storage Service (S3) data lake and 30 data sources, including on-premises data sources or other cloud systems using SQL or Python. Athena is built on open-source Trino and Presto engines and Apache Spark frameworks, with no provisioning or configuration effort required.
DataFusion is a very fast, extensible query engine for building high-quality data-centric systems in Rust, using the Apache Arrow in-memory format.
DataFusion is great for building projects such as domain specific query engines, new database platforms and data pipelines, query languages and more. It lets you start quickly from a fully working engine, and then customize those features specific to your use.
Open Source Distributed POSIX File System for Cloud. JuiceFS is a distributed POSIX file system built on top of Redis and S3.
JuiceFS is a high-performance POSIX file system released under Apache License 2.0, particularly designed for the cloud-native environment. The data, stored via JuiceFS, will be persisted in Object Storage (e.g. Amazon S3), and the corresponding metadata can be persisted in various compatible database engines such as Redis, MySQL, and TiKV based on the scenarios and requirements.
With JuiceFS, massive cloud storage can be directly connected to big data, machine learning, artificial intelligence, and various application platforms in production environments. Without modifying code, the massive cloud storage can be used as efficiently as local storage.
open source big data platform.
Trunk Data Platform is an Open Source, free, Hadoop distribution.
Druid is a high performance, real-time analytics database that delivers sub-second queries on streaming and batch data at scale and under load.
XetHub brings speedy access and Git-based collaboration to large scale repositories of data, code, or any combination of files. Our instant mount feature makes it possible to access GBs and TBs of data in seconds at the speed of localhost, while our de-duplication algorithm stores data and differences efficiently to save money and speed up development cycles. XetHub is ideal for teams who already use Git to track their code changes, and want to leverage the power of infinite history, pull requests, and difference-based tracking for larger assets such as datasets or media files. Managing complete projects with familiar Git semantics makes change tracking and continuous integration a breeze, especially for workflows that use code to generate or augment assets.
Graph Database Management System. Neo4j Graph Data Platform. Blazing-Fast Graph, Petabyte Scale. With proven trillion+ entity performance, developers, data scientists, and enterprises rely on Neo4j as the top choice for high-performance, scalable analytics, intelligent app development, and advanced AI/ML pipelines.
ClickHouse® is a free analytics DBMS for big data. ClickHouse® is an open-source column-oriented database management system that allows generating analytical data reports in real-time.
Open big JSON, CSV Files: Online Viewer, Explorer and Converter. View and convert big data files. View large or small files right in your browser and export them in any format.
Daily Earth Data to See Change and Make Better Decisions. Planet provides daily satellite data that helps businesses, governments, researchers, and journalists understand the physical world and take action.
Climate TRACE was built to collect and share greenhouse gas emissions from anthropogenic (human) activities to facilitate climate action .
Robtex is used for various kinds of research of IP numbers, Domain names, etc. Robtex uses various sources to gather public information about IP numbers, domain names, host names, Autonomous systems, routes etc. It then indexes the data in a big database and provide free access to the data. We aim to make the fastest and most comprehensive free DNS lookup tool on the Internet. Our database now contains billions of documents of internet data collected over more than a decade.
Political data for 233 countries. The world’s richest open dataset on politicians
The scalable, open source big data analytics platform for networks and services.
A modular scientific software framework. It provides all the functionalities needed to deal with big data processing, statistical analysis, visualisation and storage. It is mainly written in C++ but integrated with other languages such as Python and R.
A user-friendly, map-based tool to combine and explore real-time or historical data.
WhereHows is a data discovery and lineage tool built at LinkedIn. It integrates with all the major data processing systems and collects both catalog and operational metadata from them.
high performance, high scalability and high reliability database for big data. GridDB has a KVS (Key-Value Store)-type data model that is suitable for sensor data stored in a timeseries. It is a database that can be easily scaled-out according to the number of sensors.
Apache HBase™ is the Hadoop database, a distributed, scalable, big data store. Use Apache HBase when you need random, realtime read/write access to your Big Data. This project's goal is the hosting of very large tables -- billions of rows X millions of columns -- atop clusters of commodity hardware. Apache HBase is an open-source, distributed, versioned, non-relational database modeled after Google's Bigtable: A Distributed Storage System for Structured Data by Chang et al. Just as Bigtable leverages the distributed data storage provided by the Google File System, Apache HBase provides Bigtable-like capabilities on top of Hadoop and HDFS.
D3.js is a JavaScript library for manipulating documents based on data. D3 helps you bring data to life using HTML, SVG and CSS. D3’s emphasis on web standards gives you the full capabilities of modern browsers without tying yourself to a proprietary framework, combining powerful visualization components and a data-driven approach to DOM manipulation.
Waarp provides a secure and efficient open source MFT solution.
Waarp Platform is a set of applications and tools specialized in managing and monitoring a high number of transfers in a secure and reliable way.
It relies on its own open protocol named R66, which has been designed to optimize file transfers, ensure the integrity of the data provide ways to integrate transfers in larger business transactions.
The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using a simple programming model. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-avaiability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-availabile service on top of a cluster of computers, each of which may be prone to failures.