data-science
Superset is a modern data exploration and data visualization platform. Superset can replace or augment proprietary business intelligence tools for many teams. Superset integrates well with a variety of data sources.
YOLOv5 in PyTorch > ONNX > CoreML > TFLite. YOLOv5 is the world's most loved vision AI, representing Ultralytics open-source research into future vision AI methods, incorporating lessons learned and best practices evolved over thousands of hours of research and development.
Folium builds on the data wrangling strengths of the Python ecosystem and the mapping strengths of the Leaflet.js library. Manipulate your data in Python, then visualize it in a Leaflet map via Folium.
Conversational Data Analysis.
PandasAI is a Python platform that makes it easy to ask questions to your data in natural language. It helps non-technical users to interact with their data in a more natural way, and it helps technical users to save time, and effort when working with data.
PandasAI is a Python library that integrates generative artificial intelligence capabilities into pandas, making dataframes conversational. Chat with your database (SQL, CSV, pandas, polars, mongodb, noSQL, etc). PandasAI makes data analysis conversational using LLMs (GPT 3.5 / 4, Anthropic, VertexAI) and RAG.
A NEW ERA OF OPEN LARGE-SCALE MULTI-MODAL DATASETS | LAION.
We present a dataset of 5,85 billion CLIP-filtered image-text pairs, 14x bigger than LAION-400M, previously the biggest openly accessible image-text dataset in the world
Kylin is a high concurrency, high performance and intelligent OLAP engine that provides low-cost and ultimate data analytics experience.
statistical data visualization.
Seaborn is a Python data visualization library based on matplotlib. It provides a high-level interface for drawing attractive and informative statistical graphics.
A Free 9-Week Course on Data Engineering Fundamentals.
Master the fundamentals of data engineering by building an end-to-end data pipeline from scratch. Gain hands-on experience with industry-standard tools and best practices.
Insights, Unlocked in Real Time.
Apache Pinot™: The real-time analytics open source platform for lightning-fast insights, effortless scaling, and cost-effective data-driven decisions.
Related contents:
Unified stream and batch data processing that's serverless, fast, and cost-effective.
Dolt is Git for data. The world's first and only version-controlled SQL database.
Dolt is a SQL database that you can fork, clone, branch, merge, push and pull just like a Git repository.
Connect to Dolt just like any MySQL database to read or modify schema and data. Version control functionality is exposed in SQL via system tables, functions, and procedures.
Related contents:
Convert Anything into Structured Actionable Data.
Ingest, parse, and optimize any data format ➡️ from documents to multimedia ➡️ for enhanced compatibility with GenAI frameworks.
OmniParse is a platform that ingests and parses any unstructured data into structured, actionable data optimized for GenAI (LLM) applications. Whether you are working with documents, tables, images, videos, audio files, or web pages, OmniParse prepares your data to be clean, structured, and ready for AI applications such as RAG, fine-tuning, and more
🦘 Explore multimedia datasets at scale.
Kangas is a tool for exploring, analyzing, and visualizing large-scale multimedia data. It provides a straightforward Python API for logging large tables of data, along with an intuitive visual interface for performing complex queries against your dataset.
Multi-modal modular data ingestion and retrieval.
DataBridge is an open source library for natural language search and management of multi-modal data. Get started by installing databridge now!
DataBridge is a powerful document processing and retrieval system designed for building intelligent document-based applications. It provides a robust foundation for semantic search, document processing, and AI-powered document interactions.
S3GD is a highly optimized, PyTorch-compatible Triton implementation of the Smoothed SignSGD optimizer, meant for reinforcement learning post-training.
Related contents:
Fetch an entire site and save it as a text file (to be used with AI models).
Official API for https://www.kaggle.com, accessible using a command line tool implemented in Python 3.
Data and AI reliability. Delivered.
Data breaks. Monte Carlo ensures your team is the first to know and solve with end-to-end data observability.
Your data tell a story. Explore. Visualize. Model. Make a difference. Better insight starts with Stata.
Stata is statistical software for data science.
📊 Cube — The Semantic Layer for Building Data Applications. The Universal Semantic Layer.
Build trust with a semantic layer. Connect siloed data, define consistent metrics, and power AI and analytics with context.
Cube is the semantic layer for building data applications. It helps data engineers and application developers access data from modern data stores, organize it into consistent definitions, and deliver it to every application.
Cube was designed to work with all SQL-enabled data sources, including cloud data warehouses like Snowflake or Google BigQuery, query engines like Presto or Amazon Athena, and application databases like Postgres. Cube has a built-in relational caching engine to provide sub-second latency and high concurrency for API requests.
High-Performance Klong array language in Python.
KlongPy is a Python adaptation of the Klong array language, known for its high-performance vectorized operations that leverage the power of NumPy. Embracing a "batteries included" philosophy, KlongPy combines built-in modules with Python's expansive ecosystem, facilitating rapid application development with Klong's succinct syntax.
For better or for worse, LLMs are here to stay. We all read content that they produce online, most of us interact with LLM chatbots, and many of us use them to produce content of our own.
In a series of five- to ten-minute lessons, we will explain what these machines are, how they work, and how to thrive in a world where they are everywhere.
You will learn when these systems can save you a lot of time and effort. You will learn when they are likely to steer you wrong. And you will discover how to see through the hype to tell the difference. ?
Level up your dbt™ skills on your own time with dbt Fundamentals.Learn Analytics Engineering with dbt.
🦉 ML Experiments and Data Management with Git
Data Version Control or DVC is a command line tool and VS Code Extension to help you develop reproducible machine learning projects:
Moses, the machine translation system.
Moses is a statistical machine translation system that allows you to automatically train translation models for any language pair. All you need is a collection of translated texts (parallel corpus). Once you have a trained model, an efficient search algorithm quickly finds the highest probability translation among the exponential number of choices.
A self-contained, lightweight and OOB research platform for modern ML.
Boson is a lightweight, fully containerized, and feature-rich machine learning research platform. It centralizes essential tools to help teams keep projects lean, organized, and reproducible—while reducing overhead and boosting productivity. Think Databricks/Sagemaker but local and free.
Boson enables engineers and researchers to iterate faster without getting bogged down by infrastructure or tooling complexity.
A hyperparameter optimization framework. Optimize Your Optimization.
An open source hyperparameter optimization framework to automate hyperparameter search
The unified data layer
Connect your APIs, databases and microservices to a unified API at the edge. Delight your users with fast response times globally. Deploy globally fast GraphQL APIs with a top-notch developer experience.
Everything to Markdown.
E2M converts various file types (doc, docx, epub, html, htm, url, pdf, ppt, pptx, mp3, m4a) into Markdown. It’s easy to install, with dedicated parsers and converters, supporting custom configs. E2M offers an all-in-one, flexible, and open-source solution.
Welcome to the Python Graph Gallery, a collection of hundreds of charts made with Python. Charts are organized in about 40 sections and always come with their associated reproducible code. They are mostly made with Matplotlib and Seaborn but other library like Plotly are sometimes used.
KNIME Analytics Platform is free and open source, which ensures users remain on the bleeding edge of data science, 300+ connectors to data sources, and integrations to all popular machine learning libraries.
Retrieval Augmented Generation (RAG) chatbot powered by Weaviate.
Welcome to Verba: The Golden RAGtriever, an open-source application designed to offer an end-to-end, streamlined, and user-friendly interface for Retrieval-Augmented Generation (RAG) out of the box. In just a few easy steps, explore your datasets and extract insights with ease, either locally with HuggingFace and Ollama or through LLM providers such as OpenAI, Cohere, and Google.
A minimal dependency implementation of Apache Parquet.
A parser for the Apache Parquet file format, optimized for minimal dependencies and great performance.
Related contents:
An open source alternative to Tableau. Easily embedded as a component in web apps.
Graphic Walker is a different type of open-source alternative to Tableau. It allows data scientists to analyze data and visualize patterns with simple drag-and-drop operations.
Turn websites into LLM-ready data.
Power your AI apps with clean data crawled from any website. It's also open-source. 🔥 Turn entire websites into LLM-ready markdown or structured data. Scrape, crawl and extract with a single API.
Related contents:
Related contents:
Cross-Language Serialization for Relational Algebra. A cross platform way to express data transformation, relational algebra, standardized record expression and plans.
Substrait is a format for describing compute operations on structured data. It is designed for interoperability across different languages and systems.
Open Source Data Labeling.
The most flexible data labeling platform to fine-tune LLMs, prepare training data, or evaluate AI systems. Label Studio is a multi-type data labeling and annotation tool with standardized output format. Label Studio is an open source data labeling tool. It lets you label data types like audio, text, images, videos, and time series with a simple and straightforward UI and export to various model formats. It can be used to prepare raw data or improve existing training data to get more accurate ML models.
The Data Processor for Agents.
Marly allows your agents to extract tables & text from your PDFs, Powerpoints, etc in a structured format making it easy for them to take subsequent actions (database call, API call, creating a chart etc).
Use SQL for everything. Query anything with old-school cool SQL.
Anyquery is a CLI tool to run SQL queries on any data source, no matter if it's a file, an API, logs, or a local app. See the integrations for the full extent of what you can do.
🚀 Async-Powered Pandas.
Lightweight Pandas monkey-patch that adds async support to map, apply, applymap, aggregate, and transform, enabling seamless handling of async functions with controlled max_parallel execution.
Research and data to make progress against the world’s largest problems.
To make progress against the pressing problems the world faces, we need to be informed by the best research and data.
Our World in Data makes this knowledge accessible and understandable, to empower those working to build a better world.
A faster way to build and share data apps. Streamlit turns data scripts into shareable web apps in minutes. All in pure Python. No front‑end experience required.
Streamlit lets you transform Python scripts into interactive web apps in minutes, instead of weeks. Build dashboards, generate reports, or create chat apps. Once you’ve created an app, you can use our Community Cloud platform to deploy, manage, and share your app.
Amplify the Impact of Your People, Expertise & Data.
Altair and RapidMiner share the same vision to make data analytics simple enough for all users, but scalable, governed, and safe enough for all enterprises. RapidMiner is the enterprise-ready data science platform that amplifies the collective impact of your people, expertise and data for breakthrough competitive advantage.
Apache Parquet is an open source, column-oriented data file format designed for efficient data storage and retrieval. It provides high performance compression and encoding schemes to handle complex data in bulk and is supported in many programming languages and analytics tools.
Related contents:
Les technologies numériques sont incroyablement puissantes et redéfinissent le fonctionnement de notre société. Pour les acteurs qui œuvrent pour l'intérêt général, la technologie peut parfois être un levier démutiplicateur d'impacts positifs, cependant et malheureusement ces acteurs n'ont souvent pas les ressources technologiques ou humaines pour accélérer leur action citoyenne. Data for Good existe pour rétablir l'équilibre.
Easy web apps for data science without the compromises. No web development skills required.
Related contents:
NumPy is the fundamental package for scientific computing in Python. It is a Python library that provides a multidimensional array object, various derived objects (such as masked arrays and matrices), and an assortment of routines for fast operations on arrays, including mathematical, logical, shape manipulation, sorting, selecting, I/O, discrete Fourier transforms, basic linear algebra, basic statistical operations, random simulation and much more.
Related contents:
Generate realistic datasets for demos, learning, and dashboards. Instantly preview data, export as CSV or SQL, and explore with Metabase.
Unified Engine for large-scale data analytics.
Apache Spark™ is a multi-language engine for executing data engineering, data science, and machine learning on single-node machines or clusters.
Related contents:
Open and unified metadata platform for data discovery, observability, and governance.
A single place for all your data and all your data practitioners to build and manage high quality data assets at scale. Built by Collate and the founders of Apache Hadoop, Apache Atlas, and Uber Databook.
OpenMetadata is a unified metadata platform for data discovery, data observability, and data governance powered by a central metadata repository, in-depth column level lineage, and seamless team collaboration.
OpenMetadata is a unified metadata platform for data discovery, data observability, and data governance powered by a central metadata repository, in-depth column level lineage, and seamless team collaboration. It is one of the fastest-growing open-source projects with a vibrant community and adoption by a diverse set of companies in a variety of industry verticals. Based on Open Metadata Standards and APIs, supporting connectors to a wide range of data services, OpenMetadata enables end-to-end metadata management, giving you the freedom to unlock the value of your data assets.
Upon first encountering SQL after two decades of Fortran, C, Java, and Python, I thought I had stumbled into hell. I quickly realized that was optimistic: after all, hell has rules.
I have since realized that SQL does too, and that they are no more confusing or contradictory than those of most other programming languages. They only appear so because it draws on a tradition unfamiliar to those of us raised with derivatives of C. To quote Terry Pratchett, it is not mad, just differently sane.
Welcome, then, to a world in which the strange will become familiar, and the familiar, strange. Welcome, thrice welcome, to SQL.
CLI tool that can execute SQL queries on CSV, LTSV, JSON and TBLN. Can output to various formats.
Drasi makes it easy and efficient to detect and react to changes in databases.
Drasi is a data processing platform that simplifies detecting changes in data and taking immediate action. It is a comprehensive solution that provides built-in capabilities to track system logs and change feeds for specific events, evaluate them for relevance, and automatically initiate appropriate reactions.
Related contents:
DataFrames for the new era.
Dataframes powered by a multithreaded, vectorized query engine, written in Rust
Polars is a DataFrame interface on top of an OLAP Query Engine implemented in Rust using Apache Arrow Columnar Format as the memory model.
Polars is an open-source library for data manipulation, known for being one of the fastest data processing solutions on a single machine. It features a well-structured, typed API that is both expressive and easy to use.
Related contents:
OpenVINO™ is an open-source toolkit for optimizing and deploying AI inference.
OpenVINO is an open-source toolkit for optimizing and deploying deep learning models from cloud to edge. It accelerates deep learning inference across various use cases, such as generative AI, video, audio, and language with models from popular frameworks like PyTorch, TensorFlow, ONNX, and more. Convert and optimize models, and deploy across a mix of Intel® hardware and environments, on-premises and on-device, in the browser or in the cloud.
text2vec is an R package which provides an efficient framework with a concise API for text analysis and natural language processing (NLP).
The free AI encyclopedia. AI tools, podcasts, prompts, newsletter, and movies.
MOA is the most popular open source framework for data stream mining,
csvkit is a suite of command-line tools for converting to and working with CSV, the king of tabular file formats.