dataset
CredData (Credential Dataset) is a set of files including credentials in open source projects. CredData includes suspicious lines with manual review results and more information such as credential types for each suspicious line.
CredData can be used to develop new tools or improve existing tools. Furthermore, using the benchmark result of the CredData, users can choose a proper tool among open source credential scanning tools according to their use case. We sincerely hope that CredData will help minimize credential leaks.
Related contents:
Apache Parquet is an open source, column-oriented data file format designed for efficient data storage and retrieval. It provides high performance compression and encoding schemes to handle complex data in bulk and is supported in many programming languages and analytics tools.
Related contents:
Open source intelligence for your project. Tools and datasets to support, sustain, and secure critical digital infrastructure.
he world's most comprehensive and accurate dataset about open source production and use, for free
ecosyte.ms has indexed over six billion events across nearly two thousand sources to create the world's most comprehensive and accurate dataset about open source production and use.
Related contents:
The Arc Virtual Cell Atlas is a collection of high quality, curated, open datasets assembled for the purpose of accelerating the creation of virtual cell models. The Atlas includes both observational and perturbational data from over 300 million cells (and growing).
15 trillion tokens of the finest data the 🌐 web has to offer.
The 🍷 FineWeb dataset consists of more than 15T tokens of cleaned and deduplicated english web data from CommonCrawl. The data processing pipeline is optimized for LLM performance and ran on the 🏭 datatrove library, our large scale data processing library.
🍷 FineWeb was originally meant to be a fully open replication of 🦅 RefinedWeb, with a release of the full dataset under the ODC-By 1.0 license. However, by carefully adding additional filtering steps, we managed to push the performance of 🍷 FineWeb well above that of the original 🦅 RefinedWeb, and models trained on our dataset also outperform models trained on other commonly used high quality web datasets (like C4, Dolma-v1.6, The Pile, SlimPajama, RedPajam2) on our aggregate group of benchmark tasks.
Related contents:
Self-hosted street address database
A SQLite3 database file with over 150 million U.S. and Canada address records. Indexed for fast queries, even on fairly slow hardware.
Common Corpus is the largest open and permissible licensed text dataset, comprising over 2 trillion tokens (2,003,039,184,047 tokens). It is a diverse dataset, consisting of books, newspapers, scientific articles, government and legal documents, code, and more.
A NEW ERA OF OPEN LARGE-SCALE MULTI-MODAL DATASETS | LAION.
We present a dataset of 5,85 billion CLIP-filtered image-text pairs, 14x bigger than LAION-400M, previously the biggest openly accessible image-text dataset in the world
The RedPajama-Data repository contains code for preparing large datasets for training large language models. RedPajama-Data: An Open Source Recipe to Reproduce LLaMA training dataset.
Your Machine Learning and Data Science Community.
Inside Kaggle you’ll find all the code & data you need to do your data science work. Use over 50,000 public datasets and 400,000 public notebooks to conquer any analysis in no time.
Keshif is a web-based tool that lets you browse and understand datasets easily.