s3
Streaming SQLite Replication.
Stop building slow, complex, fragile software systems. Safely run your application on a single server.
Litestream is a standalone disaster recovery tool for SQLite. It runs as a background process and safely replicates changes incrementally to another file or S3. Litestream only communicates with SQLite through the SQLite API so it will not corrupt your database.
Related contents:
Dockerized web app for easily making clips from your library of movies using .srt subtitle files and sharing quickly via an S3 endpoint
Hybrid Cache in Rust. Seamlessly integrates both in-memory and disk cache for optimal performance and flexibility.
foyer aims to be an efficient and user-friendly hybrid cache lib in Rust. foyer draws inspiration from Facebook/CacheLib, a highly-regarded hybrid cache library written in C++, and ben-manes/caffeine, a popular Java caching library, among other projects.
Related contents:
The Filesystem That Makes S3 your Primary Storage. ZeroFS is 9P/NFS/NBD on top of S3.
ZeroFS makes S3 storage feel like a real filesystem. It provides file-level access via NFS and 9P and block-level access via NBD. Fast enough to compile code on, with clients already built into your OS. No FUSE drivers, no kernel modules, just mount and go.
Read-through cache for object storage.
High-performance read-through cache for object storage.
Store securely encrypted backups in the cloud!
Duplicati is a free, open-source backup client that securely stores encrypted, incremental, and compressed backups on cloud storage services and remote file servers.
Related contents:
A high-performance Python library for fast, concurrent S3 object downloads.
🌩 Self-hosted file management and sharing system, supports multiple storage providers.
☁️ Support storing files into Local, Remote node, OneDrive, S3 compatible API, Qiniu, Aliyun OSS, Tencent COS, Upyun.
chibisafe is a beautiful and performant vault to save all your files in the cloud.
A modern and self-hosted take on file uploading services that can handle anything you throw at it thanks to it's robust and fast API, chunked uploads support and more.
Lightweight selfhosted Firefox Send alternative without public upload. AWS S3 supported.
Gokapi is a simple, self-hosted file sharing server with automatic expiration and encryption support — ideal for teams or individuals who want privacy, control, and no clutter.
Related contents:
Your unified cloud storages interface.
Sourcerer is a CLI-based cloud storage explorer that provides a unified interface for developers and DevOps engineers to view and manage files across multiple cloud providers like GCP Storage, Azure Storage, AWS S3, and S3-compatible services.
Tiny & fast S3 client for node and edge platforms.
s3mini is an ultra-lightweight Typescript client (~14 KB minified, ≈15 % more ops/s) for S3-compatible object storage. It runs on Node, Bun, Cloudflare Workers, and other edge platforms. It has been tested on Cloudflare R2, Backblaze B2, DigitalOcean Spaces, and MinIO. (No Browser support!)
An Amazon S3-compatible object storage server that lets you build and integrate your apps faster and store your data anywhere.
Zenko CloudServer, an open-source Node.js implementation of the Amazon S3 protocol on the front-end and backend storage capabilities to multiple clouds, including Azure and Google.
Related contents:
WinCse is an application that integrates AWS S3 buckets with Windows Explorer, allowing you to treat S3 buckets as if they were part of your local file system.
Related contents:
OpenDAL represents Open Data Access Layer. Our vision is One Layer, All Storage.
OpenDAL (/ˈoʊ.pən.dæl/, pronounced "OH-puhn-dal") is an Open Data Access Layer that enables seamless interaction with diverse storage services.
OpenDAL's development is guided by its vision of One Layer, All Storage and its core principles: Open Community, Solid Foundation, Fast Access, Object Storage First, and Extensible Architecture.
Related contents:
A Cloud Native Distributed Storage System.
CubeFS is a new generation cloud-native open source storage system that supports access protocols such as S3, HDFS, and POSIX. It is widely applicable in various scenarios such as big data, AI/LLMs, container platforms, separation of storage and computing for databases and middleware, data sharing and protection，etc.
Related contents:
Raft is a consensus algorithm that is designed to be easy to understand. It's equivalent to Paxos in fault-tolerance and performance. The difference is that it's decomposed into relatively independent subproblems, and it cleanly addresses all major pieces needed for practical systems. We hope Raft will make consensus available to a wider audience, and that this wider audience will be able to develop a variety of higher quality consensus-based systems than are available today.
Related contents:
Fluent OpenStack client for Java.
OpenStack4j is a fluent OpenStack client that allows provisioning and control of an OpenStack deployment. This includes support for Identity, Compute, Image, Network, Block Storage, Telemetry, Data Processing as well as many extensions (LBaaS, FWaaS, Quota-Sets, etc)
Agile MySQL with S3-Based Storage. Improving the popular MySQL to be more developer-friendly and easier to manage, streamlining your app lifecycle.
WeSQL is an innovative MySQL distribution that adopts a compute-storage separation architecture, with storage backed by S3 (and S3-compatible systems). It can run on any cloud, ensuring no vendor lock-in.
The best way of working with Protocol Buffers. Elastic, self-hosted Kafka with Advanced Semantic Intelligence Guarantee streaming data quality and slash cloud costs 10x with Bufstream, a drop-in replacement for Apache Kafka®.
Bufstream is a Kafka-compatible streaming system which stores records directly in an object storage service like S3.
Self-hosted LiveSync is a community-implemented synchronization plugin, available on every obsidian-compatible platform and using CouchDB or Object Storage (e.g., MinIO, S3, R2, etc.) as the server.
Better Upload is a file upload library for React that makes it very simple and easy to upload files directly to any S3-compatible service. It handles both the client and your server.
While Better Upload is designed to work with Next.js, it can also work with any framework / backend server that uses standard Request and Response objects. Like Remix and Hono!
S3oosh allows users to upload multiple files at once to S3 Buckets. It provides a drag-and-drop interface for users to easily upload files to a S3 Bucket. The component supports various file types and allows users to set maximum file count, maximum file size, and accepted file types.
Command Line S3 Client and S3 Backup for Windows, Linux: s3cmd, s3express.
S3cmd is a free command line tool and client for uploading, retrieving and managing data in Amazon S3 and other cloud storage service providers that use the S3 protocol, such as Google Cloud Storage or DreamHost DreamObjects. It is best suited for power users who are familiar with command line programs. It is also ideal for batch scripts and automated backup to S3, triggered from cron, etc.
Source Available Reinvented Kafka. 10x Cost Efficiency.
AutoMQ is a cloud-first alternative to Kafka by decoupling durability to S3 and EBS. 10x cost-effective. Autoscale in seconds. Single-digit ms latency.
AutoMQ is a stateless Kafka on S3. 10x Cost-Effective. No Cross-AZ Traffic Cost. Autoscale in seconds. Single-digit ms latency. Multi-AZ Availability.
Related contents:
An embedded database built on object storage.
Unlike traditional LSM-tree storage engines, SlateDB writes data to object storage to provide bottomless storage capacity, high durability, and easy replication.
SlateDB is an embedded storage engine built as a log-structured merge-tree. Unlike traditional LSM-tree storage engines, SlateDB writes data to object storage (S3, GCS, ABS, MinIO, Tigris, and so on). Leveraging object storage allows SlateDB to provide bottomless storage capacity, high durability, and easy replication. The trade-off is that object storage has a higher latency and higher API cost than local disk.
A Little Tiny (S3 compatible) file storage server
The free SaaS template with superpowers
A free, open-source SaaS app starter for React & Node.js with superpowers. Production-ready. Community-driven.
Open Source Distributed POSIX File System for Cloud. JuiceFS is a distributed POSIX file system built on top of Redis and S3.
JuiceFS is a high-performance POSIX file system released under Apache License 2.0, particularly designed for the cloud-native environment. The data, stored via JuiceFS, will be persisted in Object Storage (e.g. Amazon S3), and the corresponding metadata can be persisted in various compatible database engines such as Redis, MySQL, and TiKV based on the scenarios and requirements.
With JuiceFS, massive cloud storage can be directly connected to big data, machine learning, artificial intelligence, and various application platforms in production environments. Without modifying code, the massive cloud storage can be used as efficiently as local storage.
SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, cross-DC active-active replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding.
An open-source distributed storage service you can self-host to fullfill many needs.
Garage is an S3-compatible distributed object storage service designed for self-hosting at a small-to-medium scale.
Garage is designed for storage clusters composed of nodes running at different physical locations, in order to easily provide a storage service that replicates data at these different locations and stays available even when some servers are unreachable. Garage also focuses on being lightweight, easy to operate, and highly resilient to machine failures.
Related contents:
Backup multiple database types on a scheduled basis with many customizable options. This will build a container for backing up multiple types of DB Servers.
Backs up CouchDB, InfluxDB, MySQL/MariaDB, Microsoft SQL, MongoDB, Postgres, Redis servers.
Your Data Pipeline, Simplified. GlareDB: An analytics DBMS for distributed data.
Data exists everywhere: your laptop, Postgres, Snowflake and as files in S3. It exists in various formats such as Parquet, CSV and JSON. Regardless, there will always be multiple steps spanning several destinations to get the insights you need.
GlareDB is designed to query your data wherever it lives using SQL that you already know.
Easy file sharing from the command line.
Easy and fast file sharing from the command-line. This code contains the server with everything you need to create your own instance. Transfer.sh currently supports the s3 (Amazon S3), gdrive (Google Drive), storj (Storj) providers, and local file system (local).
Parallel S3 and local filesystem execution tool.
s5cmd is a very fast S3 and local filesystem execution tool. It comes with support for a multitude of operations including tab completion and wildcard support for files, which can be very handy for your object storage workflow while working with large number of files.
Fully featured and highly configurable SFTP server with optional HTTP/S, FTP/S and WebDAV support. Several storage backends are supported: local filesystem, encrypted local filesystem, S3 (compatible) Object Storage, Google Cloud Storage, Azure Blob Storage, SFTP.
Duplicati is a free, open source, backup client that securely stores encrypted, incremental, compressed backups on cloud storage services and remote file servers.
simple, high-throughput file client for mounting an Amazon S3 bucket as a local file system. Mountpoint for Amazon S3 is a simple, high-throughput file client for mounting an Amazon S3 bucket as a local file system. With Mountpoint for Amazon S3, your applications can access objects stored in Amazon S3 through file operations like open and read. Mountpoint for Amazon S3 automatically translates these operations into S3 object API calls, giving your applications access to the elastic storage and throughput of Amazon S3 through a file interface.
Cross-platform backup tool for Windows, macOS & Linux with fast, incremental backups, client-side end-to-end encryption, compression and data deduplication. CLI and GUI included.
Kopia is a fast and secure open-source backup/restore tool that allows you to create encrypted snapshots of your data and save the snapshots to remote or cloud storage of your choice, to network-attached storage or server, or locally on your machine. Kopia does not 'image' your whole machine. Rather, Kopia allows you to backup/restore any and all files/directories that you deem are important or critical.
Ceph is a distributed object, block, and file storage platform.
Reliable and scalable storage designed for any organization. Use Ceph to transform your storage infrastructure. Ceph provides a unified storage service with object, block, and file interfaces from a single cluster built from commodity hardware components.
horizontally scalable, highly available, multi-tenant, long term storage for Prometheus.
Rclone is a command-line program to manage files on cloud storage. It is a feature-rich alternative to cloud vendors' web storage interfaces. Over 40 cloud storage products support rclone including S3 object stores, business & consumer file storage services, as well as standard transfer protocols.
High Performance, Kubernetes Native Object Storage. MinIO offers high-performance, S3 compatible object storage. Native to Kubernetes, MinIO is the only object storage suite available on every public cloud, every Kubernetes distribution, the private cloud and the edge. MinIO is software-defined and is 100% open source under GNU AGPL v3.
DodgerCMS is a static markdown CMS built on top of Amazon S3. It is a clean and simple alternative to heavy content management systems. There are no databases to manage, deployments to monitor, or massive configuration files. Just focus on writing your content and the results are live immediately.
The only requirement for DodgerCMS is an account with Amazon Web Services. It is well suited for small blogs, documentation, and any static website that benefits from the simplicity of markdown. Content is uploaded into the manager and then converted to HTML each time the document is updated. DodgerCMS also keeps a data JSON file with up to date URL's for each entry that can be used to build a menu system.