deduplication
turn almost any device into a file server with resumable uploads/downloads using any web browser.
Portable file server with accelerated resumable uploads, dedup, WebDAV, FTP, TFTP, zeroconf, media indexer, thumbnails++ all in one file, no deps.
Related contents:
An extension toolkit for Immich enabling advanced management capabilities through AI-powered similarity detection
Tools for deduping file systems.
Duperemove is a simple tool for finding duplicated extents and submitting them for deduplication. When given a list of files it will hash their contents on a block by block basis and compare those hashes to each other, finding and categorizing extents that match each other. When given the -d option, duperemove will submit those extents for deduplication using the Linux kernel FIDEDUPERANGE ioctl.
Best-Effort Extent-Same, a btrfs dedupe agent.
bees is a block-oriented userspace deduplication agent designed for large btrfs filesystems. It is an offline dedupe combined with an incremental data scan capability to minimize time data spends on disk from write to dedupe.
Extremely fast tool to remove duplicates and other lint from your filesystem.
rmlint finds space waste and other broken things on your filesystem and offers to remove it.
Deduplicating archiver with compression and authenticated encryption. BorgBackup (short: Borg) is a deduplicating backup program. Optionally, it supports compression and authenticated encryption. The main goal of Borg is to provide an efficient and secure way to back up data. The data deduplication technique used makes Borg suitable for daily backups since only changes are stored. The authenticated encryption technique makes it suitable for backups to targets not fully trusted.
Deduplicating Archiver. BorgBackup (short: Borg) is a deduplicating backup program. Optionally, it supports compression and authenticated encryption. The main goal of Borg is to provide an efficient and secure way to backup data. The data deduplication technique used makes Borg suitable for daily backups since only changes are stored. The authenticated encryption technique makes it suitable for backups to not fully trusted targets.