<?xml version="1.0" encoding="UTF-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
    <title>deduplication</title>
    <link rel="self" type="application/atom+xml" href="https://links.biapy.com/guest/tags/447/feed"/>
    <updated>2026-08-23T23:10:03+00:00</updated>
    <id>https://links.biapy.com/guest/tags/447/feed</id>
            <entry>
            <id>https://links.biapy.com/links/11797</id>
            <title type="text"><![CDATA[Distill]]></title>
            <link rel="alternate" href="https://distill.siddhantkhare.com/" />
            <link rel="via" type="application/atom+xml" href="https://links.biapy.com/links/11797"/>
            <author>
                <name><![CDATA[Biapy]]></name>
            </author>
            <summary type="text">
                <![CDATA[Context window optimization for LLMs.
Reliable LLM outputs start with clean context.

A reliability layer for LLM context. Deterministic deduplication that removes redundancy before it reaches your model.
Less redundant data. Lower costs. Faster responses.
More efficient &amp;amp; deterministic results.

- [Distill @ GitHub](https://github.com/Siddhant-K-code/distill).

Related contents:

- [AI fatigue is real and nobody talks about it @ Siddhant Khare](https://siddhantkhare.com/writing/ai-fatigue-is-real).]]>
            </summary>
            <updated>2026-02-13T09:30:53+00:00</updated>
        </entry>
            <entry>
            <id>https://links.biapy.com/links/11126</id>
            <title type="text"><![CDATA[dragonfire]]></title>
            <link rel="alternate" href="https://gitlab.freedesktop.org/amyspark/dragonfire" />
            <link rel="via" type="application/atom+xml" href="https://links.biapy.com/links/11126"/>
            <author>
                <name><![CDATA[Biapy]]></name>
            </author>
            <summary type="text">
                <![CDATA[dragonfire:

1. reads all the static libraries in one pass
2. deduplicates the object files inside them based on their size and naming (Rust has its own, unique naming convention for object files -- pretty useful!)
3. copies the duplicate objects into a new static library (usually called gstrsworkspace as its primary use is for the GStreamer ecosystem)
4. removes the duplicates from the rest of the libraries
5. updates the symbol table in each of the libraries with the bundled LLVM tools.

Related contents:

- [Linking and shrinking Rust static libraries: a tale of fire @ Centricular](https://centricular.com/devlog/2025-11/dragonfire/).
- [Shrinking while linking @ Tweag](https://www.tweag.io/blog/2025-11-27-shrinking-static-libs/).]]>
            </summary>
            <updated>2025-11-28T13:31:43+00:00</updated>
        </entry>
            <entry>
            <id>https://links.biapy.com/links/268</id>
            <title type="text"><![CDATA[💾🎉 copyparty]]></title>
            <link rel="alternate" href="https://github.com/9001/copyparty" />
            <link rel="via" type="application/atom+xml" href="https://links.biapy.com/links/268"/>
            <author>
                <name><![CDATA[Biapy]]></name>
            </author>
            <summary type="text">
                <![CDATA[turn almost any device into a file server with resumable uploads/downloads using any web browser.

Portable file server with accelerated resumable uploads, dedup, WebDAV, FTP, TFTP, zeroconf, media indexer, thumbnails++ all in one file, no deps.

Related contents:

- [Copyparty - Le serveur de fichiers qui tient dans un seul fichier Python @ Korben :fr:](https://korben.info/copyparty-serveur-fichiers-tient-seul-fichier.html).
- [Host Your Own File Server with Copyparty and Docker @ Noted](https://noted.lol/copyparty/).
- [Ep 63: Running Linux on an IPad @ Linux Matters](https://linuxmatters.sh/63/).]]>
            </summary>
            <updated>2026-03-25T15:21:00+00:00</updated>
        </entry>
            <entry>
            <id>https://links.biapy.com/links/512</id>
            <title type="text"><![CDATA[Immich MediaKit]]></title>
            <link rel="alternate" href="https://github.com/RazgrizHsu/immich-mediakit" />
            <link rel="via" type="application/atom+xml" href="https://links.biapy.com/links/512"/>
            <author>
                <name><![CDATA[Biapy]]></name>
            </author>
            <summary type="text">
                <![CDATA[An extension toolkit for Immich enabling advanced management capabilities through AI-powered similarity detection]]>
            </summary>
            <updated>2025-08-28T17:22:58+00:00</updated>
        </entry>
            <entry>
            <id>https://links.biapy.com/links/2912</id>
            <title type="text"><![CDATA[Duperemove]]></title>
            <link rel="alternate" href="https://markfasheh.github.io/duperemove/" />
            <link rel="via" type="application/atom+xml" href="https://links.biapy.com/links/2912"/>
            <author>
                <name><![CDATA[Biapy]]></name>
            </author>
            <summary type="text">
                <![CDATA[Tools for deduping file systems.

Duperemove is a simple tool for finding duplicated extents and submitting them for deduplication. When given a list of files it will hash their contents on a block by block basis and compare those hashes to each other, finding and categorizing extents that match each other. When given the -d option, duperemove will submit those extents for deduplication using the Linux kernel FIDEDUPERANGE ioctl.

- [Duperemove @ GitHub](https://github.com/markfasheh/duperemove).
- [Episode 588: Clearing out the Tumbleweeds @ Linux Unplugged](https://linuxunplugged.com/588).]]>
            </summary>
            <updated>2025-08-29T00:01:38+00:00</updated>
        </entry>
            <entry>
            <id>https://links.biapy.com/links/2913</id>
            <title type="text"><![CDATA[BEES]]></title>
            <link rel="alternate" href="https://github.com/Zygo/bees" />
            <link rel="via" type="application/atom+xml" href="https://links.biapy.com/links/2913"/>
            <author>
                <name><![CDATA[Biapy]]></name>
            </author>
            <summary type="text">
                <![CDATA[Best-Effort Extent-Same, a btrfs dedupe agent.

bees is a block-oriented userspace deduplication agent designed for large btrfs filesystems. It is an offline dedupe combined with an incremental data scan capability to minimize time data spends on disk from write to dedupe.

- [Episode 588: Clearing out the Tumbleweeds @ Linux Unplugged](https://linuxunplugged.com/588).]]>
            </summary>
            <updated>2025-08-29T00:01:38+00:00</updated>
        </entry>
            <entry>
            <id>https://links.biapy.com/links/4189</id>
            <title type="text"><![CDATA[rmlint]]></title>
            <link rel="alternate" href="https://rmlint.readthedocs.io/en/latest/" />
            <link rel="via" type="application/atom+xml" href="https://links.biapy.com/links/4189"/>
            <author>
                <name><![CDATA[Biapy]]></name>
            </author>
            <summary type="text">
                <![CDATA[Extremely fast tool to remove duplicates and other lint from your filesystem.

rmlint finds space waste and other broken things on your filesystem and offers to remove it.

- [rmlint @ GitHub](https://github.com/sahib/rmlint).
- [Rmlint – Pour optimiser votre espace disque sous Linux @ Korben :fr:](https://korben.info/rmlint-outil-ultime-optimiser-espace-disque-linux.html).]]>
            </summary>
            <updated>2025-08-29T03:35:07+00:00</updated>
        </entry>
            <entry>
            <id>https://links.biapy.com/links/6177</id>
            <title type="text"><![CDATA[BorgBackup]]></title>
            <link rel="alternate" href="https://github.com/borgbackup/borg" />
            <link rel="via" type="application/atom+xml" href="https://links.biapy.com/links/6177"/>
            <author>
                <name><![CDATA[Biapy]]></name>
            </author>
            <summary type="text">
                <![CDATA[Deduplicating archiver with compression and authenticated encryption.
BorgBackup (short: Borg) is a deduplicating backup program. Optionally, it supports compression and authenticated encryption.
The main goal of Borg is to provide an efficient and secure way to back up data. The data deduplication technique used makes Borg suitable for daily backups since only changes are stored. The authenticated encryption technique makes it suitable for backups to targets not fully trusted.]]>
            </summary>
            <updated>2025-08-29T09:06:57+00:00</updated>
        </entry>
            <entry>
            <id>https://links.biapy.com/links/6383</id>
            <title type="text"><![CDATA[Borg]]></title>
            <link rel="alternate" href="https://borgbackup.readthedocs.io/en/stable/" />
            <link rel="via" type="application/atom+xml" href="https://links.biapy.com/links/6383"/>
            <author>
                <name><![CDATA[Biapy]]></name>
            </author>
            <summary type="text">
                <![CDATA[Deduplicating Archiver.
BorgBackup (short: Borg) is a deduplicating backup program. Optionally, it supports compression and authenticated encryption.
The main goal of Borg is to provide an efficient and secure way to backup data. The data deduplication technique used makes Borg suitable for daily backups since only changes are stored. The authenticated encryption technique makes it suitable for backups to not fully trusted targets.

[Borg @ GitHub](https://github.com/borgbackup/borg).]]>
            </summary>
            <updated>2025-08-29T09:41:20+00:00</updated>
        </entry>
    </feed>
