// the web is temporary
ARCHIVE
EVERYTHING
Servers die. Companies pivot. Links rot. The only copies that survive are the ones somebody kept. This is a field guide for people who keep copies — tools, tactics, and resources for archiving digital things before they disappear.
01Why archive?
Because "it's on the internet, it's there forever" is a myth. The internet forgets constantly, at industrial scale — and every dead platform takes user-made culture down with it. The Google graveyard alone holds hundreds of products.
- 2009
GeoCities. Yahoo pulls the plug and roughly seven million homemade websites vanish — an entire era of the early web. Volunteers at ArchiveTeam raced the shutdown to save what they could.
- 2019
MySpace. A botched server migration silently destroys every song uploaded between 2003 and 2015 — an estimated 50 million tracks from 14 million artists. Most were the only copy.
- 2024
The quiet rot. Pew Research finds that 38% of webpages from 2013 no longer exist. No headline, no shutdown notice — the links just stopped resolving, one by one.
The lesson isn't despair. It's that the only copies that survive are the ones somebody kept. Be somebody.
02Start here
Everything in data hoarding is a variation on one strategy and four rules. Get these right and the rest is just shopping for drives.
copies
of anything you care about. The original counts as one.
different media
or at least different machines — one failure shouldn't reach both.
copy offsite
cloud, tape, or a drive at a friend's place. Fires don't respect RAID.
note: sync is not backup — sync replicates your mistakes at the speed of light.
RAID is not backup
RAID keeps you online when a drive dies. It does nothing against deletion, ransomware, or a mistyped rm -rf. Redundancy is uptime; backup is recovery. You need both, and they are not the same thing.
Test your restores
A backup you've never restored is a rumor. Pull back a random file every month and something big every quarter. The worst time to learn your backups are broken is the only time you need them.
Checksum everything
Bit rot is silent. Hash on ingest (sha256sum), verify on a schedule, scrub if your filesystem supports it, and add PAR2 parity to the truly irreplaceable so damage can be repaired, not just detected.
Prefer open formats
FLAC for audio, PNG or TIFF for images, PDF/A for documents, plain text forever. Formats with public specifications outlive the companies that invented them. Proprietary containers are future regret.
03The toolbox
Free and battle-tested, with a couple of paid exceptions that earn their keep. Almost everything here is open source.
$grab it
— get the data before it's goneyt-dlp ↗
The video downloader. Thousands of supported sites, actively maintained, and --download-archive means you never grab the same thing twice.
gallery-dl ↗
Bulk image and gallery downloader for art sites, boorus, and social media. The yt-dlp of pictures.
wget ↗
The old reliable. Mirroring entire websites with --mirror since 1996. Already installed on half the machines you own.
SingleFile ↗
Browser extension that saves a complete page — styles, images and all — into one self-contained HTML file.
ArchiveBox ↗
Self-hosted web archive: feed it URLs, get back WARC, screenshots, PDF, and readable text. Your personal Wayback Machine.
HTTrack ↗
Point-and-click website copier for offline browsing. Older, but still the gentlest on-ramp to site mirroring.
grab-site ↗
ArchiveTeam's crawler for archiving whole sites into WARC, with a live dashboard and ignore patterns.
$back it up
— copies, encrypted, elsewhererestic ↗
Fast, encrypted, deduplicated backups to local disks or nearly any cloud. Single binary, no drama.
BorgBackup ↗
Deduplicating archiver with compression and encryption. The hoarder's classic for push-style backups.
Kopia ↗
Modern cross-platform backup with a real GUI, snapshots, policies, and cloud targets. Friendly without being weak.
rclone ↗
rsync for cloud storage: sync, mount, and encrypt across 70+ providers. The duct tape of the hoard.
Syncthing ↗
Continuous peer-to-peer file sync between your own machines. No cloud middleman, no account, no fees.
FreeFileSync ↗
Visual folder comparison and sync for manual mirror jobs and offline drive rotations.
$store it
— where the hoard livesZFS ↗
The filesystem that checksums every block and heals itself on scrub. Bit rot's natural predator.
TrueNAS ↗
Battle-tested NAS operating system built on ZFS. Turn any box full of drives into serious storage.
Unraid ↗
Flexible NAS OS that lets you mix drive sizes and grow one disk at a time. Paid, but beloved for a reason.
SnapRAID ↗
Snapshot parity for JBOD arrays of mismatched drives. Ideal for media that rarely changes.
mergerfs ↗
Pool many drives into one big mount point. Pairs with SnapRAID like drives pair with regret.
smartmontools ↗
Watch your drives' S.M.A.R.T. data and catch failures while they're still just reallocated sectors.
$verify & manage
— trust nothing, checksum everythingpar2cmdline ↗
Create PAR2 parity files that can repair damaged data, not just detect the damage. Insurance for the irreplaceable.
MultiPar ↗
Windows GUI for PAR2 parity creation and verification. Same insurance, more buttons.
Czkawka ↗
Blazing-fast duplicate finder — for deduping the hoard, or proving you'd never download anything twice.
TeraCopy ↗
Copy files on Windows with checksum verification baked in. Explorer's copy dialog, but trustworthy.
Everything ↗
Instant filename search across millions of files on Windows. If you can't find it, you don't have it.
Calibre ↗
The ebook library manager. Converts, catalogs, and de-DRMs your way to a permanent personal library.
04The public record
Personal hoards save your stuff. These projects save everyone's. Use them, support them, and feed them URLs.
Wayback Machine ↗
The Internet Archive's time machine: 900+ billion pages. Save any URL right now at web.archive.org/save — it takes ten seconds and lasts decades.
archive.today ↗
Snapshot service that captures pages exactly as rendered — handy for dynamic pages and things the Wayback crawler can't reach.
ArchiveTeam ↗
The volunteer brigade rescuing dying websites since 2009. Their wiki tracks the deathwatch; their Warrior VM lets you donate bandwidth to the rescue.
Webrecorder ↗
High-fidelity capture of complex, scripted pages straight from your browser — with ArchiveWeb.page to record and ReplayWeb.page to play back.
Kiwix ↗
All of Wikipedia (and much more) as offline files. The whole encyclopedia fits on a microSD card. Apocalypse-grade knowledge storage.
Support the Archive ↗
The Internet Archive is a non-profit library holding a meaningful fraction of human digital culture. Throw them a few dollars; it's the cheapest storage you'll ever buy.
05Community & field data
Nobody hoards alone. Where to ask questions, watch for dying sites, and decide which drive to buy next.
r/DataHoarder ↗
The mothership. Hundreds of thousands of people who also just bought another drive. The wiki alone is worth the visit.
awesome-datahoarding ↗
Community-curated mega-list of hoarding tools, guides, and resources. When this page isn't enough, that one is.
Backblaze Drive Stats ↗
Quarterly failure rates from a fleet of hundreds of thousands of spinning drives. Real reliability data — know before you buy.
diskprices.com ↗
Live price-per-terabyte tracker across drive types and capacities. The only stock ticker that matters here.
06Field notes
Hard-won rules from people who learned them the expensive way.
- Download it now.
The tab will be dead before you get back to it. Content disappears faster than you procrastinate — grab first, sort later (or, realistically, never).
- Two is one, one is none.
A single copy is a countdown timer. If it exists in one place, it's already halfway to not existing at all.
- A backup you've never restored is a rumor.
Verification is the difference between a backup strategy and a comforting ritual.
- Bit rot doesn't announce itself.
Checksum on the way in, verify on a schedule, scrub if you can. Silent corruption is only silent until you open the file.
- The cloud is someone else's computer.
With someone else's terms of service, someone else's pricing changes, and someone else's shutdown timeline. Fine as your offsite copy; reckless as your only one.
- If you can't find it, you don't have it.
An unindexed hoard is a landfill with good intentions. Name things consistently and keep a searchable index.
- Label your drives.
Future-you has no idea what's on
disk7-new-FINAL, and future-you is the only person who can be trusted with the hoard. - LOCKSS: lots of copies keep stuff safe.
Mirror what you love and share it forward. Preservation is a team sport played across decades.