[Duplicacy] Backing up going forward

It seems creating a robust backup system has become somewhat of a hobby/obsession (hobbsession?) for me. The latest addition to my stable of software is Duplicacy.

Pros

  • Deduplication
    • works across backup sets
    • works across computers
  • Compression
  • Retains revisions with pruning
  • Supports multiple backup threads
  • Command line version is free (GitHub or Homebrew)
  • GUI is fairly cheap (Pricing)
  • Fast
  • Mature product
  • Industrial strength
  • Supports many backends:
    • Local disk, SFTP, Dropbox, Amazon S3, Wasabi, DigitalOcean Spaces, Google Cloud Storage, Microsoft Azure, Backblaze B2, Google Drive, Microsoft OneDrive, Hubic, OpenStack Swift, WebDAV (under beta testing), pcloud (via WebDAV), Box (via WebDAV), File Fabric

Cons

  • Intermediate to advanced skills required

The terminology used in the Duplicacy world is a little different. The things you back up are called repositories, and the places you back up to are called storage.

I’m using the GUI.
The software supports multiple repositories, and multiple storage destinations. Duplicacy stores revisions of backups, which are usually quite small compared to the initial backup, since they only contain changes from the last backup.

As might be expected, backup sets can be created with files or folders explicitly included or excluded, if desired. The number of simultaneous backup threads can be configured too, as well as the use of snapshots of the repository (source) so that backups which represent a point in time are ensured.

The scheduler allows scheduling backup, check, and prune tasks alone or in combination.

  • backup - self explanatory. Can include multiple defined backups.
  • check - performs an integrity check on the storage.
  • prune - removes revisions as they age.
    • The default is to retain all revisions for 7 days
    • One revision a day from 8 to 30 days
    • One revision per week after 30 days

In my case, I am backing up

  • iMac Pro
    • Home folder (464 GiB)
    • External drive called Data (1751 GiB)
  • M1 MacBook Pro
    • Home folder (473 GiB)

The Storage is SSH/SFTP running on my TrueNAS 12 CORE server (free)

That’s a total of 2688 GiB being backed up.
The space occupied on the server (after deduplication and compression) is (401 GiB + 618 GiB + 64 GiB) = 1083 GiB, and checking the folder on the server, it shows 1303 GiB. Not sure where the discrepancy comes from - possibly block size on the ZFS file system. Still, a 50% reduction in size.

This large reduction is due to my having multiple copies of the same dataset on the Data drive, and my home folders on my iMac and MBP are largely the same. Thus, deduplication means that the duplicated blocks in the files, whether on the same or different drives, and on the same or different computers, can be saved only once and reused.

Using Duplicacy to back up to my TrueNAS system with 8 drives in a RAIDZ2 configuration seems like a safe and reliable system.

But just in case, I have others :slight_smile:

I use multiple backup programs (Duplicacy, Carbon Copy Cloner, Arq 5, Backblaze, and even one Time Machine task) to back up to multiple targets (drives that are always attached, drives that live in a drawer and are attached once a month, and SFTP, SMB, and S3-Compatible targets on both my Synology NAS and TrueNAS.) Backup intervals range from hourly to monthly. I also put a drive on the shelf about every two years.

I think I’m pretty well covered for whatever might come up. I might could add a six-month and one-year backup too.

Why? I enjoy thinking through scenarios and strategies, and learning about really cool tech for risk reduction (Duplicacy and ZFS being the latest). I also had an aerospace customer “back in the day” that lost six months worth of work (head crash on the working disk pack, retrieved the backup (crash), retrieved the longer-term backup (crash)). No idea what it cost them in terms of lost work and contract delays.

8 Likes

The de-duplication feature is a nice one. Generally speaking, I keep everything in Dropbox, and only backup Dropbox on one computer to Backblaze.

Thanks for sharing!

3 Likes

After the coming Zombie apocalypse is over, those of us left will be seeking you out to re-install the software that runs the electrical grid in Atlanta. Assuredly, given the diligence of your backup strategy, it is buried somewhere on one of those drives of yours. :slight_smile:


J J Weimer

3 Likes

Something something something Peachtree.

I am trialling Duplicacy as well as Arq - can anyone share the pros and cons between the two as I am going to sign up for one of these two. Obvisoulsy de-duplication feature is a plus for duplicacy.

Arq does deduplication, block level backups, etc. all listed on their page.

I’ve used both a good bit.

Arq

  • Proven
  • Reliable
  • Good support by email
  • Minimal fiddling, set it up and it works
  • Good options for retention (how many backups per day/week/month/months)
  • Includes the GUI
  • Developer’s livelihood depends on it (i.e. skin in the game) (could be true of Duplicacy too)

Duplicacy

  • Command-line version is free
  • GUI is browser-based, and costs $20 for the first year then $5/yr after. Additional computers are $10 first year, $2/yr after.
  • Support for personal edition is via a forum only, response is erratic, and given with expectations that one knows how to, e.g. edit a .json file somewhere (which I do, but still)
  • I had some certificate issues, and other things that were wonky (my forum posts). I don’t want to deal with things like this where backups are concerned, I just want it to work
  • Perhaps built for enterprise people, with personal edition as an afterthought

TL;DR - If you want something reliable with minimal fuss, use Arq. If you want something to fiddle with use Duplicacy.

3 Likes

thanks @JohnAtl , this is a very thorough review / comparison.

1 Like