Cobra's bits (Posts about backup)

Android file transfer

Cobra — Fri, 24 Dec 2021 10:32:48 GMT

My home and office computers are synchronized via the ownCloud server located at my workplace. This synchronization takes place via an TLS encrypted connection with an A+ Qualys rating. In addition, I encrypt files containing sensitive information prior to their transfer on an individual basis. Since all systems accessing this cloud folder are driven by an operating system (OS) that I trust and largely control (Archlinux), I feel very comfortable regarding the security and privacy of my data. To keep this warm and cozy feeling, I wouldn't give devices with an OS beyond my control (such as Windows, MacOS, Android, iOS) access to this folder.

Now, I fully appreciate that even a hardened desktop Linux would have difficulties to compete with the level of security offered by an up-to-date Android – with “security” being defined here in the conventional context with respect to a potential third-party adversary. But concerning the privacy of my data, and thus mine, the threat of an overly nosy first party is much more palpable. Actually, I shouldn't call it a threat. It's in fact a promise.

How, then, am I supposed to transfer or even better synchronize data from and with the newest member of my gadget zoo? Since I've acquired this gadget as a two-factor authentication for my banks, I cannot simply root it and install LineageOS without any Google services. Therefore, I won't trust the device beyond its specialized purpose and I won't give it access to my cloud folder.

But that's actually not a big deal in this case. Because of its specific function as a two-factor authenticator for my banks and several other services, the phone will remain stationary. Hence, I need to synchronize within my LAN, but not outside of it. When looking for apps that would be suitable for this task, I was initially attracted by those appealing to the nerd in me, such as, for example, croc installed (pkg install croc) and running within termux, or juiceSSH. In the end, these apps turned out to be fun for a limited time, but too tedious for everyday use. I'm very fond of controlling computers with a keyboard, but for using termux efficiently, you'd need the eyes of an eagle, the fingers of an elf and the dexterity of a spider monkey.

For ordinary human beings, syncthing is the tool of choice. It's available on f-droid, easy to set up on all participating systems, and works reliably without manual intervention. In my case, I've simply created a folder (~/androidshare) on my desktop that automatically receives all files from my phone that may be worth to keep, including the backups of the andOTP and keepassDX databases and all photographs of my cats. 😍

A simple automated backup scheme

Cobra — Fri, 21 Dec 2018 14:41:21 GMT

I haven't talked about backups since more than two years, but as detailed in my previous post, I was recently most emphatically reminded that it isn't clear to most people that their data can vanish anytime. Anytime? Anytime.

I've detailed what I expect from a backup solution and what I've used over the years in several previous posts (see one, two, and three) and do not need to repeat that here. My current backup scheme is based on borg with support from rsync and ownCloud. The following chart (which looks way more complicated than it is) visualizes this scheme and is explained below.

My home installation is on the left, separated from my office setup on the right by the dashed line in the center. The desktops (top) are equipped with an SSD containing the system and home partitions as well as an HDD for archiving, backup, and caching purposes. Active projects are stored in a dedicated ownCloud folder and synchronized between all clients, including my notebooks. The synchronization is done via the ownCloud server (the light gray box at the lower right) and is denoted by (1) in the chart. The server keeps a number of previous versions of each file, and in case a file is deleted, the latest version is kept, providing a kind of poor man's backup. But I'm using ownCloud to sync my active projects between my machines, not for its backup features.

The actual backup from the SSD to the HDD is denoted by (2) or by (2') for my notebooks (to the same HDD, but over WiFi), and is done by borg. The backup is then transferred to a NAS (the gray boxes at the bottom) by simple rsync scripts indicated by (3). All of that is orchestrated with the help of a few crontab entries – here's the one for my office system as an example:

$ crontab -l
15  7-21        *   *   *       $HOME/bin/backup.borg
30  21          *   *   *       source $HOME/.keychain/${HOSTNAME}-sh;  $HOME/bin/sync.backup
30  23          *   *   *       $HOME/bin/sync.vms
30  02          *   *   *       $HOME/bin/sync.archive

The first entry invokes my borg backup script (see below) every hour between 7:15 and 21:15 (I'm highly unlikely to work outside of this time interval). The second entry (see also below) takes care of transferring the entire backup folder on the HDD 15 min after the last backup to the NAS. Since my rsync script invokes ssh for transport, I use keychain to inform cron about the current values of the environment variables SSH_AUTH_SOCK and SSH_AGENT_PID. The third entry induces the transfer of any changed virtual machine to the internal HDD. And finally, the fourth entry syncs the archive on the internal HDD to an external one (3'). I do that since once a project is finished, the corresponding folder is moved out of the ownCloud folder to the archive, effectively taking it out from the daily backup. This way, the size of my ownCloud folder never increased beyond 3.5 GB over the past five years. Since the projects typically don't change anymore once they are in the archive, this step effectively just creates a copy of the archive folder.

What's not shown in the chart above: there's a final backup level involving the NAS. At home I do that manually (and thus much too infrequently) by rsyncing the NAS content to the HDDs I've collected over the years. ` 😉 At the office, the NAS is backed up to tape every night automatically. The tapes are part of a tape library located in a different building (unfortunately, too close to survive a nuclear strike 😉 ), and are kept ten years as legally required.

What does all of that mean when I prepare an important document such as a publication or a patent? Well, let's count. It doesn't matter where I start working: since the document is in my ownCloud folder, it is soon present on five different disks (two desktops and notebooks, and the ownCloud server). The backup and sync adds four more disks, and the final backup of the NAS results in two more copies (one on disk, one on tape). Altogether, within one day, my important document is automatically duplicated to ten different storage media (disks or tape) in three different locations. And when I continue working on this document the next days, my borg configuration (see below) keeps previous copies up to six month in the past (see cron mail below).

You're probably thinking that I'm a complete paranoid. 10 different storage media in 3 different locations! Crazy! Well, the way I do it, I get this kind of redundancy and the associated peace of mind for free. See for yourself:

(1) ownCloud

My employee runs an ownCloud server. I just need to install the client on all of my desktops and notebooks. If you are not as lucky: there are very affordable ownCloud or nextCloud (recommended) servers available in the interwebs.

(2) backup.borg

A simple shell script:

#!/bin/bash

#`https://github.com/borgbackup/borg <https://github.com/borgbackup/borg>`_
#`https://borgbackup.readthedocs.io/en/stable/index.html <https://borgbackup.readthedocs.io/en/stable/index.html>`_

ionice -c3 -p$$

repository="/bam/backup/attic" # directory backing up to
excludelist="/home/ob/bin/exclude_from_attic.txt"
hostname=$(echo $HOSTNAME)

notify-send "Starting backup"

          borg create --info --stats --compression lz4                  \
          $repository::$hostname-`date +%Y-%m-%d--%H:%M:%S`             \
          /home/ob                                                      \
          --exclude-from $excludelist                                   \
          --exclude-caches

notify-send "Backup complete"

          borg prune --info $repository --keep-within=1d --keep-daily=7 --keep-weekly=4 --keep-monthly=6

          borg list $repository

(3) sync.backup

A simple shell script:

#!/bin/bash

# `http://everythinglinux.org/rsync/ <http://everythinglinux.org/rsync/>`_
# `http://troy.jdmz.net/rsync/index.html <http://troy.jdmz.net/rsync/index.html>`_

ionice -c3 -p$$

RHOST=nas4711

BUSPATH=/bam/backup/attic
BUDPATH=/home/users/brandt/backup

nice -n +10 rsync -az -e 'ssh -l brandt' --stats --delete $BUSPATH $RHOST:$BUDPATH

The two other scripts in the crontab listing above are entirely analogous to the one above.

That's all. It's just these scripts and the associated crontab entries above, nothing more. And since (2) and (3) are managed by cron, I'm informed about the status of my backup every time one is performed. The list of entries you see in the mail below are the individual backups I could roll back to, or just copy individual files from after mounting the whole caboodle with 'borg mount -v /bam/backup/attic /bam/attic_mnt/' (see the screenshot below). You see how these backups are organized: hourly for the last 24h, daily for the last week, weekly for the past month, and monthly for the five months after.

From: "(Cron Daemon)" <ob@pdi282>
Subject: Cron <ob@pdi282> $HOME/bin/backup.borg

------------------------------------------------------------------------------
Archive name: pdi282-2018-12-21--14:15:01
Archive fingerprint: d31db1cd8223ca084cc367deb62e440bfe2dfe4fd163aefc6b6294935f1877b8
Time (start): Fri, 2018-12-21 14:15:01
Time (end):   Fri, 2018-12-21 14:15:21
Duration: 19.52 seconds
Number of files: 173314
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
                                           Original size      Compressed size    Deduplicated size
This archive:               30.14 GB             22.17 GB              1.40 MB
All archives:              975.54 GB            715.02 GB             43.41 GB

                                           Unique chunks         Total chunks
Chunk index:                  216890              6892358
------------------------------------------------------------------------------
pdi282-2018-06-30--21:15:01          Sat, 2018-06-30 21:15:01 [113186c574898837d0fb11e6fb7b71f62b0a5422d71b627662aec0d2d6a0e0bf]
pdi282-2018-07-31--21:15:01          Tue, 2018-07-31 21:15:01 [8af0cccbab5645490fcec5e88576dad1a3fbbfd3d726a35e17851d7bec545958]
pdi282-2018-08-31--21:15:01          Fri, 2018-08-31 21:15:01 [2d763ea253d18222015d124c48826425e75b83efedeedcc11b24cf8f0d7e8899]
pdi282-2018-09-30--21:15:01          Sun, 2018-09-30 21:15:01 [39932a0d8c081bc05f9cdff54637e3962fd9e622edce8ef64160e79ae767541f]
pdi282-2018-10-31--21:15:01          Wed, 2018-10-31 21:15:02 [49386980b5270554c6c92b8397809736dea5d07c7ccb3861187a6ed5065ba7a6]
pdi282-2018-11-18--21:15:01          Sun, 2018-11-18 21:15:02 [c2eb215ce883fa5a0800a9d4b9a6c53ac82ace48151180e6a15e944dbf65e009]
pdi282-2018-11-25--21:15:01          Sun, 2018-11-25 21:15:01 [e99c2f3baed4a863b08551605eb8ebeaa5ed6a02decccdb88268c89a9b9b9cc0]
pdi282-2018-11-30--21:15:01          Fri, 2018-11-30 21:15:01 [882f6466adcbc43d7e1a12df5f38ecc9b257a436143b00711aa37e16a4dbf54d]
pdi282-2018-12-02--21:15:02          Sun, 2018-12-02 21:15:02 [7436da61af62faf21ca3f6aeb38f536ec5f1a4241e2d17c9f67271c3ba76c188]
pdi282-2018-12-09--21:15:01          Sun, 2018-12-09 21:15:02 [82e6c845601c1a12266b0b675dfeaee44cd4ab6f33dafa981f901a3e84567bbb]
pdi282-2018-12-13--21:15:01          Thu, 2018-12-13 21:15:01 [9ac3dfd4aca2e56df8927c7bc676cd476ea249f4dd2c1c39fc2a4997e0ada896]
pdi282-2018-12-14--21:15:02          Fri, 2018-12-14 21:15:02 [c8c1358f58dae6eb28bd66e9b49c7cfe237720de21214ebd99cc4b4964ec9249]
pdi282-2018-12-15--21:15:01          Sat, 2018-12-15 21:15:01 [e24d3b26dcdf81d0b0899085fb992c7a7d33d16671fba7a2c9ef1215bd3ae8fb]
pdi282-2018-12-16--21:15:01          Sun, 2018-12-16 21:15:01 [27a8a6943f1053d106ced8d40848eccbfb6c145d80d5e2a9e92f891ed98778ce]
pdi282-2018-12-17--21:15:02          Mon, 2018-12-17 21:15:02 [14118ea958387e0a606e9f627182e521b92b4e2c2dd9fb5387660b84a08971a6]
pdi282-2018-12-18--21:15:01          Tue, 2018-12-18 21:15:01 [842c3f7e301de89944d8edf7483956aff2b7cf9e15b64b327f476464825bd250]
pdi282-2018-12-19--21:15:01          Wed, 2018-12-19 21:15:01 [b7f99c56a8e6ee14559b3eddec04646c8a756515765db562c35b8fbefcd4e58e]
pdi282-2018-12-20--15:15:01          Thu, 2018-12-20 15:15:01 [e832afd41762a69cb8c5fe1c14395dde313dc4368871fd27073fdc50e9f7c6c9]
pdi282-2018-12-20--16:15:01          Thu, 2018-12-20 16:15:01 [8471ccb87d513604d31320ff91c2e0aaf0d31e5ff908ff41b8653c55ee11c1e5]
pdi282-2018-12-20--17:15:01          Thu, 2018-12-20 17:15:01 [73a3ae72815a10732fc495317a7e0f8cd9d05eb2ea862f8c01b437138ac82103]
pdi282-2018-12-20--18:15:01          Thu, 2018-12-20 18:15:01 [7eced8e18b52d00300c8f1b17e188fbfc1124dc60adf68ef2924425677615a96]
pdi282-2018-12-20--19:15:01          Thu, 2018-12-20 19:15:01 [6b7dbc4095b704209921424a52ed37d854b3a61c49cc65ac6889d215aad95a6f]
pdi282-2018-12-20--20:15:01          Thu, 2018-12-20 20:15:01 [66da0f57d6c93b149a9fdf679acf5e43fc22ce6b582db4da3ab606df741bdf82]
pdi282-2018-12-20--21:15:01          Thu, 2018-12-20 21:15:01 [1fce9aa4751be905a45ccce7fca3d44be3cf580d5e4b7c4f5091167099df57ad]
pdi282-2018-12-21--07:15:01          Fri, 2018-12-21 07:15:02 [ee551653a18d400719f9ffe1a67787326f5d5dad41be7d7b5482d5610ed86d43]
pdi282-2018-12-21--08:15:01          Fri, 2018-12-21 08:15:01 [264d7ce1dab3bc1578b521a170ee944598fa99f894d6ca273793ad14824b1689]
pdi282-2018-12-21--09:15:01          Fri, 2018-12-21 09:15:01 [b37de3616438e83c7184af57080690db3a76de521e77fd1ae6e90262f6beb1cc]
pdi282-2018-12-21--10:15:01          Fri, 2018-12-21 10:15:01 [6862d0136b2e4ac7fc0544eb74c0085e7baceca7147bd59b13cd68cbf00cb089]
pdi282-2018-12-21--11:15:01          Fri, 2018-12-21 11:15:01 [e5c6ee4ea65d6dacb34badb850353da87f9d5c19bb42e4fb3b951efecd58e64f]
pdi282-2018-12-21--12:15:01          Fri, 2018-12-21 12:15:01 [5b93f864b9422ed953c1aabb5b1b98ce9ae04fe2f584c05e91b87213082e2ff0]
pdi282-2018-12-21--13:15:01          Fri, 2018-12-21 13:15:01 [461f976422c45a7d10d38d1db097abd30a4885181ec7ea2086d05f0afd9169eb]
pdi282-2018-12-21--14:15:01          Fri, 2018-12-21 14:15:01 [d31db1cd8223ca084cc367deb62e440bfe2dfe4fd163aefc6b6294935f1877b8]

Here's a screenshot of nemo running on my notebook with an sftp connection to my office desktop, after having mounted the available backups with the command given above.

The age of (digital) decline

Cobra — Mon, 17 Dec 2018 20:09:08 GMT

About a decade ago, AppleInsider presented an enthusiastic report on the latest innovation from the iPhone inventor:

Apple is dramatically rethinking how applications organize their documents on iPad, leaving behind the jumbled file system [...].

Outside of savvy computer users, the idea of opening a file by searching through hierarchical paths in the file system is a bit of a mystery.

Apple has already taken some steps to hide complexity in the file system in Mac OS X, [...] the iPhone similarly abstracts away the file system entirely; there is no concept of opening or saving files.

I remember reading that and being very sceptical, perhaps as one of a few, but not the only one:

Heck, my 6 year old daughter can understand the idea of saving some files to a folder with her name on it, and others to different locations.

Another lone voice in the digital wilderness:

While this might sound like some kind of user experience utopia, I have a grave concern that eliminating a file system in this manner misses a huge audience. Us.

Now, almost 10 years later, we begin to pay the price for this development. How's that? Well, my experience shows that users who grew up with iOS or Android as their prime computing environment have difficulties to grasp the basic paradigms that still dominate professional work with computers. In particular, only few young users seem to understand the concept of a file (yes, a file, but see above), file types, and file systems. Even less understood is the client-server model, a concept that is indispensable in a modern IT infrastructure.

Consequences of this erosion of knowledge range from the comical to the disastrous. As an example for the former: when I ask for original data I do not mean ASCII data embedded in an MS Word document and a photo- or micrograph embedded in a Powerpoint presentation. However, many young users do not know that a 'data.dat' or a 'photograph.tiff' are valid file formats that can be viewed and edited by suitable applications. A secretary at an associated university had the opposite tendency: she wrote invitations for seminars with Word, printed them, scanned the printout with 1200 dpi, and attached the resulting 100 MB bitmap to electronic invitations sent by e-mail.

That's funny if your e-mail account has no size limit. But even if, you may see that this development has also much less amusing consequences. On a very general level, these users are incapable to appropriately interact with professional IT infrastructures, including common desktop environments (regardless of their provenience). More specifically, users with these deficiencies should not be trusted with handling and managing important data at all. Because ... they will lose them.

At least that's what's happening here: the number of users who experience a total loss of their data increased rapidly over the past few years. In most cases, the cause was not negligence and carelessness, but an alarming level of ignorance. Often, the root cause arose simply from bypassing the infrastructure we provide, and employing the private notebook for data analysis and presentation instead of the dedicated office desktop. Now, our employees can bring their own devices if they like, but if they don't register them with our IT staff, they will be classified as guest devices that have no access to our intranet – with the rather obvious result that the data on these devices cannot be synced to the home directory of the respective user on our file server (which is part of a daily incremental backup on tape, covering every day over the last 10 years as legally required).

The users, naturally, don't find that obvious at all (although they have been informed at length about these facts). They claim to have acted in the firm believe that the data on their private notebook will be automatically backed up to “the cloud” as soon as they enter a certain “zone” around their working place. When I asked how they imagined this miracle backup would work, one of them referred to Apple commercials in which photographs were transferred from an iPhone to an iPad “magically”. “That's the state-of-the-art, right? I expected that it would be implemented here!” She also said that she imagined the mechanism to work wirelessly, but that she wouldn't care how it worked, as long as it did.

Now, when people approach me with these stories, they want (i) forgiveness and understanding and (ii) an immediate solution. Well...

Fortunately, we now have first level support consisting of an invariably cheery youth who finds these problems most entertaining. Let's see what he says in a few years from now, when pampering the “digital natives” has become the next big thing.

And let's see where we are then, with our big hopes and high flying dreams of, for example, artificial intelligence and quantum computing, autonomous electric mobility, populating the Mars, establishing controlled fusion on Earth, and controlling the world's climate. Personally, when I see the present generation of which the majority has difficulties to count to three, well, you know, I'm not all that optimistic. 😉

This backup is futile

Cobra — Sun, 29 May 2016 14:10:42 GMT

A couple of month ago, I noticed that I could not mount the filesystem created by attic, the program I was using to backup my data. A temporary inconvenience, I thought. Will certainly be fixed in a matter of days, I reasoned. But the situation was unchanged after two weeks, and to my great dismay, I found that I was also unable to extract single files from the backups. Attic was thus dutifully doing its hourly job, but I could have equally well directed all of them to /dev/null.

When searching for the issue, I discovered this thread. Attic was not yet completely broken at that time, but the stance of its developer is still remarkable:

Attic is my "feel good" pet project that I want to be able to code on whenever I happen to get some spare time. And when I have as little spare time as I currently have, I really do not want to spend it all on reviewing, merging, debugging and rewriting other peoples code.

Why on earth did he ever put his “pet project” on github then?

Since the community was interested, the project was forked. I switched to the fork and never looked back.

If you have an existing attic backup, all you have to do is to upgrade it by issuing

borg upgrade -v /path/to/repository

and to replace 'attic' by 'borg' in all backup scripts you may have. There is one major difference, though: if you want compression, you have to explicitly enable it, since borg allows you to chose the compression algorithm as well as the compression level.

As an example, here's the shell script I use to backup deepgreen (my Fujitsu Lifebook) to the big disk in my desktop (blackvelvet) via wifi:

	`#!/bin/bash`

	`#<https://github.com/borgbackup/borg>`
	`#<https://borgbackup.readthedocs.io/en/stable/index.html>`

	`ionice -c3 -p$$`

	`repository="cobra@blackvelvet:/bam/backup/attic/deepgreen" # directory backing up to`
	`excludelist="/home/cobra/bin/exclude_from_attic.txt"`
	`hostname=$(echo $HOSTNAME)`

	`notify-send "Starting backup"`

	`borg create --info --stats --compression lz4 \`
	$repository::$hostname-`date +%Y-%m-%d--%H:%M:%S` \
	`/home/cobra \`
	`--exclude-from $excludelist \`
	`--exclude-caches`

	`notify-send "Backup complete"`

	`borg prune --info $repository --keep-within=1d --keep-daily=7 --keep-weekly=4 --keep-monthly=6`

	`borg list $repository`

Back up the NAS

Cobra — Sat, 31 Oct 2015 13:23:54 GMT

Two weeks ago, my NAS suddenly emitted a high-pitched, enervating sound, and the indicator of one of the hard disk trays blinked nervously. The info line in the display screamed 'hard disk failure, RAID degraded', my wife's face was frozen in terror, the cats stared at me with eyes as big as saucers, and I was close to a heart attack. What prevented the disaster was my sudden realization that I had made a backup copy just a week ago...

In any case, since I've set up a RAID 5 for the four disks in my NAS, nothing was lost yet. And I actually wanted to upgrade the disks for quite some time, since their total capacity of somewhat less than 6 TB was close to be exhausted. Hence, I ordered four 3 TB NAS disks right away, but kept brooding over the question how to create backup copies when the capacity would be close to 9 TB. Should I really invest in a second NAS? Naa: I really don't need another always-on device in my flat.

The solution was so simple. With three 2 TB disks freed from the NAS, two 1 TB disks just lying around, and 6 TB in three USB hard disks, there's already more than enough disk space to back up the NAS. And the orphan SATA disks can be utilized by simply employing a docking station! Yes, it's that easy.

I've acquired this one and, just for he fun of it, partioned and formatted three of the orphan disks. Copying data from the NAS (mounted via cifs over a GB switch) worked flawlessly.

When the new disks arrived, I exchanged one by one, as one does for a RAID. It always took a few hours to rebuild the system, so I simply waited from one night to the other. Finally, it just took 10 min to expand the RAID to the larger disk size. The almost 9 TB should suffice for some time. 😊

Pinheads

Cobra — Thu, 01 Jan 2015 13:35:22 GMT

I've been using cloud-based storage services to synchronize files between my different machines since years. Recently, I've synchronized active projects via the ownCloud server we've set up at the office. For private purposes, such as sharing cat photos with the family, I've used Wuala since it offered ample space for this purpose for free, featured client-side encryption, and used servers located in the EU.

In June 2014, Wuala announced a new pricing plan which included a termination of the free storage offer they had previously. However, the original press release also included the statement "Remaining customers will keep their free storage." All right.

This morning, I was thus mildly surprised to receive an e-mail from Wuala stating that I've exceeded my "available Wuala storage space". Starting the client told me that my "available Wuala storage space" is 0 bytes, which I've indeed exceeded. What happened?

It turned out that Wuala didn't keep their promise and didn't communicate their business moves properly. As documented by Caschy, Wuala changed the original press release in October 2014. I haven't received any mail regarding this change.

Instead, they now inform me "that we reserve the right to delete your account if you don't upgrade or delete files within the next 30 days."

And then: "We're sure that you will continue to enjoy the benefits of Wuala, the secure Cloud Storage."

Sure. :D

Close call

Cobra — Fri, 24 Oct 2014 12:36:24 GMT

If you're using btrfs and its snapshot function: stay away from kernel 3.17! It may corrupt your filesystem. I've just spent the better part of the morning trying to coax my desktop into getting back to its normal self.

After updating to 3.17 and editing syslinux.cfg to take care of the now manual microcode updates, I've noticed the following anomalies:

KDE needed several minutes to start, just as every KDE application afterwards
System load was high (up to 5) despite the fact that the system was essentially idling (no CPU or I/O activities)
the btrfs formatted system partition switched to RO mode after a few minutes of use. WTF?
after a further reboot with the microcode updates disabled, avahi didn't come up and journald dumped one core after the other (105, to be precise).
oh, and there was no network at this point 😒

After rolling back to kernel 3.16.4, reinstalling systemd, and purging all (corrupted) journal files in /var/log/journal, the worst seems to be over.

Note that this bug is not restricted to users of Archlinux, but was reported by users of Fedora 20, Debian Sid, Gentoo, and OpenSUSE Tumbleweed.

Attic

Cobra — Sun, 07 Sep 2014 14:25:06 GMT

Attic is the candidate in this list of backup programs which managed to convince me entirely. It has a lot in common with the one I've written about below, i.e., Obnam: it is written in python and offers very similar features, namely, snapshot backups, data de-duplication across files and backup generations, and optional AES (instead of GPG) encryption. It's available for Debian (Jessie and Sid) and Archlinux, and for all others (including Debian Wheezy and Fedora) you can use use 'pip install attic' instead.

Attic doesn't have a separate configuration file. The following bash script handles everything:

	`#!/bin/bash`

	`#https://attic-backup.org/quickstart.html#quickstart`
	`#http://peterjolson.com/full-system-backup-using-attic-backup-to-nfs/`

	`ionice -c3 -p$$`

	`repository="cobra@blackvelvet:/bam/backup/attic/deepgreen"`
	`excludelist="/home/cobra/bin/exclude_from_attic.txt"`
	`hostname=$(echo $HOSTNAME)`

	`notify-send "Starting backup"`

	`attic create --stats \`
	$repository::$hostname-`date +%Y-%m-%d--%H:%M:%S` \
	`/home/cobra \`
	`--exclude-from $excludelist \`
	`--exclude-caches`

	`notify-send "Backup complete"`

	`attic prune -v $repository --keep-within=1d --keep-daily=7 --keep-weekly=4 --keep-monthly=6`

Attic is a little more verbose than Obnam in its reports:

Date: Thu,  4 Sep 2014 19:00:31 +0200 (CEST)
From: "(Cron Daemon)" cobra@blackvelvet.localdomain
To: cobra@blackvelvet.localdomain
Subject: "Cron" cobra@blackvelvet; /home/cobra/bin/backup.attic

Archive name: blackvelvet-2014-09-04--19:00:04
Archive fingerprint: e20c50dbdf43f9d31ee138338aab929ae78d82e4564bf6f05f55edd439a73e35
Start time: Thu Sep  4 19:00:04 2014
End time: Thu Sep  4 19:00:31 2014
Duration: 27.40 seconds
Number of files: 166064

       Original size      Compressed size    Deduplicated size
This archive:               41.71 GB             30.56 GB             21.05 MB
All archives:              995.98 GB            732.81 GB             26.54 GB
------------------------------------------------------------------------------
Keeping archive: blackvelvet-2014-09-04--09:00:01     Thu Sep  4 09:00:29 2014
Keeping archive: blackvelvet-2014-09-04--08:00:01     Thu Sep  4 08:00:31 2014
Keeping archive: blackvelvet-2014-09-04--07:00:01     Thu Sep  4 07:00:33 2014
Keeping archive: blackvelvet-2014-09-03--23:00:01     Wed Sep  3 23:00:23 2014
Keeping archive: blackvelvet-2014-09-03--22:00:01     Wed Sep  3 22:00:29 2014
Keeping archive: blackvelvet-2014-09-03--21:00:01     Wed Sep  3 21:00:29 2014
Keeping archive: blackvelvet-2014-09-03--20:00:01     Wed Sep  3 20:00:27 2014
Keeping archive: blackvelvet-2014-08-31--23:00:01     Sun Aug 31 23:00:24 2014
Keeping archive: blackvelvet-2014-08-30--23:00:01     Sat Aug 30 23:00:27 2014
Keeping archive: blackvelvet-2014-08-29--07:53:53     Fri Aug 29 07:54:13 2014
Keeping archive: blackvelvet-2014-08-28--22:31:53     Thu Aug 28 22:32:18 2014
Pruning archive: blackvelvet-2014-09-03--19:00:01     Wed Sep  3 19:00:25 2014
Pruning archive: blackvelvet-2014-09-03--18:00:01     Wed Sep  3 18:00:22 2014
Pruning archive: blackvelvet-2014-09-03--17:00:01     Wed Sep  3 17:00:26 2014
Pruning archive: blackvelvet-2014-09-03--16:00:01     Wed Sep  3 16:00:24 2014
Pruning archive: blackvelvet-2014-09-03--15:00:01     Wed Sep  3 15:00:26 2014
Pruning archive: blackvelvet-2014-09-03--14:00:01     Wed Sep  3 14:00:23 2014
Pruning archive: blackvelvet-2014-09-03--11:00:01     Wed Sep  3 11:00:23 2014
Pruning archive: blackvelvet-2014-09-03--10:00:01     Wed Sep  3 10:00:26 2014

I've decided to use Attic instead of Obnam for one main reason: speed. On the mini, the slowest member of my computing batallion, 15000 files occupying 4.7 GB take 17 s to backup for Attic, but 95 s for Obnam. On blackvelvet, the fastest by far, 165000 files occupying 42 GB take 25 s instead of 70 s.

What are the most important factors determining Attic's performance? Well, it depends. If you backup to a remote location, your upload speed may ultimately limit the overall speed. If that's not the case, since you're in a GBit LAN or backup to an internal hard disk, it still depends. If a lot of files have to be accounted for, the critical factor is the I/O performance of the mass storage device you're using. In the opposite case, Attic puts a 100% load on one core of your CPU independent of its performance class. Attic is thus utilizing the available resources quite nicely. Consequently, the performance of Attic can be roughly scaled with the file number for machines with comparable performance. For example, my lifebook here at home has 16000 files spread over 3.3 GB, and it takes Attic a mere 6 s for a snapshot. My office computer has essentially the same disk and cpu performance (Core 2 Duo class CPU, 320 GB HD), but more data: 255000 files (x15) spread over 67 GB (x20) taking 90 s (x15).

Three more remarks:

Attic's deduplication and compression schemes are apparently more efficient than those of Obnam, since the total size of the repository with a similar number of snapshots is significantly smaller. Those obtained by rsnaphot or rdiff-backup are anyway much larger since these tools do not compress the data.

A backup to a remote location is accelerated if Attic is installed on both, client and server. However, you can still backup to an atticless server using sshfs (with a small performance hit).

Finally, you can restore your data in the same way as with Obnam, namely, by mounting the repository:

attic mount cobra@blackvelvet:/bam/backup/attic/deepgreen /home/cobra/attic_mnt/

Nice.

Obnam

Cobra — Sun, 31 Aug 2014 13:35:20 GMT

Obnam is one of the two backup programs in my list which I highly recommend. It offers snapshot backups (in the spirit of btrfs snapshot subvolumes), data de-duplication across files and backup generations, and optional GnuPG encryption. Archers can get it on the AUR, and Debilians may obtain the current version from the developer's repository.

Obnam is ridiculously easy to configure and use:

[config]

repository = sftp://blackvelvet/bam/backup/nb_snapshot/deepgreen/

keep = 7d,4w

lru-size = 1024
upload-queue-size = 512

log = /home/cobra/.obnam/logs/default.log
log-level = warning
log-max = 10mb
log-keep = 10
log-mode = 0600

exclude = \.cache$, \.thumbnails$, \.tmp$, /cache/, /Downloads/, /temp/, /Trash/, /VirtualBox VMs/, /wuala/

Save this file as ~/.config/obnam/default.conf, for example, modify it to your needs, and execute obnam either directly

obnam --verbose backup $HOME

or via a small shell script:

	`#!/bin/bash`

	`notify-send "Backup started, please be patient..."`

	`obnam backup $HOME`

	`if [ "$?" -ne 0 ]; then`
	`notify-send "Unable to complete backup."`
	`exit 1`
	`else`
	`notify-send "Backup successfully completed."`
	`fi`

On my notebooks, I run this script manually, but on my desktops, I've added an entry to my crontab:

0    7-23  * * *   /home/cobra/bin/backup.obnam

The cron daemon sends a mail to report what has happened:

Date: Fri, 29 Aug 2014 09:01:11 +0200 (CEST)
From: "(Cron Daemon)" cobra@blackvelvet.localdomain
To: cobra@blackvelvet.localdomain
Subject: "Cron" cobra@blackvelvet /home/cobra/bin/backup.obnam

Backed up 77 files (of 187140 found), uploaded 63.0 MiB in 1m9s at 934.9 KiB/s average speed

That's a snapshot of my desktop with a total backup volume of 42 GB. Without the two lines in the config customizing the lru- and upload-queue sizes, a snapshot takes about 10 min, i.e., nine times longer. This mediocre performance with the default settings is certainly one of the reasons for the numerous reports of obnam being nice but slow. The speedup obtained by changing these settings, however, depends on your hardware: on the Mini, obnam is CPU limited, and it takes 90 s for one snapshot (of 5 GB size) no matter what the lru- and upload-queue sizes.

Restoring data is as easy as creating the backup. You can simply mount the backup, like that:

obnam mount --to /bam/obnam_mnt/

and access the resulting read-only filesystem with anything you like. Nice.

Better backup

Cobra — Sun, 24 Aug 2014 15:50:42 GMT

There's no such thing as too many backups. An example.

A guy I know (lets call him user C) amused himself by playing with an incremental backup scheme based entirely on tar:

	`#!/bin/bash`
	`# Based on ideas of Alessandro "AkiRoss" Re`
	`#http://blog.ale-re.net/2011/06/incremental-backups-with-gnu-tar-cron.html`

	`MONTH=$(date +%Y%m)`
	`DAY=$(date +%Y%m%d)`
	`BCKDIR=/bam/backup`
	`ARCDIR=/bam/backup/archives`
	`SRCDIR=/bam/backup/snapshots/deepgreen`

	`TARGETFILE="$ARCDIR/nb_snapshot_$DAY.tar.zz"`
	`LOGFILE="$ARCDIR/nb_snapshot_$MONTH.snar"`
	`EXCLUDEFILE="$BCKDIR/nb_socketlist"`

	`tar -c -X $EXCLUDEFILE -g $LOGFILE -f - $SRCDIR \| pigz --fast --rsyncable --zlib > $TARGETFILE`

The script worked well, and the backups kept piling up. User C realized that he needed a mechanism to restrict the time span over which backups are kept, and he thus added the following line to the end of the script to implement this mechanism:

find $ARCDIR -type f ! -newermt "1 month ago" -print0 | xargs -0 rm -f

Well, actually ... the line user C really used was ... hmmm ... slightly different. Instead of referring to' $ARCDIR', he addressed '.'.

Ooops.

User C is of course my own dumb self. But dumb or not, my backup concept has worked: I had a backup taken minutes before the unintentional purge of my entire home directory, and was able to resume normal operation within the hour. This hourly incremental backup resides on an internal disk different from the one holding my system and home partitions, and is synchronized every night with my NAS.

Now, let's get technical. How do I ensure that I have a backup when I need it?

That's really very simple: I use software which makes the task to create backups and to restore them as foolproof as possible. And that means:

(i) The backup software is command-line oriented and thus fits seamlessly into a cron- and script-based infrastructure, making automation of the whole process a breeze.
(ii) The backup software's configuration is script-based and thus transparent and straightforward to revise, keep, and document.

For several years now, I've used rsnapshot at home and rdiff-backup at the office. To save the home directories of my notebooks, I transfered them to my desktop using plain rsync. The above tar script then created an incremental backup also for these directories. That's a very heterogeneous and altogether outdated solution, and although it has served me well for years, I've recently decided to give my backup scheme a complete overhaul. In particular, I wanted one solution covering all my use cases. To my delight, I've discovered that a number of new backup programs for Linux are actively developed at present, some of which boast features such as global deduplication, efficient compression, and optional GPG encryption.

Here's a (certainly incomplete) list of contenders I will examine in the near future:

attic
backshift
burp
hashbackup
obnam
zbackup

In any case, when the day ends, and all hourly backups have been done, I sync them to the NAS:

	`#!/bin/bash`

	`rsync -az /bam/backup/ cobra@thecus::snapshot_blackvelvet`

I know, I know: if the house burns down, I'm going to lose this replica of my backup as well. Gimme a 100 Mbit/s upload. Then we'll talk.