Tuesday, October 4, 2011

Make a drive mirror on linux

A very brief post to quickly describe what I had to do to have an uncompressed automatic backup of my external hard drive on to another external hard drive of the same capacity.

I have an external USB 1TB hard drive where I store all media files from movies and music to family photos. Once I started having a pretty decent number of family photos I started to think that a backup was definitely in order so I got another 1TB external USB hard drive where I wanted to backup my data to. Here's what I wanted, along with the options that I considered, why they work or not for my application and finally my actual solution for this need:

Requirements

RAID1 Simple Backup rsync

I do not need the files to be compressed since most of the files on my main HD are media and compressed anyway. I don't want to loose time with compression.

OK

I think it compresses by default

OK

I want to be able to remove anyone of the disks and the data is still available on the local computer.

OK

OK

OK

I want to make incremental backups, i.e. data is only sync'ed not entirely copied over each time.

OK

OK

OK

I'm not really interested in keeping several versions of the same files. What I delete on the main, gets deleted on the backup drive, what gets added to the main disk, gets added to the backup.

OK

OK

OK

Any of the drives can just be unmounted and used on another machine with no extra software.

NO

OK

OK

So, from the table above you can see which of the options was right for me. I ended up using the following command using rsync:

rsync -avh --delete /path/to/localfiles /path/to/externaldrive >>rsyncLog.txt 2>>rsyncErr.txt

This command tells rsync to copy the files in (-a) archive mode, basically keepin
g all the same as the original in terms of symbolic links, devices, attributes, permissions and ownership. It requests an increse in verbosity (-v), i.e. more information is generated by the program (useful for debugging and detecting potential problems and all numbers appear in human-readable format (-h). I additionally pipe the program output to two files, one is a simple log of actions taken and the other contains any potential error messages generated. The --delete option deletes files from the destination in case they have been deleted on the origin.

As I final step I use cron to run the command above everyday at 04:00am. To simplify I use gnome-schedule to add the command and set the trigger time.

It all works well. Hope this helps someone.