Encrypted offsite backups via rsync

I love rsync to simply send files to other machines for backing up my data.
Unfortunately sometimes you can't trust the other side. So you would like to encrypt your data..
Rsync (via ssh) is encrypted during transport but not on disk...

I tried several solutions but they all have some issues:

duplicity
It's nice but doesn't do what rsync does. It just makes an encrypted initial full-backup and sends increments. It's also adviced to do a full backup regularly. That's not nice thing when you have to send >200GB over the internet..

rsyncrypto
Rsyncrypto encrypts files in such a way they are rsync friendly. (Big files don't completely change when a byte is changed).
Problem with rsyncrypto is that it requires an encrypted copy of all your data. I've got a LOT of files, and keeping this copy up to date costs too much time. I need to run rsyncrypto just before rsyncing the data. And my experiences with it is that it was pretty slow. And I even haven't taken into account the issue that I'm required to store my data on disk 2 times..

Meet encfs

Thanks to a reaction on serverfault ( http://serverfault.com/questions/160014/encrypted-remote-backups-via-rsync )
I've found encfs. Which makes it possible to create and mount a virtual filesystem via fuse-fs which shows an encrypted representation of your files..
Exactly what I want...

Install encfs on FreeBSD

It's required to enable fuse in FreeBSD. This is a kernel module
so add the following line to '/boot/loader.conf'

fuse_load="YES"

Next install the port (or package) encfs. (I love ports)

cd /usr/ports/sysutils/fusefs-encfs
make install clean

Creating a Backup

# mount read-only encrypted virtual copy of unencrypted local data:
encfs --reverse --idle=60 -o ro ~/data/ ~/.tmp_encrypted/

rsync -ai  ~/.tmp_encrypted/ name@example.com:backup/

umount ~/.tmp_encrypted/

First time a menu appears. I choose the following options:

Configuration mode: x - expert configuration
Cipher algorithm: AES
Key size: 256
Block size: 1024
Filename encoding: null*
Per-file initialization vectors: No
Password: *****

*I don't have the requirement to encrypt my filenames

Restore

To restore you data..

Take your encrypted file(s) Only the one's you need :D
Copy them into an empty folder ~encrypted-stuff
Copy your .encfs6.xml key to the same folder

Mount it:

encfs ~encrypted-stuff ~decrypted-stuff

Encfs asks for the password and behold: you can acces your files again in the ~decrypted-stuff folder :D

WARNING

You should backup your plain text .encfs6.xml file (which is on the unencrypted volume).
on a really safe location. It's your only key to decrypt your data.

Notes

A possible extra safety measure could be to NOT sync the encrypted .encfs6.xml file...
So use --exclude=encfs6.xml

For me the filenames aren't really sensitive so I don't encrypt them.
It makes it much more easy to find the correct file in my encrypted data..

Btw a scripting tip could be the following '--sdtinpass' and supply your password:

echo 'PASSWORD' | encfs --reverse --stdinpass --idle=60 -o ro ~/data/ ~/.tmp_encrypted/

If you have suggestions or advise, please drop a line!!

rsync output compressor

Rsync is a very nice tool for automating remote backups. (Specially in combination with daily snapshots (like zfs snapshot) ).
Like many others I have automated the process of running rsync on a daily basis via a cron job. Cron nicely sends me an email with the output of the rsync command.

I usually use the -v option so I can see what files have been changed. This worked nicely several years ago when I didn't have much changes on my server. But nowadays I often receive e-mails of 10 MB or larger. That's not very useful.

Removing the -v option is an option, but I don't see anything anymore (perhaps a total summary).

To solve this problem I've hacked together an rsync-output-compressor script :)
You can find it on https://github.com/gamecreature/rsync-output-compressor

This scripts summarizes the output of rsync -v based on a given rules file. You can specify what files/folders should be explicitly mentioned and what folders/files should be grouped together.

This little script is written for ruby 1.9 and higher.

An example

For example let's view the following output: (... = many more lines)

rsync -avz --delete user@example.com:/data /backups/remote_data

receiving incremental file list
/home/emma/public_html/important_file.txt
/home/emma/public_html/important_file1.txt
/home/emma/public_html/important_file2.txt
/home/emma/public_html/important_file3.txt
...
/home/sarah/public_html/index.html
/home/sarah/public_html/images/
...
/home/david/private/special_file.txt
/home/david/public_html/downloads/new_download.zip

Using the following filter: (compress-rules.txt)

/home/*/public_html/

results in the following output:

rsync -avz --delete user@example.com:/data /backups/remote_data | rsync-output-compressor.rb --rules compress-rules.txt

receiving incremental file list
   123    -5 /home/emma/public_html/
    40       /home/sarah/public_html/
     1       /home/david/private/special_file.txt
     2       /home/david/public_html/

The column with positive values are changed/added files and the column with negative values are the number of deleted files.

The tool has several other options like storing the original full output to an external location (option -f).

Using this script my daily emails have been reduced from 10MB to 30KB :)
And I still know what is happening with my backup.

Feel free to use and improve this little script!

RSync Hangs when rsyncing via SSH

I've got an automatic backup script that makes a daily copy of my server to an offsite location. I use rsync for this. I've noticed last week that nothing the sync never completed...

Btw. The offsite location (a SuSE Linux 9.3 server) initiates and rsync over SSH with my FreeBSD 6.2 server.

After some debugging I noticed rsync is hanging in the middle of the backup!?
I found several articles on the internet about this problem. But I'm still searching for a solutions for my problem... :(

Well every time it hangs on:

/somepath/wordpress/xmlrpc.php is uptodate

WordPress.. Coincidence?!? ;-)

---

11 august 2007 - Well it was a false alarm. Rsync is just extremely slow... :-(