Adding a raw mail-file to the postfixqueue

Today I had an issue that I needed to resend quarantined emails.
You can simply execute the following command to inject the mails to postfix

sendmail -t < name_of_the_raw_email_file

Source:
Thanks Wietse (Author of Postfix)
http://postfix.1071664.n5.nabble.com/best-way-to-send-quot-raw-quot-email-files-td45960.html

Encrypted offsite backups via rsync

I love rsync to simply send files to other machines for backing up my data.
Unfortunately sometimes you can't trust the other side. So you would like to encrypt your data..
Rsync (via ssh) is encrypted during transport but not on disk...

I tried several solutions but they all have some issues:

duplicity
It's nice but doesn't do what rsync does. It just makes an encrypted initial full-backup and sends increments. It's also adviced to do a full backup regularly. That's not nice thing when you have to send >200GB over the internet..

rsyncrypto
Rsyncrypto encrypts files in such a way they are rsync friendly. (Big files don't completely change when a byte is changed).
Problem with rsyncrypto is that it requires an encrypted copy of all your data. I've got a LOT of files, and keeping this copy up to date costs too much time. I need to run rsyncrypto just before rsyncing the data. And my experiences with it is that it was pretty slow. And I even haven't taken into account the issue that I'm required to store my data on disk 2 times..

Meet encfs

Thanks to a reaction on serverfault ( http://serverfault.com/questions/160014/encrypted-remote-backups-via-rsync )
I've found encfs. Which makes it possible to create and mount a virtual filesystem via fuse-fs which shows an encrypted representation of your files..
Exactly what I want...

Install encfs on FreeBSD

It's required to enable fuse in FreeBSD. This is a kernel module
so add the following line to '/boot/loader.conf'

fuse_load="YES"

Next install the port (or package) encfs. (I love ports)

cd /usr/ports/sysutils/fusefs-encfs
make install clean

Creating a Backup

# mount read-only encrypted virtual copy of unencrypted local data:
encfs --reverse --idle=60 -o ro ~/data/ ~/.tmp_encrypted/

rsync -ai  ~/.tmp_encrypted/ name@example.com:backup/

umount ~/.tmp_encrypted/

First time a menu appears. I choose the following options:

Configuration mode: x - expert configuration
Cipher algorithm: AES
Key size: 256
Block size: 1024
Filename encoding: null*
Per-file initialization vectors: No
Password: *****

*I don't have the requirement to encrypt my filenames

Restore

To restore you data..

Take your encrypted file(s) Only the one's you need :D
Copy them into an empty folder ~encrypted-stuff
Copy your .encfs6.xml key to the same folder

Mount it:

encfs ~encrypted-stuff ~decrypted-stuff

Encfs asks for the password and behold: you can acces your files again in the ~decrypted-stuff folder :D

WARNING

You should backup your plain text .encfs6.xml file (which is on the unencrypted volume).
on a really safe location. It's your only key to decrypt your data.

Notes

A possible extra safety measure could be to NOT sync the encrypted .encfs6.xml file...
So use --exclude=encfs6.xml

For me the filenames aren't really sensitive so I don't encrypt them.
It makes it much more easy to find the correct file in my encrypted data..

Btw a scripting tip could be the following '--sdtinpass' and supply your password:

echo 'PASSWORD' | encfs --reverse --stdinpass --idle=60 -o ro ~/data/ ~/.tmp_encrypted/

If you have suggestions or advise, please drop a line!!

Ruby compare sort en nonzero?

Just reminder for myself...
The method Numeric#nonzero?
returns self if the number is != 0 else it returns nil

Makes this code possible:

  objects.sort do |a, b|
    (a.lastname <=> b.lastname).nonzero? || (a.firstname <=> b.firstname)
  end

Incompatible Rails Local Configuration

Today I experienced a problem when upgrading to rails 4.2 from rails 3.

When updating an Activerecord object with a DateTime value, I've got the following error:

NoMethodError: undefined method `getlocal' for Mon, 01 Jan -4712 00:00:00 +0000:DateTime

I had the following configuration in my Application object:

  config.active_record.default_timezone = :local  # store all values in the local time-zone 
  config.active_record.time_zone_aware_attributes = false # ignore the time zone

I don't want time-zone information for this project.
But getting an error isn't nice so I solved it by enabling the time_zone_aware_attributes.

  config.active_record.default_timezone = :local  # store all values in the local time-zone 
  config.active_record.time_zone_aware_attributes = true # ignore the time zone

rsync output compressor

Rsync is a very nice tool for automating remote backups. (Specially in combination with daily snapshots (like zfs snapshot) ).
Like many others I have automated the process of running rsync on a daily basis via a cron job. Cron nicely sends me an email with the output of the rsync command.

I usually use the -v option so I can see what files have been changed. This worked nicely several years ago when I didn't have much changes on my server. But nowadays I often receive e-mails of 10 MB or larger. That's not very useful.

Removing the -v option is an option, but I don't see anything anymore (perhaps a total summary).

To solve this problem I've hacked together an rsync-output-compressor script :)
You can find it on https://github.com/gamecreature/rsync-output-compressor

This scripts summarizes the output of rsync -v based on a given rules file. You can specify what files/folders should be explicitly mentioned and what folders/files should be grouped together.

This little script is written for ruby 1.9 and higher.

An example

For example let's view the following output: (... = many more lines)

rsync -avz --delete user@example.com:/data /backups/remote_data

receiving incremental file list
/home/emma/public_html/important_file.txt
/home/emma/public_html/important_file1.txt
/home/emma/public_html/important_file2.txt
/home/emma/public_html/important_file3.txt
...
/home/sarah/public_html/index.html
/home/sarah/public_html/images/
...
/home/david/private/special_file.txt
/home/david/public_html/downloads/new_download.zip

Using the following filter: (compress-rules.txt)

/home/*/public_html/

results in the following output:

rsync -avz --delete user@example.com:/data /backups/remote_data | rsync-output-compressor.rb --rules compress-rules.txt

receiving incremental file list
   123    -5 /home/emma/public_html/
    40       /home/sarah/public_html/
     1       /home/david/private/special_file.txt
     2       /home/david/public_html/

The column with positive values are changed/added files and the column with negative values are the number of deleted files.

The tool has several other options like storing the original full output to an external location (option -f).

Using this script my daily emails have been reduced from 10MB to 30KB :)
And I still know what is happening with my backup.

Feel free to use and improve this little script!