Backing up encrypted and compressed VM snapshot to Azure Cloud

For some time now I was thinking about a good backup solution for our root server. We are using our hard drives in RAID0 mode which means that the two hard drives are not mirrored. Therefore we could use the complete 5TB space. In this scenario, complete data loss is quite likely from time to time, in case one of the two hard drives gets corrupted.
One way to solve this issue is a remote backup of the single vm images. Using LVM it’s possible to take a snapshot from a running virtual machine image. So we can safely compress and transfer the image at a specific state.
Since my home server wouldn’t have enough space to store the backup, I was looking for a cheap “cloud storage”. Besides Amazon AWS there’s also Microsoft Azure. The price per gigabyte is quite good for a low latency and low redundancy option. To register at Azure you’ll need a valid credit card. After that, you can test the service in trail mode for free.

Create backup

Transferring large files to Azure is a bit tricky. I had difficulties using the offical client software called azcopy. I found an other version of this tool, an not yet released preview: azcopy-v10. Using this version, I was able to copy bigger files with 500gb+ successfully. I created an AUR package, so it is easy to install in ArchLinux.
Together with LVM and GnuPG, I combined several commands, so that I could compress, encrypt and transfer the VM snapshot at once in a single step :D Considering the active image you want to backup is called “mail” and resides in a volume group “vg0”. You can create a snapshot with this command:

lvcreate -s -n mail_snap -L 20G /dev/vg0/mail

Install azcopy-v10 and start the transfer:

pacaur -S azcopy-v10
pv -cN source /dev/vg0/mail_snap | gpg --batch --passphrase "my_secret_password" --symmetric --compress-algo zlib | azcopy cp "https://myaccount.blob.core.windows.net/mycontainer/mail_$(date +"%Y-%m-%d").img.gpg?sas"

This is what the command does:

• With the command pv, we are piping the contents of the snapshot to gpg and we’ll have an additional progress bar in our terminal.
• GPG is encrypting the snapshot with a specific passphrase, which you’ll have to define. Please note that this usage is considered unsafe because you should never type or provide your passwords in plain text. Please consult the gpg manual on how to setup asymetric encryption for better security. Further, gpg is using zlib to compress the archive.
• The last part in this chain is azcopy, which will read our encrypted and compressed data stream from STDIN. There you’ll have to define the URL to your storage account on Azure, the destination filename and the one time session key called SAS. This information can be found in the Azure portal where you can create your blob storage account. Be sure to also replace “myaccount” and “mycontainer” in the url with the correct names.

After the transfer is complete, you can remove the snapshot from LVM:

lvremote /dev/vg0/mail_snap

Restore backup

To restore a backup, just use azcopy as well:

azcopy cp "https://myaccount.blob.core.windows.net/mycontainer/mail.img.gpg?sas" /mnt/playground.img.gpg
gpg -o /mnt/playground.img -d /mnt/playground.img.gpg

Gpg will ask for the passphrase you specified before.