How to Backup Your [Perforce] Server to Amazon S3

In today’s post I’ll be covering how to configure a set of simple scripts to perform daily backups on your Debian-based server. This script will log into your Amazon S3 account, upload a series of compressed files, and manage full/partial backups. No user intervention is required after initial setup, and a detailed backup summary will be e-mailed to the address of your choosing every time the backup is complete.

For this tutorial you will need the following:

  1. Debian-based server. I am using an Ubuntu 12.0.4 LTS installation on a VPS
  2. An Amazon Web Services (AWS) account with a set of access and secret keys
  3. An S3 bucket specifically made for this purpose. The script does not like capitalization, symbols, dashes, etc. so name it something simple (like your host name). You can create a new bucket using the web interface.

I should also point out that I will be including a couple extra configuration parameters for my installation of Perforce. These may be ignored if you’re only trying to back up directories and files. Let’s get started!

The first thing we have to do is install some utilities that the script depends upon. s3cmd is the utility which will allow your server to communicate with S3. Duplicity is a utility which manages creating the backup archives, compressing them, and handing them off to s3cmd for upload.

sudo apt-get install s3cmd
sudo apt-get install duplicity duply python-boto

The next step is to configure s3cmd. Run the following command under the account you wish to use to run backups (don’t use sudo).

s3cmd --configure

The utility will ask you for information such as your S3 access and secret keys. It will also ask for a password to encrypt these keys.

s3cfg

The script I’m using is made by the guys over at zertrin.org. They’ve done a great job setting up the script and making it easy to configure! Be sure to head over to their site and thank them for their hard work!! You can download the script and configuration file here. You will need both the .sh and the .conf.example files. Use wget to download them to your home directory, or use WinSCP to get them on your server.

You will also need to write a small script to call the main backup script. Use your favorite text editor to create a file called “duplicity” without the quotes, paste the following into it, and save it in your home directory.

#!/bin/sh
p4 verify //...
p4 admin checkpoint
test -x /etc/scripts/duplicity-backup.sh || exit 0
cd /etc/scripts && ./duplicity-backup.sh -c /etc/scripts/duplicity-backup.conf --backup

Note that if you are not using this script to back up a perforce server, you do not need lines 2 and 3. The first command will verify the perforce file structure and database. The second will create a checkpoint and save the entire database to your Perforce data directory.

Once you save and close the “duplicity” file, run the following commands in to set permissions, create folders, and move files into the proper directories. Be sure to replace “YOURUSERACCOUNT” with the user account you would like to run the backups.

chmod +x duplicity
chmod +x duplicity-backup.sh
sudo mkdir /etc/scripts/
sudo mv duplicity-backup.sh /etc/scripts/
sudo mv duplicity-backup.conf /etc/scripts/
sudo chown YOURUSERACCOUNT /etc/scripts/duplicity-backup.sh
sudo chown YOURUSERACCOUNT /etc/scripts/duplicity-backup.conf

The next step is to configure and fix the e-mail daemon. If you’re running Ubuntu, you will need to follow my guide here to fix it. Go ahead and get that done, then proceed with the rest of this guide.

Now that that’s been fixed and your e-mail settings saved, let’s proceed with configuring your backups!

nano /etc/scripts/duplicity-backup.conf

Change your settings to fit your configuration. The configuration file is well documented and should be fairly straightforward to configure. Be sure to add your perforce data and installation directories, add you bucket name, change your e-mail settings, and tweak the backup interval!

backup_conf

Once that’s ready to go, let’s move the script to its final home.

sudo mv duplicity /etc/cron.daily
sudo chmod +x /etc/cron.daily/duplicity
sudo chown YOURUSERACCOUNT /etc/cron.daily/duplicity

Any script that you place in this folder will be executed once daily. If you’d like the backup to run every hour, there’s also a folder called cron.hourly where you can place the script.

Now, let’s try the script out! Run the following command:

cd /etc/cron.daily/
./duplicity

You should see something similar to this in the terminal:

backup_term_complete

Note: I have quite a few commits in my Perforce SVN, so your output will be different from the image above.

Now, let’s check AWS to see whether new files were added:

backup_aws

And finally, let’s check our e-mail to see whether the correct message was sent:

backup_email

If everything checks out, then you’re finished! If you need to backup any new directories in the future, just add them to the configuration file! Simple!!

I hope this helped you! If so, feel free to leave comments or questions below!

Comments

comments