Backups with Amazon S3 and Duplicity

I had to set up proper full backups for a client website, S3 seemed like an economical way to go (time will tell). Duplicity is a script which uses rsync to back up incrementally, which will save data transfer. It can also encrypt the data and a bunch of other goodies.

Installing Duplicity

Since the site was running on Centos (Debian forever!) and the Yum repositories don’t have Duplicity, I had to add some the rpmforge repository:

rpm --import
rpm -K rpmforge-release-0.5.2-2.el5.rf.*.rpm
rpm -i rpmforge-release-0.5.2-2.el5.rf.*.rpm

Now duplicity will install:

yum install duplicity

It also requires Google’s boto library

yum install boto

Make the backup script

Now make a shell script something like this:


dirs="/var/svn \
     /var/www \
     /home \

for d in $dirs; do
  prefix=$(basename $d)
  echo duplicity $d s3+$prefix
  duplicity --full-if-older-than 30D $d s3+$prefix
  echo ""

PASSPHRASE can be anything you want, just make it long so the encryption is strong.

The keys come from AWS – see

“–full-if-older-than 30D” will make duplicity perform a full backup every 30 days. Otherwise it does incremental backups.

You can also make a restore script by swapping the duplicity arguments

duplicity s3+$prefix $d

Add the script to cron

0 0 * * * /root/scripts/etc/ >>/var/log/duplicity/etc.log


DrupalCon Sydney 2013

DrupalCon Sydney went down last weekend, and it was special for a few reasons. It was the first DrupalCon in the southern hemisphere, the first outside of the US and Europe, and  my first national Drupal event.

There have been two previous Australia wide Drupal camps, Drupal Down Under 2011 and  2012. I had tickets and flights for the former, but broke my ankle at a bucks do shortly before. For the latter I was busy touring India. So this was my first opportunity to see how amazing the Drupal community is in the real life. And trust me, it was awesome.

I went there looking for a mentor to help me delve further into core development. Instead I met a bunch of inspiring folks and, well, got drunk with them. Great fun, would do again.

It all went down on Sydney’s Coogee beach, hosted at the Crowne Plaza Hotel. Great venue that managed to supply mostly-fast wifi for 400+ Drupal geeks.

Dries’ keynote was first up. He started out discussing the difference Drupal has made in the world, mentioning the Sandy hurricane response website, the ALRC website where citizens can review upcoming laws and the Egypt Independent News websites role during the Arab Spriong. The point being, we have come this far, where next?

Following on, he outlined the various Drupal 8 initiatives and the progress they have been making – especially in an authoring experience direction. This section ended with a roadmap to the D8 release and an emphasis on getting things ready for the upcoming feature freeze (2 days from the time of writing!).

Next was a discourse on Drupal in the enterprise, and the things these organisations need to manage – content, community and commerce – and how Drupal is well placed to handle all three. This led into the concept of web experience management(WEM) which says that basically all business websites perform the same functions – attract, engage, influence, convert and retain.

He had a great diagram showing the various components of WEM, and how there was opportunity for the Drupal community to build businesses around each component.

Then the call to action went out – we need larger Drupal agencies that can work with global companies, we need more startups and we need to “embrace the elephants”. More than anything we need more talent, but I think everyone working with Drupal is aware of that.

Here’s a quick run down of sessions that stood out to me on the Thursday:

  • Drupal 8 Web Services with Moshe Weitzman – Services module is committed to core. You can enable the module, select entities and bundles to expose, then start writing your client application, such as a mobile app. The service is RESTful and can return XML and JSON-LD. Authentication relies on Drupal authentication at the moment, but they want to add HTTP authentication. Lots of interesting use cases like apps, integration with backbone.js and a large role in Drupal 8’s in-place editing functionality. Content sharing and content staging are made simpler as well.
  • Symfony Components: A Travel Guide with Piers Warmers -Drupal 8 is replacing a bunch of code with Symfony components, and I’ve been trying to get my head around the changes. This talk helped a lot with that. Topics covered include PSR-0, Composer and Packagist, Doctrine, Guzzle, Assetic and Twig. In summing up he mentioned that it would be really cool if Drupal could start contributing packages back to Packagist. The thought of abstracting and decoupling something like FormAPI scared the crap out of me!
  • Performance Tough Love with Mark Sonnabaum – This was an interesting Q&A session  on performance and how general strategies apply to Drupal. The most important thing I got out of it was the importance of profiling code in diagnosing performance issues. Mark posted up a link to a gist on github telling people how to setup xhprof, a PHP profiler developed at Facebook. I forked his gist and updated it with detailed instructions for setting up xhprof with Ubuntu 12.10.

Friday I went to all the sessions in the Core Conversations Summit track.

Firstly webchick took us through upgrading the Drupal 7 ‘Pants’ module to Drupal 8, live on stage. Many pants related LOLs were had. Upgrading blocks was particularly eye-opening. Blocks are now defined using the new Plugins system which looks a lot more complicated, but I guess its just a matter of getting used to it. Symfony Routing and annotations are a bit trippy too 🙂

The remainder of the Summit was discussion and Q&A with the Core Initiative leads, some of whom were dialed in on via a Google Hangout. This was all super exciting, D8 is going to be a mad release!

The final night was Trivia night, and the WA team started off well but fell apart by the end, unfortunately. Some of the questions were pretty damn hard. It’s a bit blurry from then on out, I know I ended up discussing T1 Calculators, first programming experiences and US gun politics until around 4am when we ended up sitting on Coogee beach eating Maccas. Great night.

Next day was a total wipe, the best we could think to do was head to the airconditioning in the Airport and chill. Took a few days to recover from the non-stop pace of Drupalcon, but I can’t wait for the next one. Bring on DrupalSouth Wellington!




Spotify in Ubuntu 12.10

Spotify is a cool streaming music service – $12 a month for high quality music streams on mobile and desktop. They have a decent Linux client too:

sudo apt-key adv --keyserver --recv-keys 4E9CFF4E
sudo sh -c 'echo "deb stable non-free" >> /etc/apt/sources.list'
sudo apt-get update && sudo apt-get install spotify-client-qt

Sounds like Spotify is being added to the Software Center soon anyway.

Optional extras:

Mucking About with Voxel.js


Installing Voxel.js

I just discovered voxel.js, which is a web-based voxel engine that can be used to build block-based games like Minecraft. It’s written in node.js, so lets get that set up. Voxel.js requires node.js 0.8.0 or better, and Ubuntu’s repositories supply 0.6.19, so we need to set up a PPA to get the latest version.

sudo add-apt-repository ppa:chris-lea/node.js
sudo apt-get update
sudo apt-get install nodejs npm

Under Ubuntu 12.10 the node.js executable is name ‘nodejs’, I believe its ‘node’ on some other systems.

Grab the voxel hello world repo from github:

git clone
cd voxel-hello-world
npm install
npm start

Now you should be able to check out voxel at localhost:8080. You will have be inside a crater with a couple of minecrafty types in front of you. They would be Maxogden and Substack, the voxel.js devs.

“npm install” downloads all the dependencies for voxel-hello-world. ‘npm start’ makes npm run a node js webserver, and rebuild index.js on each request using browservefy.

Note: I can only get this working in Firefox – according to Google Chrome doesn’t support WebGL with my laptop’s Nvidia FX Go5200 🙁

Switch to Perlin Noise Terrain Generation

Instead of using the predefined ‘Valley’ terrain, we can switch out to a module, such as the Perlin Noise Terrain Generator.

npm install voxel-perlin-terrain

Add the following at the top of index.js as below:

var createTree = require('voxel-forest')

Then update createGame:

var chunkSize = 32
var chunkDistance = 2

var generate = perlin({
  chunkDistance: chunkDistance,
  chunkSize: chunkSize,
  scaleFactor: chunkDistance * chunkSize / 4

var game = createGame({
  generateVoxelChunk: generate,
  cubeSize: 25, 
  chunkSize: chunkSize,
  chunkDistance: chunkDistance,
  startingPosition: [185, 100, 0], 
  texturePath: texturePath,
  worldOrigin: [0,0,0],
  controlOptions: {jump: 6}

You can muck around with the value of scaleFactor to make mountains, hills or relatively flat terrain.

Plant Some Trees

In order to generate some trees we need to include voxel-forest then add a few lines of code..

npm install voxel-forest
var createTree = require('voxel-forest')
// Create some trees.
for (var i = 0; i < 25; i++) {
createTree(game, {bark: 2, leaves: 3});

From here..

That’s probably enough for one post but jeez there’s a lot more here to play with so watch out for a followup.