Month: March 2013

Backups with Amazon S3 and Duplicity

I had to set up proper full backups for a client website, S3 seemed like an economical way to go (time will tell). Duplicity is a script which uses rsync to back up incrementally, which will save data transfer. It can also encrypt the data and a bunch of other goodies.

Installing Duplicity

Since the site was running on Centos (Debian forever!) and the Yum repositories don’t have Duplicity, I had to add some the rpmforge repository:

wget http://packages.sw.be/rpmforge-release/rpmforge-release-0.5.2-2.el5.rf.x86_64.rpm
rpm --import http://apt.sw.be/RPM-GPG-KEY.dag.txt
rpm -K rpmforge-release-0.5.2-2.el5.rf.*.rpm
rpm -i rpmforge-release-0.5.2-2.el5.rf.*.rpm

Now duplicity will install:

yum install duplicity

It also requires Google’s boto library

yum install boto

Make the backup script

Now make a shell script something like this:

export PASSPHRASE=XXXXXXXXXXXXXXXXXXXXXXXXXXXXX
export AWS_ACCESS_KEY_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXX
export AWS_SECRET_ACCESS_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXX

dirs="/var/svn \
     /var/www \
     /home \
     /usr/local/stuff"

for d in $dirs; do
  prefix=$(basename $d)
  echo duplicity $d s3+http://backup.bucket.name/$prefix
  duplicity --full-if-older-than 30D $d s3+http://backup.bucket.name/$prefix
  echo ""
done

PASSPHRASE can be anything you want, just make it long so the encryption is strong.

The keys come from AWS – see https://portal.aws.amazon.com/gp/aws/securityCredentials

“–full-if-older-than 30D” will make duplicity perform a full backup every 30 days. Otherwise it does incremental backups.

You can also make a restore script by swapping the duplicity arguments

duplicity s3+http://backup.bucket.name/$prefix $d

Add the script to cron

0 0 * * * /root/scripts/etc/backup.sh >>/var/log/duplicity/etc.log

References

DrupalCon Sydney 2013

DrupalCon Sydney went down last weekend, and it was special for a few reasons. It was the first DrupalCon in the southern hemisphere, the first outside of the US and Europe, and  my first national Drupal event.

There have been two previous Australia wide Drupal camps, Drupal Down Under 2011 and  2012. I had tickets and flights for the former, but broke my ankle at a bucks do shortly before. For the latter I was busy touring India. So this was my first opportunity to see how amazing the Drupal community is in the real life. And trust me, it was awesome.

I went there looking for a mentor to help me delve further into core development. Instead I met a bunch of inspiring folks and, well, got drunk with them. Great fun, would do again.

It all went down on Sydney’s Coogee beach, hosted at the Crowne Plaza Hotel. Great venue that managed to supply mostly-fast wifi for 400+ Drupal geeks.

Dries’ keynote was first up. He started out discussing the difference Drupal has made in the world, mentioning the Sandy hurricane response website, the ALRC website where citizens can review upcoming laws and the Egypt Independent News websites role during the Arab Spriong. The point being, we have come this far, where next?

Following on, he outlined the various Drupal 8 initiatives and the progress they have been making – especially in an authoring experience direction. This section ended with a roadmap to the D8 release and an emphasis on getting things ready for the upcoming feature freeze (2 days from the time of writing!).

Next was a discourse on Drupal in the enterprise, and the things these organisations need to manage – content, community and commerce – and how Drupal is well placed to handle all three. This led into the concept of web experience management(WEM) which says that basically all business websites perform the same functions – attract, engage, influence, convert and retain.

He had a great diagram showing the various components of WEM, and how there was opportunity for the Drupal community to build businesses around each component.

Then the call to action went out – we need larger Drupal agencies that can work with global companies, we need more startups and we need to “embrace the elephants”. More than anything we need more talent, but I think everyone working with Drupal is aware of that.

Here’s a quick run down of sessions that stood out to me on the Thursday:

  • Drupal 8 Web Services with Moshe Weitzman – Services module is committed to core. You can enable the module, select entities and bundles to expose, then start writing your client application, such as a mobile app. The service is RESTful and can return XML and JSON-LD. Authentication relies on Drupal authentication at the moment, but they want to add HTTP authentication. Lots of interesting use cases like apps, integration with backbone.js and a large role in Drupal 8’s in-place editing functionality. Content sharing and content staging are made simpler as well.
  • Symfony Components: A Travel Guide with Piers Warmers -Drupal 8 is replacing a bunch of code with Symfony components, and I’ve been trying to get my head around the changes. This talk helped a lot with that. Topics covered include PSR-0, Composer and Packagist, Doctrine, Guzzle, Assetic and Twig. In summing up he mentioned that it would be really cool if Drupal could start contributing packages back to Packagist. The thought of abstracting and decoupling something like FormAPI scared the crap out of me!
  • Performance Tough Love with Mark Sonnabaum – This was an interesting Q&A session  on performance and how general strategies apply to Drupal. The most important thing I got out of it was the importance of profiling code in diagnosing performance issues. Mark posted up a link to a gist on github telling people how to setup xhprof, a PHP profiler developed at Facebook. I forked his gist and updated it with detailed instructions for setting up xhprof with Ubuntu 12.10.

Friday I went to all the sessions in the Core Conversations Summit track.

Firstly webchick took us through upgrading the Drupal 7 ‘Pants’ module to Drupal 8, live on stage. Many pants related LOLs were had. Upgrading blocks was particularly eye-opening. Blocks are now defined using the new Plugins system which looks a lot more complicated, but I guess its just a matter of getting used to it. Symfony Routing and annotations are a bit trippy too 🙂

The remainder of the Summit was discussion and Q&A with the Core Initiative leads, some of whom were dialed in on via a Google Hangout. This was all super exciting, D8 is going to be a mad release!

The final night was Trivia night, and the WA team started off well but fell apart by the end, unfortunately. Some of the questions were pretty damn hard. It’s a bit blurry from then on out, I know I ended up discussing T1 Calculators, first programming experiences and US gun politics until around 4am when we ended up sitting on Coogee beach eating Maccas. Great night.

Next day was a total wipe, the best we could think to do was head to the airconditioning in the Airport and chill. Took a few days to recover from the non-stop pace of Drupalcon, but I can’t wait for the next one. Bring on DrupalSouth Wellington!