Weakly Typed

Over the last 8 months we’ve received close to 45 million classification clicks from the fantastic Galaxy Zoo community. Averaged over the 8 months that’s roughly 8,000 clicks per hour – not bad going!

The challenge for our development team has been to design a web stack that’s able to cope with big traffic spikes like the one we had earlier this week from APOD but to also keep the hosting costs reasonably low. As I’ve mentioned before, the pricing model of Amazon Web Services means that we can scale our web and API layers based on how busy we are however what’s not so straightforward is scaling the database layer in realtime.

» The Problem

If scaling databases is hard (and we don’t want to buy our way out of the problem) then is there an alternative strategy that we can employ? It turns out there is and the solution is asynchronous processing of our classifications.

In the past, when you reached the end of the classification decision tree on the Galaxy Zoo site there was a pause between answering the final question and the page refreshing with the next galaxy for analysis. During this pause the classification was being saved back to the database and then a second request made to get a new galaxy. The problem with this approach is obvious – the busier the site gets, the busier (and slower) the database becomes and the longer it takes for the page to refresh.

A better approach then would be to decouple the classification-saving from the website user interface and remove the delay between classifications.

» The Solution – Asynchronous Processing

About 3 weeks ago we made a change to the Galaxy Zoo site to remedy this situation – the solution was to use a message queue. Message queues are basically a web-hosted queue of small snippets (or messages) of information – in our case a classification! Handy for us, Amazon have a message queueing service called Simple Queue Service (SQS) and we’re using it to help us scale.

The old model of saving a classification was to send an XML snippet back to the Galaxy Zoo API and wait for confirmation of a successful save to the database. The difference now is that this XML snippet is written to a SQS queue and we have a separate daemon that processes the queue. By posting the XML classifications to SQS I’m pleased to say that we’ve dramatically improved the responsiveness of the Galaxy Zoo site and managed to avoid paying for a significantly more expensive database!

» A resounding success?

Before I get too self-congratulatory here it’s important to realise that whilst using a message queue has helped us a great deal, there are some undesirable consequences that can arise during busy periods.

By using a queue we haven’t actually increased the rate at which we can save classifications back to the database, instead we’ve just created a buffer that we can store the classifications in until the site quietens down and we can process the backlog. Typically there are less than 5 messages in the queue (i.e. we’re keeping up with the current classification rate) however during very busy periods this isn’t the case: Earlier this week we had a couple of very busy days which meant that at one point there were 30,000 classifications in the queue waiting to be saved! The consequence of these messages being queued is that it’s possible that you could classify a whole bunch of galaxies but not see them in your recent history in ‘My Galaxies’ until minutes or hours later.

» Conclusions

Overall we’ve been very pleased with the new queue-based system – we’ve successfully managed to decouple the user interface from a database that’s starting to get a little sluggish. The issue of ‘My Galaxies’ being slightly out of date only arises during particularly busy periods and usually resolves itself within less than an hour.

With the launch of Amazon’s RDS this week realtime resizing of a database may finally be a reality, but for now message queueing can definitely be used as an effective scaling strategy.

This morning Jeff Bar announced a new service offering from Amazon Web Services (AWS): Amazon Relational Database Service.

The Amazon Relational Database Service (or RDS for short) is a new web service that allows users to create, manage and scale relational databases without having to configure the server instance, database software and storage that the database runs on. In short, this is a service that has the potential to take much of the headache out of database management.

» The current setup

At Galaxy Zoo we run our database on a combination of MySQL 5.1, EC2/Ubuntu Hardy and XFS/EBS storage. While there are some excellent guides on how best to configure a database running on AWS, operating a database in a virtualised environment requires that you plan for the worst case scenario of the virtualised server failing and the filesystem disappearing along with it. Because of this the steps required to configure a new database using ‘persistent’ storage (i.e. on an Elastic Block Store volume) are numerous:

1. Launch a new EC2 instance

Launching a new EC2 instance and installing the database software is pretty simple however for convenience we have the Galaxy Zoo database image saved out as a custom AMI.

2. Update server and install database engine

Launching a new EC2 image without a database engine installed means that you probably need to update the server software (e.g. apt-get update on Ubuntu) and then install MySQL or your database of choice.

3. Create a new Elastic Block Store volume and attach it to your instance

Next up you need to create a new EBS volume, attach it to the EC2 instance and format the filesystem on the EBS volume.

4. Create database

Setting up a new database instance on EC2 is clearly non-trivial and requires knowledge of EBS, mount points, filesystems, not to mention configuring the MySQL settings for the chosen size of EC2 instance that you have.

» A different way?

With the introduction of RDS, Amazon has removed almost all of the difficulty in setting up and configuring a new MySQL database that is both scalable and reliable.

Creating a new database instance now is as simple as issuing a single command:

>> rds-create-db-instance --db-instance-identifier mydatabase --allocated-storage 20 --db-instance-class db.m1.small --engine MySQL5.1 --master-username root --master-user-password password

With this command I have created a new m1.small MySQL 5.1 database server with 20Gb of storage and configured the master username and password. Provisioning a new RDS instance took a few minutes and during the provisioning you can check on the progress with the command:

>> rds-describe-db-instances

Once available, your new RDS instance is given a hostname that you can then use to connect with the standard MySQL port of 3306. Actually, it’s not quite that simple – before you can connect you need to assign which AWS security groups are allowed to connect to your RDS instances. I found this step a little confusing but essentially you need to configure is which EC2 instances running under their respective security groups are allowed to connect. For Galaxy Zoo, we have a default security group for all of our web servers called ‘web’ and so to allow access from these servers I had to add this ‘web’ security group to the defaults for the RDS servers:

>> rds-authorize-db-security-group-ingress default --ec2-security-group-name web --ec2-security-group-owner-id 1234567789

» The devil is in the details

At this point you have a RDS instance running MySQL 5.1 ready and waiting to serve up your databases. That’s not where the benefits stop though – not only do you get the ease of creating new database instances but there are some very nice extras you also get by using the service.

Scaling/resizing

At Galaxy Zoo, we usually our main ‘classifications’ database on a single EC2 small instance. In the last 8 months we’ve received something close of 45 million classifications and while the database has started to get a little sluggish, by writing user classifications to SQS and processing them asynchronously we are able to keep the site feeling nippy. Each month however, we try to send a newsletter to our 250,000 strong community and the increased load that this causes on the database means that for a couple of days we switch to a m1.large instance. The overhead of switching database servers is pretty annoying – place the site into maintenance mode, stop the MySQL server, detach the EBS volume, launch a new EC2 instance, attach the EBS volume to the new server… the list goes on.

With RDS not only can you change the amount of disk space available to your RDS instance but you can also dynamically resize the server size (i.e. RAM/CPU). I can see that this is going to be a real win for us.

A tuned MySQL instance

If there’s on thing that my time at The Sanger Institute taught me, it’s that managing and scaling large databases is a dark art. For the majority of small web applications it’s not crucial whether the MySQL server configuration you’re running is absolutely optimised for your hardware however now that we’re reaching the limits of our current instance size, making sure the MySQL server is well configured is becoming important. Deciding how large your innodb_buffer_pool or key_buffer size should be is not obvious for most of us and so having a MySQL server configured to work well for the resources available to it is very comforting.

Over the next couple of days I’m going to be benchmarking our standard MySQL setup to see how it compares against a RDS instance with the same resources. Watch this space!

A few months back I wrote about how we switched the Galaxy Zoo HAProxy load balancers to Amazon Web Services (AWS) Elastic Load Balancers (ELB). At that point we had basically just swapped out HAProxy (running on its own EC2 small instance) for an ELB but weren’t making any use of the auto-scaling features also on offer.

For the past few days I’ve been playing around with auto-scaling our API layer with the ELB that’s already in place and this morning I pushed the changes into production.

» Getting started

As I mentioned earlier, we already had an ELB in place so we didn’t need to create a new one – instead we’re adding here auto-scaling to an ELB that’s already in place. For completeness however, this is the command used to create the original ELB:

>> elb-create-lb ApiLoadBalancer --zones us-east-1b --listener "lb-port=80, instance-port=80, protocol=TCP" --listener "lb-port=443, instance-port=8443, protocol=TCP"

» De-register existing ELB instances

As we already had a couple of instances registered with the ELB I found the easiest way to get auto-scaling up and running was to remove the existing instances before proceeding:

>> elb-describe-instance-health ApiLoadBalancer

INSTANCE i-abcdefgh InService
INSTANCE i-ijklmnop InService

>> elb-deregister-instances-from-lb -lb ApiLoadBalancer --instances i-abcdefgh i-ijklmnop

No Instances currently registered to LoadBalancer

» Create a launch configuration

Before you can introduce auto-scaling you need to have a couple of things in place – an Amazon Machine Image (AMI) that upon boot is immediately ready to serve your application and a launch configuration compatible with your currently ELB-scaled nodes (security groups etc.).

Depending upon your setup, always having an AMI ready to launch with the latest version of your production codebase is probably the hardest thing to achieve here. Once you have your AMI in place and your security group and key-pair settings to hand you’re ready to create your launch configuration:

>> as-create-launch-config ApiLaunchConfig --image-id ami-myamiid --instance-type m1.small --key ssh_keypair --group "elb security group name"

OK-Created launch config

» Create an auto-scaling group

Once you have a launch configuration in place it’s time to create an auto-scaling group. Auto-scaling groups need as a minimum to know what launch configuration, which load-balancer to use, which availability zone and the minimum and maximum to scale to.

We never run the Galaxy Zoo API on anything less than 2 nodes and so to create our auto-scaling group I issued a command something like this:

>> as-create-auto-scaling-group ApiScalingGroup --launch-configuration ApiLaunchConfig --availability-zones us-east-1b --min-size 2 --max-size 6 --load-balancers ApiLoadBalancer

OK-Created AutoScalingGroup

At this point it’s worth noting that although we’d removed all of the instances being load balanced by the ApiLoadBalancer ELB, because the auto-scaling group set a minimum number of instances of 2 checking the status of the auto-scaling group showed that 2 new instances were spinning up:

>> as-describe-scaling-activities ApiScalingGroup

ACTIVITY 78bf4e0d-f72b-4b5b-a044-6b99942088ed 2009-08-24T07:19:28Z Successful "At 2009-08-24 07:16:12Z a user request created an AutoScalingGroup changing the desired capacity from 0 to 2. At 2009-08-24 07:17:17Z an instance was started in response to a difference between desired and actual capacity, increasing the capacity from 0 to 2."

I don’t know about you but I think that’s pretty AWESOME!

» Create some launch triggers

To complete the auto-scaling configuration, you need to define the rules that increase and decrease the number of load-balanced instances. Currently we have a very simple rule based upon CPU load – if the average CPU load over the past 120 seconds is greater than 60% we introduce a new instance, if the CPU average drops below 20% then we remove an instance:

>> as-create-or-update-trigger ApiCPUTrigger --auto-scaling-group ApiScalingGroup --namespace "AWS/EC2" --measure CPUUtilization --statistic Average --dimensions "AutoScalingGroupName=ApiScalingGroup" --period 60 --lower-threshold 20 --upper-threshold 60 --lower-breach-increment=-1 --upper-breach-increment 1 --breach-duration 120

OK-Created/Updated trigger

These triggers will almost certainly require refinement but helpfully the as-create-or-update-trigger command will create a new trigger if one doesn’t exist or update an existing trigger based upon the new parameters.

» That’s it!

Once again it’s been a breeze to introduce the latest AWS features into our production stack. Moving Galaxy Zoo to AWS has completely changed the way we think about running our web applications – we’ve gone from having a group of ‘pet’ servers we each know the name of to having a fault-tollerant, auto-scaled web-stack ready for the future.

Over the last few days we’ve been running a little project to see if the Galaxy Zoo community can help find new supernovae from the Palomar Transient Factory). Turns out it works pretty well.

Developing the website to power Supernova Zoo was a fun challenge; the Supernova and Galaxy Zoo websites look pretty similar and obviously share many features but there were some new problems to solve that we hadn’t faced before…

A moving target

Galaxy Zoo 2 had a static number of galaxy images to classify. Within the Galaxy Zoo domain model, we refer to an image as an ‘Asset’, Galaxy Zoo 2 had 245609 Assets to classify. One of the most exciting things about Supernova Zoo is that images are taken from the Palomar Transient Factory in near-realtime and sent up to the website for analysis. Being able to handle these images in an automated fashion is crucial and we’ve built API methods for uploading and auto-registering new Assets with the Supernova Zoo website.

Priority assets

For Galaxy Zoo, each Asset in the database has an equal priority of being shown to a Zooite. Upon reaching the classification interface, the Asset presented is therefore essentially random. For a static number of Assets this works well however for Supernova Zoo we wanted something a little different. The very nature of Supernova hunting means that you want to find the newest supernovae as quickly as possible and so to help us with achieve this we implemented a couple of new features:

Asset priority – When serving up an Asset to the Supernova Zoo interface we pick from the most recent supernova candidates. That way we are always going to be classifying the newest candidates first before heading further back in time to look at older ones.

Asset escalation – So that we could alert Mark and Sarah at the WHT to new supernova candidates as rapidly as possible we needed a mechanism for escalating the priority of the Asset in the system. We achieved this by essentially ’scoring’ your classifications as they came in. When creating the decision tree we attached a score to some of the answers. When your classification was complete we kept an average total score for the Asset that you had just classified. By keeping track of the scores as you classified, if you gave the ‘correct’ sequence of answers for a potentially real candidate, then it would immediately become a higher priority target to show to other Zooites.

A retrospective

Supernova Zoo was our first opportunity to test the codebase that myself and the team at SIUE have been working hard on for the last few months. Handling a continual stream of new Assets and changing the behaviour of the system in real time based on your classifications has been a fun challenge and overall we’re pretty happy with the results.

In the next day Supernova Zoo will be taken offline so that we can have a good look at the results from the past few days. Based upon your excellent feedback there will almost certainly be some tweaks to the classification interface and refinements to the decision tree. Supernova hunting is a very different challenge to galaxy classification and we’re delighted that our Zooites appear to equally adept at classifying galaxy morphologies as finding new supernovae!

For the past few months we’ve been loading balancing the Galaxy Zoo web and API layers using HAProxy. Overall this has worked pretty well; HAProxy is easy to configure and hasn’t missed a beat, however having to spend $150 per month just to load balance our other EC2 nodes seems a little excessive.

For some time Amazon have been promising load balancing and auto-scaling as part of their EC2 offerings and a few weeks back now a public beta of their auto-scaling and load balancing products was announced on their blog.

It’s been a busy few weeks at the Zoo and so I’ve only just got around to playing with the new tools and I have to say, I’m impressed. In approximately 15 minutes I’ve managed to swap out one of our HAProxy nodes for an elastic load balancer (ELB). Count the steps:

1. Create a new load balancer

First we need to create an elastic load balancer. Note I’m using http and https, unfortunately ELB doesn’t have SSL termination capability so you need to route traffic on port 443 to an alternative port (in my case I’m routing SSL to port 8443).

>> elb-create-lb LoadBalancerName --zones us-east-1b --listener "lb-port=80, instance-port=80, protocol=TCP" --listener "lb-port=443, instance-port=8443, protocol=TCP"

2. Register the instances to be load balanced

>> elb-register-instances-with-lb LoadBalancerName --instances instance_id

3. Create a CNAME record for the elastic load balancer

Each load balancer is given an AWS hostname such as loadbalancername-123456789.us-east-1.elb.amazonaws.com. This needs to be aliased to the actual hostname you want to use your load balancer using a CNAME record.

4. Add a health check

Last thing to do is add a instance health check to the load balancer so that it doesn’t send requests to a unresponsive node. You can configure a health check like this:
>> elb-configure-healthcheck LoadBalancerName --healthythreshold 2 --interval 30 --target "TCP:8443" --timeout 3 --unhealthythreshold 2

This health check is set up to verify the status of each load balanced node every 30 seconds on port 8443, removing it from service if it fails more than two times.

5. Done!

And that’s it. A couple of points to note: At the moment it’s a limitation of the service that you can’t have a root domain url load balanced using ELB. This is basically because you can’t have a CNAME record pointing to the root of a domain. This is a known limitation and and should be fixed in the next release. Also elastic load balancing obviously isn’t free (what is these days). The good news is though, at $0.025/hour, running an elastic load balancer is significantly cheaper than running a single EC2 HAProxy node ($0.10/hour).

» What’s next?

Next up is configuring auto-scaling and monitoring using cloudwatch. More of that later…

I’m currently in Las Vegas attending RailsConf 2009. This morning I heard Heroku’s Adam Wiggins give an excellent overview of Rails Metal, Rack and Sinatra.

Some time ago now, Rails adopted Rack as its middleware layer. For those not in the know (myself included before Adam’s talk), according to RailsGuides Rack is:

Rack provides a minimal, modular and adaptable interface for developing web applications in Ruby. By wrapping HTTP requests and responses in the simplest way possible, it unifies and distills the API for web servers, web frameworks, and software in between (the so-called middleware) into a single method call.

Metal is essentially a thin wrapper around the Rack middleware layer of Rails. Why is this important? Well, by dropping down into Metal it’s possible to completely bypass the Rails framework and squeeze the absolute maximum performance out of your stack. Specifically this can be useful for a common request to your application where the response time is crucial and you want to avoid the extra overhead of passing through the Rails routing mechanism before serving a response.

» The Test

OK, so enough talk, what do the numbers look like? This is by no means meant to be a thorough test of all possible permutations of Rails, Sinatra and Metal, rather I’m interested in replacing a simple API method with a Sinatra application and a Metal endpoint. The API I’m testing is the Galaxy Zoo API layer. Within the Galaxy Zoo API we have the concept of ‘Assets’. An Asset is something like a SDSS galaxy image and a frequently accessed API url looks like:

http://api_url/api/assets/:id

This API call returns a simple XML snippet that looks something like this:

<?xml version="1.0" encoding="UTF-8"?>
<asset>
  <id>1</id>
  <location>http://s3.aws.com/1.jpg</location>
  <project_id>1</project_id>
  <external_ref></external_ref>
</asset>


I used Apache benchmark to test each option. Passenger 2.2.2 / Rails 2.3.2 and my MacBook Pro (2.53 GHz) were used to serve the application. Also, to ensure a reasonably fair test I rebooted the OS for each variant and ‘warmed up’ Apache by running the test 4 times before taking the benchmark results from the 5th and final time I issued this command:

ab -n 1000 -c 4 http://api_url/api/assets/1

This is basically making 1000 requests with 4 concurrent connections.

» The Results

OK, so first up I used a standard Rails controller action, the code for which is shown below:

def show
  @asset = Asset.find(params[:id])

  respond_to do |format|
    format.xml { render :xml => @asset.to_xml }
  end
end


This came out at a very reasonable 230 requests per second:
Requests per second: 229.64 [#/sec] (mean)

Next up I added in a Sinatra ‘application’ to respond to the api/assets/:id url. Because of the way that Rails uses Rack, the Sinatra/Metal endpoints are picked up before the Rails routing mechanism kicks in, no modification is therefore required to the routes.rb config to make the Sinatra application pick up the request url.

By default Sinatra/Metal endpoints are picked up if they are placed in the RAILS_ROOT/app/metal/ and have a class name that represents the filename for the Sinatra application:

RAILS_ROOT/app/metal/sinatra_asset.rb

require 'sinatra'

class SinatraAsset < Sinatra::Application
  set :environment, 'production'

  get '/api/assets/:id' do
    Asset.find(params[:id]).to_xml
  end
end

Benchmarking Sinatra produces the following results:
Requests per second: 416.61 [#/sec] (mean)

Wow! So we’ve gone from ~230 requests per second using a standard Rails controller action up to over 400 requests per second using Sinatra. This is obviously a pretty serious speed bump and for really not very much work.

Finally I tested a Metal endpoint to intercept the same request url. Once again, Metal endpoints need to be installed in:

RAILS_ROOT/app/metal/metal_asset.rb

class MetalAsset
  def self.call(env)
    url_pattern = /^\/api\/assets\/(\d+)$/

    if m = env['PATH_INFO'].match(url_pattern)
      asset = Asset.find(m[1])
      [ 200, {"Content-Type" => "text/xml"}, asset.to_xml]
    else
      [ 404, {}, '']
    end
  end
end


So Sinatra was fast – how fast is Metal? Well it’s pretty nippy:
Requests per second: 522.12 [#/sec] (mean)

» Conclusions

As I mentioned earlier, this is by no means meant to be a through test of how Rails controller actions perform compared to their Sinatra and Metal equivalents, however the numbers are pretty spectacular: a bare Metal endpoint more than doubles the number of requests this application can handle per second. This is not to say that the Sinatra results weren’t pretty damn good too – using Sintara gave an 80% speed boost for this simple API request.

It seems clear that a significant speed boost can be had by getting down to ‘the metal’. Personally I prefer the clear syntax of Sinatra over the url regex that Metal requires to achieve the same result, although the additional ~100 requests per second that Metal offers over Sinatra is hard to ignore.

David Heinemeier Hansson talked this week about the refactoring that’s going on with the Rails routing mechanism for the upcoming Rails 3 release so it’s possible that these numbers could significantly change when Rails 3 makes it into the wild. For now though, if you’ve got a Rails application with a frequently accessed url, drop in a Sinatra application or a Metal end point and watch it fly!

Last week at Galaxy Zoo as part of the 100 hours of Astronomy we challenged the Zooites to do 1 million clicks in 100 hours – a big challenge. In the week before the 100 hours we’d received about 1 million clicks so although the challenge of reaching 1 million was a big one but it seemed perfectly realistic. I don’t know about everyone else but I couldn’t stop refreshing the Galaxy Zoo homepage to check on the latest total. In the end we reached our goal of 1 million clicks about 12:45pm on the Saturday a mere 72 hours into the challenge!

» 1.45 million clicks in 100 hours

I wondered what would happen once we’d reached 1 million – would people stop classifying? Absolutely not! In the final 28 hours we added a further 450,000 clicks to the Zoonometer™ total reaching a grand total of 1.45 million clicks in 100 hours… Or did we?

» What the Zoonometer™ should have been reading

As I mentioned earlier, in the week before the 100 hours challenge we’d had about 1 million clicks and so with all the extra publicity surrounding the 100 hours of Astronomy I was secretly hoping that we might get closer to 2 million clicks. It turns out we did…

When writing the code for the Zoonometer™ I had to make a few changes to the Galaxy Zoo website and API. Without really thinking I decided that rather than count the total number of clicks each time we wanted to update the Zoonometer™ (a MySQL query that takes about 6 seconds) I’d keep the total as a separate counter. Each time someone classified a galaxy I’d add 1 to the total and this way the current total could be checked very quickly and so we could update the Zoonometer™ more frequently.

What a great idea Arfon! Erm no… It turns that this was a really bad idea and here’s why.

In the API we have a Project and Classification model. The Project has_many classifications and so I was keeping a counter column on the Galaxy Zoo project entry. In the code I had something like this as an after_create callback on the Classification model:

def update_counter
  self.project.classification_count = self.project.classification_count + 1
  self.project.save
end

Simple right? When a classification comes in, add one to the project total and keep going. I had tests, the method worked, everything looked peachy. What I didn’t consider is what happens when you’re getting 30-40 classifications per second. Let’s consider what happens when two (or more) classifications are processed simultaneously. If the database is very busy then it’s possible that in the time it takes to create the classifications, when both after_create callbacks run the classification_count column on the project is the same. That is, if both callbacks get a value of 1000 for the current project classification_count then they are both going to update to the new value of 1001. Oh dear.

So what does this mean? Well the bad news is that the Zoonometer™ was reporting the wrong total. The great news is that we didn’t record 1.45 million clicks in the 100 hours of Astronomy, we actually had 2,617,570! Yes you heard me, that’s:

Turns out that Zoonometer™ was a little off the mark…

» A retrospective

So 2,617,570 not 1,450,000 clicks? Pretty impressive stuff. I knew we were busier than the Zoonometer™ was reporting, I just couldn’t figure out why it wasn’t counting properly! 2,617,570 is an amazing number to have reached in just 100 hours and I’d like to thank all the people who worked so hard to help us reach this total.

I’m putting this down to experience. To be honest I’ve never worked on a project quite so popular as Galaxy Zoo and problems like this only arise in very busy environments such as ours. When we next have to bring out the Zoonometer™ you can be assured of an accurate total!

Getting ActiveRecord to talk to multiple databases is easier than you might think.  It’s possible to override the connection settings in database.yml at the model level by doing something like:

establish_connection(
:adapter => "mysql",
:host => "localhost",
:username => "myuser",
:password => "mypass",
:database => "somedatabase"
)

Calling the establish_connection method at the model level simply overrides the ActiveRecord connection object for the local model.

At Galaxy Zoo we needed to do something a little different: initially we were write dominated at the database layer – 16 million classifications in the last month peaking at about 50 classifications per second on launch day. However as things have settled down and we’ve been adding more user-centric features to the Galaxy Zoo site, we’ve been finding that a significant amount of out database load has been coming from more complicated queries (reads) rather than lots of writes.

An ideal solution for a situation like this is to introduce some kind of MySQL replication thus distributing the load across multiple databases. Rather than introducing the complexity of offset primary keys in a Master/Master configuration we’ve opted for a standard MySQL Master/Slave configuration sending the writes to the Master and reads to the Slave. But how to accomplish this with ActiveRecord?

» Enter Masochism

Masochism is a Rails plugin by Rails core team’s technoweenie (Rick Olson). It works by overriding the ActiveRecord connection object with ConnectionProxy that (by default) sends writes to the Master MySQL database and reads to the Slave. We’ve been running in production now for about 2 weeks using Masochism and so far there’s not much to say other than it works!

We’ve made a couple of optimisations along the way after examining the production logs: When writing a classification to the database there’s a couple of writes, then some reads, then some writes… In the log you see something like this:

Switching to Master
Switching to Slave
Switching to Master
Switching to Slave
Switching to Master
Switching to Slave


Obviously this switching between the Master and Slave database repeatedly in the same method call is less than ideal. Thankfully it’s possible to override the database so that within the method only one of the databases is used:

around_filter ActiveReload::MasterFilter, :only => [:create]

Masochism is a nice solution to a common problem – using ActiveRecord in a replicated database environment. I can already see us outgrowing Masochism – specifically it doesn’t support multiple slave databases which is a shame. When that day comes we’ll no doubt look to an alternative such as FiveRuns’ DataFabric or MySQL Proxy. But for now, Masochism works, and I can highly recommend it.

So on the morning of launch we were running a stack of 14 servers – two load balancers, 5 web nodes, 5 API nodes and a database layer.  Because AWS makes it so easy, we we’re also taking hourly EBS snapshots of the database stored in S3.

This setup kept us going for about the first hour until Chris appeared on BBC breakfast and the web traffic went through the roof.  Thanks to some seriously smart auto-bootstrapping of AWS EC2 nodes we were able to easily scale the web layer to 10 servers to handle the load, combined with a more beefy MySQL AWS instance and some on-the-fly code optimisations we managed to keep the site up.

I’ve been lucky enough to work on some big Rails projects in the past but this was my first experience of Rails in a high-traffic environment.  If I had to do the launch again would I do anything different?  Sure.  Could we have done with some more time to test the production stack?  Definitely.  But we survived the flood and I can’t wait for the next big launch…