Note that if you’re streaming large audio or video files, this list may not be a good one for you.  The questions are biased towards StyleFeeder’s needs, namely the fact that we have tens of millions of small product images floating around.

CDN Evaluation Criteria:

1. Do you support HTTP compression?  Does content have to exceed a minimum size in order to be compressed?  (i.e. content less than 2Kb is not compressed)
2. Do you allow us to override the Expires headers being sent by the origin?
3. Will you obey long Expires headers (5 years) and cache content accordingly?
4. Once a piece of content has been cached on your network, under what conditions would your CDN re-request that content from the origin?
5. Do you allow full and partial (based on regexes) cache flushes?
6. Can you demonstrate that you’re better than one of the “value” CDNs?
7. How many nodes do you have?
8. Do you have a lazy-loading caching proxy scheme (like Akamai, Panther, Voxel) to allow for easy deployment?
9. Is there an API we can use to add or flush content?

Let me know if you have other questions that should be on the list!

First, you may ask, why use AWS at all? We don’t use it for everything, but this particular set of processes handles image storage, resizing, and delivery, through a content delivery network. We have many millions of images. If we wanted to stick them on a big file system, we would run out of inodes, so that’s out. We could put them in a sharded mysql database, and we have a big sharded mysql infrastructure for a lot of our data anyway, but that’s not how we started out, and it’s not exactly using the database for database-y things. To do this ourselves, we would have had to install a distributed file system of some kind, which seemed like a lot of work, so we decided to use AWS’s S3 for storage. Once your images are in S3, there are certain things it makes sense to do in EC2, since an EC2 instance can network to an S3 bucket pretty fast. Vendor lock-in? You bet, but hey, it’s giving us a pretty good value. To get the images out of S3 and on their way to our users’ computers and other shopping-enabled gadgets, we have some EC2 instances that resize them, store the resized versions for future use, and serve them up. The resizer/cacher logs indicate that although on average we don’t serve a given image up to too many users, we’re serving it for the Nth time, where N>1, about 96% of the time. If the CDNs could just keep them all around, forever, our servers wouldn’t be working as hard as they do. Our actual origin hit rates at the CDNs are something like 50%. What’s up with that? They can’t handle sparse sets of content? I don’t remember a disclaimer about that. Can you hear me, CDN people? I’m talking to you!!

But I digress. Back to AWS. AWS is a good value, of course, only as long as we’re efficiently using EC2 resources. That’s where the clusters come in. We’ll talk about this in terms of what you do in RightScale, rather than AWS alone. RightScale used to offer a lot of features that AWS just plain didn’t have. AWS has been filling those gaps, but we’re not planning to ditch RightScale any time soon, because they still make AWS resource management a lot easier than the raw services in the Amazon interface. If you’re a hard-core command-liner, this blog post and this manual tell you everything you need to know. We’ve gotten into the habit of doing these kinds of things with the free features of RightScale, until we get to the point where scripts save us real time or money. Here’s what we did:

• Built up a medium-sized resizer/cacher, to do any and all of the miscellaneous things our various big boxes were doing.
• ‘Bundled’ it into an ami image.
• Created a ‘Server Template’ based on that image. Huh? Why do I need a ‘Server Template’? What’s wrong with the image? You can’t use raw images as the basis for servers in a ‘Deployment’ (see below). You need to have a server template. Does it have to be that way? I can’t see why, but hey, this is a free service (RightScale, that is), so who’s complaining?
• Wrote some ‘Right Scripts’ so our server can start up and add itself into the mix in a fully functional state (starting apache and whatnot).
• Created a ‘Deployment’ and added the server template to it.
• Created more server templates, based on the same image, and added them to the deployment.
• Created a ‘Load Balancer’, and registered the servers in the deployment with it ‘on boot’, and, for the ones that were already running, ‘now’.
• Put a CNAME in our DNS for the load balancer. Huh? Couldn’t we just take an elastic IP address (one of those ‘permanent’ ips Amazon gives you) and assign the thing to that, so it takes over right away for the big old instance that was handling this? No, no we couldn’t. This is a pay service, and that sucks, but with a short time-to-live, it only sucks for a few minutes, so we’re going to overlook this. AWS seems to want to be able to scale the load balancer, or move it around, whenever they want, which I guess you might need in some circumstances. Note to people running java: watch out for the infamous java DNS caching problem, if you have jvms that talk to one of these load balancers. If Amazon switches the ip underneath you, your jvms will be talking to the void, unless you’ve configured the jre properly.

Now we can start all the nodes in our clusters, or just some of them, or whatever. If we’re feeling really ambitious, we can set these to auto-scale, but we’re already saving quite a bit of money and serving things faster than we were, so that will be for another day.

How is it deciding which node gets the traffic? The documentation seems to say that it’s round robin between availability zones, and then based on load within them. On this page Amazon says “Elastic Load Balancing metrics such as request count and request latency are reported by Amazon CloudWatch.” So, based on load, but measured in a black-box-y way. “Elastic Load Balancing automatically checks the health of your load balancing Amazon EC2 instances. You can optionally customize the health checks by using the elb-configure-healthcheck command.” You can do this in the RightScale interface as well. You can either accept the default that it checks for the presence of a TCP listener on port 80 (target=”TCP:80″), or you can give it something like “HTTP:80/path/to/my/image.jpg” that returns a “200 OK” when all is well. The default seems to work surprisingly well for these particular CPU-intensive activities that we’re clustering. We don’t see one server with a load of .3 while another is at 4. We do see some occasional differences, but they seem to even out pretty fast. We’ll be more precise if the differences start to get out of hand.

About a month or two ago, we looked in more detail at the chain of events that occurs when a browser hits our site to see if there were any possibilities for making things better.  This is increasingly hard for us to do given that we’re already serving stuff up pretty quickly, especially when you consider the size of our dataset and the sparse nature of our requests.

Anyway, we noticed some room to improve our DNS numbers, which I mentioned in a post last April.  Shortly thereafter, I met up with Jeremy Hitchcock from Dynect and he said that Dynect was definitely a lot faster than anybody else.  How could I resist a claim like that?  Especially if it was true, since it had some obvious benefits for StyleFeeder :)

So, I ran some tests using Pingdom for the entire month of June, 2009 that compared Dynect, Enom, DNSMadeEasy and Jerky.  Huh?  Jerky?  What’s that?  It’s our control point; more specifically, it’s a personal server owned by Eric: an old underused single Pentium 4 with ~2Gb of memory sitting on a Cogent network in a datacenter in Waltham, MA.

I mentioned my intent to do this testing to Dharmesh a few months ago and he gave me permission to test OnStartups.com and hubspot.com as part of my experiment, so that’s what I was testing against those services.

The short story is that I took the detailed logs from Pingdom, processed them with R and made some pretty graphs.  The raw numbers are here, but you can click through the graphic to see it in more detail.  Basically, I threw away the slowest 0.5% of the requests and made histograms of the frequency of the request times.  Note that the X axis has the same scale in all cases, so you can compare easily.

As you can see, not only does Dynect have the fastest response times by a wide margin, their times also exhibit a much smaller standard deviation than anything else that I tested (I should note that I didn’t test UltraDNS because they are stupidly expensive and I had a bad experience with them in the past, but they would be worth looking at if you are doing a bakeoff).

Enom’s DNS service is included with their domain registration service while DNSMadeEasy costs money.  All in all, Jerky did pretty well considering what it is.  Obviously, you don’t get redundancy out of a single box, so there are other features that the other DNS vendors give you that are very much worthwhile.

But the real news here is that Dynect demolished the other participants in this test with a 37ms response time, less than half that of the first runner up.  It’s not everyday that you can chop 40-70ms off of your mean response time, so when those opportunities arise, it’s definitely worthwhile.  Companies like StyleFeeder that have tons of new users who don’t have these DNS entries in their resolver caches will definitely benefit from the speedup.

If you don’t like my test, I’m happy to run it again under different conditions, so let me know if you think there’s a flaw in how I processed the data.

We currently use DynDNS for DNS service. We switched to them probably two years ago after I decided that UltraDNS was stupidly expensive for what they were giving us. In the end, I managed to cancel our UltraDNS service over the phone by providing publicly available information, which was awesome as their standard contract at the time ran on a yearly basis and they ended up screwing me for another year of service; the whole thing left a very bad taste in my mouth. Although, UltraDNS runs some TLDs and big sites, so they obviously have some redeeming qualities. Pricing isn’t one of them. DynDNS was costing us 60x less at the time.

I’ll use the same format as JohnPhilipGreen’s original post to present my numbers. One difference is that my figures were generated using 30 days worth of data, not 3 days as John used.

John basically disqualified DNSMadeEasy as part of his tests because the results were so erratic. I haven’t noticed anything that warrants disqualification like that, but they’re clearly the slowest of the three that I was monitoring.

Some notes: the HubSpot folks are friends of ours, so I figured they wouldn’t mind if I put monitors on their service. Contegix (by far the best managed hosting company I’ve ever worked with) runs our non-cloud infrastructure for us; they offer DNS service at no additional charge to hosting clients and we use it for one of our auxiliary domains, primarily for legacy reasons at this point.

Dynect was the big winner in John’s shootout, which is interesting, given that DynDNS and Dynect are part of the same overall parent company. They currently charge USD $27.50 per year, a far cry from what UltraDNS charges and – I’m guessing – substantially less than what Dynect charges. Is it worth it? For most small sites, probably not… for StyleFeeder, it may very well be worth switching to a faster provider, especially to gain a >50% reduction in response time depending on the cost (John was seeing 42ms as an average response time).

John, maybe you can republish your numbers after 30 days to see how things stand at that point?

Dan mentioned that someone he knows did some basic benchmarks that clocked in around 700 requests per second.  I wanted to see what our numbers were like.

(Before I share these numbers, I want to emphasize that I’m not ready to hang my hat on these numbers yet, but I figured I’d share them for comments.)

100,000 get requests executed serially:

Memcached: Requests per second: 684
MySQL: Requests per second: 884

Surprising, eh?  This is for the same data coming out of one our shards and the same data coming out of memcached.

I have more unanswered questions: instead of doing this serially, what happens when I have 20 concurrent threads pulling data out?  Does the memcached client library make a big difference?

I also wonder in what cases it makes sense to use memcached.  If you’re like us and have more data than you can reasonably hold in memory, you probably can’t use memcached unless you’re able to hit your main data store without a big penalty.  If you have an amount of data that can fit in memory, you should use something like Whirlycache (only relevant if you’re using a jvm), which did 2,500 requests per millisecond for the same test.

If you simply need to share data across a wide range of nodes, does memcached even make sense at that point?  Perhaps in the case of a more dynamic architecture, memcached and libketama are pretty key.  Rigging that machinery manually with a MySQL backend is possible, but not the kind of thing you’d want to focus on unless you’re doing systems work.

I’m curious to hear what people think, because there’s certainly a lot of conventional wisdom behind memcached that I can’t understand right now.  Francois seems to be in the same camp as me.