The differences between this cloud platform and EC2 (which we still use for some other needs) are striking.  The application that we will host on this new vm sometimes needs a lot of memory.  With Contegix, we can grow that all the way up to 128Gb with 32 cores.  Amazon doesn’t even come close to that – their max is 15Gb.  Or you can figure out how to distribute your application over a bunch of hosts.  But sometimes you just need 20Gb of memory and all the problems go away.  Plus we don’t have to compete for these resources – they’re guaranteed to us.

I also like the fact that the machine doesn’t disappear into oblivion when it reboots, which is a feature (?) of EC2 instances.  We can grow our storage needs past that point that I care to think about on this platform as well.  Plus, we get all the Contegix support that we want if we choose to do crazy things with this host.

The virtualization technology is VMWare ESX, which is darn cool stuff (having just set it up on an integration server here a week or so ago, I have to say that I like what I have seen so far).  We’ve already seen our VM get hot-migrated to another physical box in order to maximize the resources available to us.  Things got slow for a little bit, but then they got lightning fast.  I think we were copying data into the machine at that point and saw no impact to open connections, etc.  Don’t ask me why, but I’m still surprised that this works reliably.

So far so good.  We’ll report back with more later.

Dan mentioned that someone he knows did some basic benchmarks that clocked in around 700 requests per second.  I wanted to see what our numbers were like.

(Before I share these numbers, I want to emphasize that I’m not ready to hang my hat on these numbers yet, but I figured I’d share them for comments.)

100,000 get requests executed serially:

Memcached: Requests per second: 684
MySQL: Requests per second: 884

Surprising, eh?  This is for the same data coming out of one our shards and the same data coming out of memcached.

I have more unanswered questions: instead of doing this serially, what happens when I have 20 concurrent threads pulling data out?  Does the memcached client library make a big difference?

I also wonder in what cases it makes sense to use memcached.  If you’re like us and have more data than you can reasonably hold in memory, you probably can’t use memcached unless you’re able to hit your main data store without a big penalty.  If you have an amount of data that can fit in memory, you should use something like Whirlycache (only relevant if you’re using a jvm), which did 2,500 requests per millisecond for the same test.

If you simply need to share data across a wide range of nodes, does memcached even make sense at that point?  Perhaps in the case of a more dynamic architecture, memcached and libketama are pretty key.  Rigging that machinery manually with a MySQL backend is possible, but not the kind of thing you’d want to focus on unless you’re doing systems work.

I’m curious to hear what people think, because there’s certainly a lot of conventional wisdom behind memcached that I can’t understand right now.  Francois seems to be in the same camp as me.

Innotop (more info here) is sort of like the friendly unix top command, but instead for database status.  There are pages to show buffer statuses, deadlocks, i/o status, current queries, and lots more.  They all update on screen, at configurable increments.

MySQLReport from hackmysql.com runs a few status commands and formats them nicely on screen in a nice grouped format.  This guide summarizes the sections, which include more detail on many of the same things that innotop covered.  I find the sections on SELECT types, and InnoDB Buffer Pool use, are especially useful to me.

Using the command type summary, we discovered an inordinate number of com_rollback calls in our main database, which we were able to reduce by using this technique.  The root cause was Hibernate’s love of transactions, combined with connection pooling.  A simple driver parameter seems to clear it up.

After evaluating these options and a few ideas of our own for primary key generation, we settled on a simple algorithm motivated by group theory. The advantages of this algorithm are numerous:

• Short Keys (6 characters yield 57 billion unique keys using only alphanumeric characters)
• Universal Uniqueness (no guessing to which table a key value refers)
• Pseudo-randomness (keys don’t follow an obvious pattern)
• No Duplicate-Checking (keys are guaranteed to be unique until a limit is reached)
• Block Generation (keys are generated in blocks to minimize lock contention)

Our generator uses one tiny bit of group theory: if k and n are coprime (aka relatively prime), the sequence of numbers generating by successively adding k (mod n) will not repeat through the first n values. This leads to the following algorithm for generating unique keys:

• Pick a size n
• Pick a value k which is coprime with n
• To generate the next key: nextKey = (lastKey + k) % n

You’ll be guaranteed to not see duplicates until you’ve generated n keys. The sequence you’d see with n=5 and k=3 is { 0, 3, 1, 4, 2, 0, 3, … }.

Note that the choices of n and k are quite important—they must be fixed and can never change. However, selecting reasonable values is not difficult. For n, select a character set and string length, then set n to be the number of possible unique strings. To get the 57 billion value above, use a string length of 6 and a character set of [0-9a-zA-Z] (62 characters). 57 billion is simply the number of unique, 6 character alphanumeric strings (62^6). If you grow to the point that you are worried about key collisions, switch to using 7 character strings (where n=62^7, appx. 3.5 trillion). Note that conversion from the key number value to string value is simply a conversion from base 10 to base 62 (or whatever # of characters you are using).

For k, we need a value that is coprime with n. To achieve pseudo-randomness, k should also not be too small (the same order as n is a good choice). Note that this “randomness” is quite weak in a mathematical sense, but was sufficient for our purposes. One way to select such a k is to multiply together prime numbers larger than the character set size. For our example, a reasonable choice would be k=67*71*73*79*83*89. If you don’t have your own prime number generator, consult the bear.

To put this algorithm into practice, one needs to ensure that keys are generated serially. We did this by creating a table with a single row with a single column storing the last key value. When we want to generate a key (or block of keys), we start a SERIALIZABLE transaction, read the last key value, generate key(s) per the above algorithm, then write back the last key value we generated and close the transaction. To minimize contention and since next key computation is much faster than a transaction, we generate keys in blocks and serve them out of memory via a synchronized HashMap. This causes key values to occasionally be permanently lost when a webapp is shut down, but the lossage is too small to be of any real concern.

We’ve been using this system for many months now and have yet to run into any problems. It satisfies all of our current needs and has the advantage that it can easily scale either by using longer character strings or increasing the key generation block size. Furthermore, it seems to be extremely lightweight, exerting minimal pressure on our database. We would love to hear what other solutions for primary key generation are used. How does ours compare?