StyleFeeder Tech Blog » linux

Moving to another cloud

Philip Jacob — Tue, 25 Aug 2009 21:48:20 +0000

We are in the process of migrating one of our backend dataprocessing servers from a legacy hosting company in NYC to Contegix. What’s unusual about this transition is that we’re moving the machine onto Contegix’s new cloud platform rather to a traditional server. We’ve noticed a few things already. When we were copying over a huge backup of our databases, we noticed that they were transferring across the network from NYC to St Louis at 93Mbps, which is not frigging bad! As I write this, we’re loading over 100Gb of data into a MySQL server on our new Contegix cloud machine at ~30K blocks/second (as measured by vmstat), which means that this thing has lightning fast i/o… not surprising since the storage is on an EqualLogic SAN (Update: we later saw this increase to ~70K blocks/second).

The differences between this cloud platform and EC2 (which we still use for some other needs) are striking. The application that we will host on this new vm sometimes needs a lot of memory. With Contegix, we can grow that all the way up to 128Gb with 32 cores. Amazon doesn’t even come close to that – their max is 15Gb. Or you can figure out how to distribute your application over a bunch of hosts. But sometimes you just need 20Gb of memory and all the problems go away. Plus we don’t have to compete for these resources – they’re guaranteed to us.

I also like the fact that the machine doesn’t disappear into oblivion when it reboots, which is a feature (?) of EC2 instances. We can grow our storage needs past that point that I care to think about on this platform as well. Plus, we get all the Contegix support that we want if we choose to do crazy things with this host.

The virtualization technology is VMWare ESX, which is darn cool stuff (having just set it up on an integration server here a week or so ago, I have to say that I like what I have seen so far). We’ve already seen our VM get hot-migrated to another physical box in order to maximize the resources available to us. Things got slow for a little bit, but then they got lightning fast. I think we were copying data into the machine at that point and saw no impact to open connections, etc. Don’t ask me why, but I’m still surprised that this works reliably.

So far so good. We’ll report back with more later.

doing a tail on rotating log files

kilby — Fri, 07 Aug 2009 19:59:40 +0000

We learned something today, not something new, but new to us. If you do a tail -f to view the end of a log file, it all goes great until your logging system rotates out the file. Then you’re stuck wondering if the program has halted or if your ssh connection died or if you hit Ctrl-S and froze your terminal.

But if you use tail -F (note the big F) it will check to see if the file has been rotated out and another put in its place, and will resume tailing on the new file. Happy tails to you!

Congrats to Contegix

Philip Jacob — Tue, 28 Apr 2009 20:44:16 +0000

I’ve been accused by a few people of being a Contegix fanboi, which is an label that I completely accept and agree with. We’ve been hosting with them for around two and a half years after I decided that I didn’t want to handle a growing cluster of machines myself. Since I ran a small hosting operation on the side for many years, I have some opinions about how things should be done. When I found out that Contegix had not only institutionalized many of my beliefs as part of their support process, but also shared my opinions about how customer service should be run, I was thrilled.

We’ve grown with them over the years and they’ve kept up and have gone overboard for us on more than a few occasions, including dedicating a pair of Foundry hardware load balancers to us after our required configuration wasn’t available in a shared configuration. Their support team is legendary and these are not isolated incidents. Normal mode of interaction with them basically involves:

Send email to Contegix support requesting some arcane version of a custom built source package running in some non-standard way.
Wait 30 seconds.
Check email and read the “we are reviewing this and will reply shortly” message, which, as far as I can tell is actually written by a human each time. It means that someone with a name is looking into things for you.
Wait two to five minutes.
Check email and read the “we have completed this request” message.

It’s like that every time. And it’s been like that ever since we started working with Contegix.

When I came home yesterday to see that they had won the “Best Linux Friendly Hosting Provider” category in the readers’ choice wards in the latest issue of Linux Journal, it came as absolutely no surprise at all. With service and support like this, it’s well deserved.

I’d like to the LJ page on their site… if it existed, which does not appear to be the case as of this writing. It’s an odd twist that my paper magazine is more current than the LJ website.

How Big is that Process?

Jason Rennie — Fri, 15 Aug 2008 18:47:52 +0000

As I have, you might be tempted to think that top “VIRT” and ps “VSZ” values represent the entire size of a process, including all shared libraries and swapped-out data. Today, we learned that this can’t possibly be the case:

top - 14:41:53 up 86 days,  3:22,  3 users,  load average: 13.57, 6.67, 4.00
Tasks: 162 total,   3 running, 159 sleeping,   0 stopped,   0 zombie
Cpu(s): 11.7%us,  5.7%sy,  0.0%ni, 45.0%id, 37.1%wa,  0.0%hi,  0.4%si,  0.0%st
Mem:   8102704k total,  8056796k used,    45908k free,     9004k buffers
Swap:  4008208k total,  1274816k used,  2733392k free,  2075736k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 1472 mysql     15   0 19.8g 4.0g 7440 S    1 52.1   1374:57 mysqld

You can’t have a 19.8 gig process on a machine with only 12 gigs of memory! One anonymous commenter of Virtual Threads: Understanding memory usage on Linux suggests a plausible explanation: memory fragmentation.