StyleFeeder Tech Blog

As part of our effort around product data feeds, we get a lot of delimited text files of varying size and content delivered on a daily basis.  Previously, we primarily used OpenOffice.org Calc to view these files, since it has a fairly flexible text import facility and can deal with CSV files pretty well.  For some of the larger files it could take minutes to open and more alarmingly minutes to close, and this was placing major time constraints when a large number of new files came in and needed evaluation.

A survey around the web showed a few different approaches to dealing with delimited files.  Most involved configuring a desktop database like MS Access or a local install of MySQL to import the file, but that would involve creating a table and running a loader.  Really all we want to do is view the file and see that the important fields are being filled in and that the values are not abusive of the column semantics.  Other approaches mirrored our own, using OpenOffice.org or MS Excel as a viewer; there were also a few specialized tools to open delimited files but not spreadsheets.  Having tried a couple of these, they didn’t impress with their speed or usability.

Our answer came from halfway around the world, in New Zealand.  Kiwi Log Viewer, a tool designed to view web server logs on Windows, could open up tab delimited files in an incremental fashion.  This provided a grid view of the file in a really speedy manner.  The problem is that most of our files are pipe delimited, and Windows isn’t so friendly at doing ’sed’ on a file before passing it into a program.  Seeing that we were 90% of the way to what we wanted, we wrote to their support line and sure enough they’ve added a configurable delimiter to the features of their next version, currently available in beta.  While we’re big Open Source fans here, a responsive software company that cares about their curstomers is the next best thing.  If you’re looking for a good program to view delimited files or log files on Windows, check them out.  There is a free (as in beer) version as well if you want to try it first.

I’ve been having some odd performance issues with some of my MySQL queries since moving over to InnoDB.  I picked up the new O’Reilly title High Performance MySQL to try to track down the problem.  The book in turn recommends a couple of pretty cool monitoring/reporting tools that summarize a lot of the MySQL variable displays in a more friendly format.

Innotop (more info here) is sort of like the friendly unix top command, but instead for database status.  There are pages to show buffer statuses, deadlocks, i/o status, current queries, and lots more.  They all update on screen, at configurable increments.

MySQLReport from hackmysql.com runs a few status commands and formats them nicely on screen in a nice grouped format.  This guide summarizes the sections, which include more detail on many of the same things that innotop covered.  I find the sections on SELECT types, and InnoDB Buffer Pool use, are especially useful to me.

Using the command type summary, we discovered an inordinate number of com_rollback calls in our main database, which we were able to reduce by using this technique.  The root cause was Hibernate’s love of transactions, combined with connection pooling.  A simple driver parameter seems to clear it up.

Boston was a key city in the War for Independence, and to celebrate that there is a big fireworks display over the Charles River every year.  This year the Erics of StyleFeeder (Savage and Kilby) were looking for a place to watch the show.  Down by the river was a mass of humanity, even compared to previous years, so we were worried about missing the show.  Then it occurred to us, doesn’t our office have a view down towards the river without many tall buildings in the way?  This proved to be the case, and we had a spectacular view of the show directly over the famous MIT Dome.

(photo by Eric Savage)

(photo by Eric Kilby)

The rest of the pictures can be found at these links via Flickr.  Remember, Creative Commons licensing is your friend, so feel free to use these as you wish.

• More July 4, 2008 photos by Eric Savage
• More July 4, 2008 photos by Eric Kilby