Getting to “bare metal” (September 2, 2010)

Have recently finished reading Peter Seibel’s “Coders at Work”. If you haven’t heard about this book yet, it’s a very well-written collection of interviews with some of the most influential people in our industry; the names of those author talks to vary from Crockford and Bloch to Norvig and Knuth. Apart from being a very inspiring read, the book gives a very interesting perspective on some questions Peter keeps asking over and over. “Do you consider yourself a scientist, an artist, an engineer, or a craftsman?” “Should all progammers be able to deal with low-level things even despite that this ability is rarely needed nowadays?” “Do you still write code?” The question on low-level programming is among the most interesting ones. Are there any differences in one’s performance with and without that knowledge?

I guess there are. Without having a big desire to fall back to the original topic of my blog, I want to point to a recent issue we had with our biggest SLEE-based project. This is the first project where we are using big branchy trees of SBBs in our application and the first one that showed such a bad results during the initial performance tests. What is happening? Why is it so slow? After several nights and weekends spent staring at the CPU profiler and thread dumps and a help from our vendor, the root cause of the problem became clear.

When the event flow is being delivered to the SBBs in the same tree, event router thread does not have any choice except for locking the whole tree. After all, there’s a SLEE transaction ongoing and the state of the SBB tree should be kept consistent. Other event router threads at that moment are waiting for the tree to be freed, even if they try to deliver the event to another SBB in the tree. More blocking time, higher latency, lower performance.

Interesting is that JAIN SLEE just like any similar event-driven framework tries to isolate the developer from threading almost completely. But it’s clear that a developer can’t even diagnose a problem like ours without resorting to knowledge about thread states. So personally I have no doubts about the necessity of this knowledge. And I think our playful age of Mongrel and Cucumber gives more opportunities to learn low-level things with less pain. Ruby folks seem to discover fibers lately (green threads whose execution is controlled programmatically) so there will be a lucky few who will master the art of its scheduling. I myself had a chance to play with the Java emulator of x86 PC — seriously, having the possibility to compile your C program, take the binary, and see how Processor.processProtectedModeInterrupts() is executed in your Java emulator is astonishing.

I’m excited to read about other people thinking the same.