Friday, November 7, 2008

RubyConf 2008 Day 2 - Patterns in Distributed Processing

Distributed computing and threading have been my favorite topics to learn about over the past year. In general this stuff is just way over my head and I feel like I learn something new every time I attend a talk on these subjects. This conference is filled with all sorts of interesting presentations including one where Mike Perham is discussing a library he created for clustered systems. He gave us a little background on himself and I was impressed to find out that he is responsible for shard support in ActiveRecord.
He seems like a brilliant fellow and is demonstrating some advanced threading concepts, only using distributed machines using DRb. This is an entirely different approach from a NeverBlock, which used Fibers, i.e. continuations, to allow green threads to "work" faster. I believe that based on the code I have seen today, Mike's approach is probably easier to understand and use.
He is also describing "Paxos" and encouraged us to take a look at Google's white paper "Paxos Made Live". Paxos is slow because there is lots of network chatter and Mike decided to head in a less formal direction using a designated "leader" and follower workers. Quote from his blog:
  • TokenWorker ensures that one process from a group of redundant processes will be elected to perform processing for the next N seconds.
  • StaticQueueWorker distributes a predefined set of work to a dynamic set of processes every N seconds.
When Brian Geotz discussed multi-threading in Java he seemed really excited about it. This talk would have been better if Mike were a little more energized.

No comments: