In our discussion of distributing web requests to different servers via the analogy of a translation company, we ended up last week with a question. To recap, the analogy compares application serving computers to translation offices and instances of the application as translators. Further, a manager (reverse proxy) sat in front of the offices distributing tasks to each office. Last week we realized that to increase efficiency of our translators, we could put a manager in each office in order to buffer requests. That way, any individual bottleneck could not hold up the queue from being processed. For distributing web requests, I created a reverse proxy called drproxy to do exactly this.
These types of systems can be found over and over again in the real world. Just this weekend I was in line to order a polish dog from Costco. There was a single line with two servers processing the line. I watched as a father couldn't get decisions from each of his three children, yet the other server's booth moved along smoothly and brought me closer and closer to the polish dog. You can find similar lines at Nordstrom Rack, Fry's Electronics, any restaurant, and many other locations.
One system that works like the less efficient "round-robin" method described a few weeks ago are lines found in grocery stores. You get in a line praying that the people in front of you don't like to write checks or count out change because if they take their time, they hold you up. How many times did you choose what looked like the fastest checkout line only to watch people go through other lines faster due to one price-check on isle five? Some grocery stores have started implementing self-checkout systems. I find that whenever given the choice, I tend to go directly to the self-checkout because it is always faster. One of the reasons it is faster is because there is a single line for four checkout machines. You could get one person counting change, one person doing a price-check and still have two machines checking people out smoothly. It is amazing that grocery stores have not realized this and implemented better line processing.
The question I posed at the end of last week was whether there was an even more efficient way to distribute requests. I propose that there is. Here is why: just as any individual translator might get a backed up queue of requests, translation offices could become overwhelmed. Based on pure probability, one office might build up a queue of 100 requests while another might sit there queue-less. There are a few ways to tackle this problem, but whichever way you choose to handle it, you must know the size of the queues in each office at any given time. Given this knowledge, you could choose to only hand requests to the least busy of the offices. I am not a big fan of this approach because it seems like you could imagine situations where large groups of documents go to the same office in a row and I like to distribute requests randomly to prevent buildups and attacks on the system. The way that I implemented drproxy to distribute requests was by randomly picking any off! ice, except for the busiest office. If one office builds up a backup of requests and the others have no requests, no new requests will be sent to the busy office until it frees up. This load balancing system works very effectively.
I hate to do this to you again, but can you think of any other major bottlenecks in our system? I can.
You should follow me on twitter here.
Technoblog reader special: click here to get $10 off web hosting by FatCow!