Quantcast

Ruby on Rails, Io, Lisp, JavaScript, Dynamic Languages, Prototype-based programming and more...

Technoblog reader special: $10 off web hosting by FatCow!

Monday, August 28, 2006

How I processed a log file 20x SLOWER than before

Sometimes you have to sit back and laugh at yourself. Having recently written Starfish to help speed up slow tasks, I tried to find as many uses as I could for it. At MOG we have to parse huge log files so I thought I would be clever and try to use Starfish for the task. After running it for a while, I looked at the stats only to find that it had been processing my file 20x slower than it would have without distributing it. At first I was puzzled, until I realized a very important thing about distributing processes. You have to make sure that the task you distribute takes longer than the distribution process.
if overhead_time > processing_time then puts "Don't use Starfish" end

It turns out that I could process 10,000 lines of the log in about a second... so to send each one of those lines over the network to have them processed was just silly. Even sending 10,000 lines at a time is relatively unnecessary.

I share this story so that you might not make the same mistake that I did. However I realized that Starfish can know when the overhead makes it not worth the trouble, I can actually warn people using Starfish when it is and is not a good use of resources. I will be adding this to the next release which shall come out shortly.

You should follow me on twitter here.

Technoblog reader special: click here to get $10 off web hosting by FatCow!

11 Comments:

Anonymous Anonymous said...

We frequently sit back and laugh at you, you may as well enjoy it too!

6:22 PM, August 28, 2006

 
Anonymous Jenna said...

Hehe, mistakes are fun to get cool idea's from.

5:55 PM, August 29, 2006

 
Anonymous KirinDave said...

Starfish makes distributed programming more accessible, but there is a moral here.

Some things create inherent complexity, and just because code can hide that doesn't mean reality follows suit. For example, people like to assume that Quicksort is the "fastest sort algorithm", and they apply it like a club. But it's really much slower for lists that tend to be mostly in order.

12:08 PM, August 30, 2006

 
Blogger Tim said...

It great you have the guts to laugh at yourself, share it and move on. We are all learning here. Keep contributing!

anonymous: don't be an ass

1:23 PM, July 12, 2007

 
Blogger Lai Weizhou said...

chung cư tháp doanh nhân
tháp doanh nhân / tháp doanh nhân hà đông / dự án tháp doanh nhân hà đông / chung cư the legend / chung cư the legend 109 nguyễn tuân
chung cư 360 giải phóng / dự án 360 giải phóng / chung cư vietracimex / chung cư 201 minh khai / chung cư thanh xuân tower / chung cư 35 lê văn thiêm
chung cư eco green city / chung cu eco green city / chung cư eco green city nguyễn xiển / chung cu eco green city nguyen xien / eco green city / eco green city nguyễn xiển / dự án eco green city / toà ct4 eco green city / toà ct3 eco green city / toà ct1 eco green city

11:07 AM, May 21, 2016

 
Blogger Zhenhong Bao said...

longchamp handbags
coach outlet
true religion canada
oakley sunglasses
mulberry handbags
reebok shoes
burberry outlet
ralph lauren outlet
ralph lauren polo
ralph lauren
hermes belt for sale
versace sunglasses on sale
fitflops clearance
nike air max 90
thomas sabo outlet
pandora outlet
tory burch outlet
coach outlet
kobe shoes
oakley sunglasses
ghd hair straighteners
ray ban sunglasses
coach outlet online
tory burch shoes
coach outlet
ralph lauren outlet
mulberry handbags
michael kors outlet online
discount oakley sunglasses
burberry sunglasses on sale
ralph lauren pas cher
replica watches
oakley sunglasses
chrome hearts outlet online
longchamp pliage
20160531zhenhong

6:20 PM, May 30, 2016

 
Blogger Zhenhong Bao said...

toms outlet
tiffany jewellery
beats headphones
oakley sunglasses wholesale
ray-ban sunglasses
oakley sunglasses wholesale
michael kors uk
rolex watches
gucci sunglasses uk
michael kors outlet store
michael kors outlet
fitflops sale clearance
prada sunglasses for women
michael kors outlet
dior sunglasses
rolex uk
ralph lauren outlet
michael kors outlet
lululemon outlet
longchamp pas cher
camisetas futbol baratas
ferragamo outlet
air force 1 shoes
louis vuitton sunglasses for women
police sunglasses
futbol baratas
reebok shoes
louis vuitton handbags outlet
cartier watches
cheap nba jerseys
prada handbags
versace sunglasses
polo ralph lauren
jordan pas cher
ray-ban sunglasses
20160531zhenhong

6:25 PM, May 30, 2016

 
Blogger Zhenhong Bao said...

toms outlet
tiffany jewellery
beats headphones
oakley sunglasses wholesale
ray-ban sunglasses
oakley sunglasses wholesale
michael kors uk
rolex watches
gucci sunglasses uk
michael kors outlet store
michael kors outlet
fitflops sale clearance
prada sunglasses for women
michael kors outlet
dior sunglasses
rolex uk
ralph lauren outlet
michael kors outlet
lululemon outlet
longchamp pas cher
camisetas futbol baratas
ferragamo outlet
air force 1 shoes
louis vuitton sunglasses for women
police sunglasses
futbol baratas
reebok shoes
louis vuitton handbags outlet
cartier watches
cheap nba jerseys
prada handbags
versace sunglasses
polo ralph lauren
jordan pas cher
ray-ban sunglasses
20160531zhenhong

6:26 PM, May 30, 2016

 
Anonymous شركة تخزين اثاث بالرياض said...

sank you

8:56 AM, July 28, 2016

 
Blogger chenlina said...

chenlina20161018
ralph lauren polo
michael kors handbags
gucci uk
michael kors outlet
polo ralph lauren
uggs sale
coach outlet
moncler uk
coach outlet online
ray ban wayfarer

7:48 PM, October 17, 2016

 
Blogger Nancy Eyre said...

It is quite inspirational that you have the courage to laugh at yourself. But keep sharing posts like this since we are learning from here. Just don't be underestimated.buy dissertation online is the best help that any student can get for completing the dissertation on time

11:18 PM, December 01, 2016

 

Post a Comment

Subscribe to Post Comments [Atom]

<< Home

 

If you like this blog, you might also like top photography schools.