Quantcast

Ruby on Rails, Io, Lisp, JavaScript, Dynamic Languages, Prototype-based programming and more...

Technoblog reader special: $10 off web hosting by FatCow!

Monday, August 28, 2006

How I processed a log file 20x SLOWER than before

Sometimes you have to sit back and laugh at yourself. Having recently written Starfish to help speed up slow tasks, I tried to find as many uses as I could for it. At MOG we have to parse huge log files so I thought I would be clever and try to use Starfish for the task. After running it for a while, I looked at the stats only to find that it had been processing my file 20x slower than it would have without distributing it. At first I was puzzled, until I realized a very important thing about distributing processes. You have to make sure that the task you distribute takes longer than the distribution process.
if overhead_time > processing_time then puts "Don't use Starfish" end

It turns out that I could process 10,000 lines of the log in about a second... so to send each one of those lines over the network to have them processed was just silly. Even sending 10,000 lines at a time is relatively unnecessary.

I share this story so that you might not make the same mistake that I did. However I realized that Starfish can know when the overhead makes it not worth the trouble, I can actually warn people using Starfish when it is and is not a good use of resources. I will be adding this to the next release which shall come out shortly.

You should follow me on twitter here.

Technoblog reader special: click here to get $10 off web hosting by FatCow!

42 Comments:

Anonymous Anonymous said...

We frequently sit back and laugh at you, you may as well enjoy it too!

6:22 PM, August 28, 2006

 
Anonymous Jenna said...

Hehe, mistakes are fun to get cool idea's from.

5:55 PM, August 29, 2006

 
Anonymous KirinDave said...

Starfish makes distributed programming more accessible, but there is a moral here.

Some things create inherent complexity, and just because code can hide that doesn't mean reality follows suit. For example, people like to assume that Quicksort is the "fastest sort algorithm", and they apply it like a club. But it's really much slower for lists that tend to be mostly in order.

12:08 PM, August 30, 2006

 
Blogger Tim said...

It great you have the guts to laugh at yourself, share it and move on. We are all learning here. Keep contributing!

anonymous: don't be an ass

1:23 PM, July 12, 2007

 
Blogger Lai Weizhou said...

chung cư tháp doanh nhân
tháp doanh nhân / tháp doanh nhân hà đông / dự án tháp doanh nhân hà đông / chung cư the legend / chung cư the legend 109 nguyễn tuân
chung cư 360 giải phóng / dự án 360 giải phóng / chung cư vietracimex / chung cư 201 minh khai / chung cư thanh xuân tower / chung cư 35 lê văn thiêm
chung cư eco green city / chung cu eco green city / chung cư eco green city nguyễn xiển / chung cu eco green city nguyen xien / eco green city / eco green city nguyễn xiển / dự án eco green city / toà ct4 eco green city / toà ct3 eco green city / toà ct1 eco green city

11:07 AM, May 21, 2016

 
Anonymous Anonymous said...

longchamp handbags
coach outlet
true religion canada
oakley sunglasses
mulberry handbags
reebok shoes
burberry outlet
ralph lauren outlet
ralph lauren polo
ralph lauren
hermes belt for sale
versace sunglasses on sale
fitflops clearance
nike air max 90
thomas sabo outlet
pandora outlet
tory burch outlet
coach outlet
kobe shoes
oakley sunglasses
ghd hair straighteners
ray ban sunglasses
coach outlet online
tory burch shoes
coach outlet
ralph lauren outlet
mulberry handbags
michael kors outlet online
discount oakley sunglasses
burberry sunglasses on sale
ralph lauren pas cher
replica watches
oakley sunglasses
chrome hearts outlet online
longchamp pliage
20160531zhenhong

6:20 PM, May 30, 2016

 
Anonymous Anonymous said...

toms outlet
tiffany jewellery
beats headphones
oakley sunglasses wholesale
ray-ban sunglasses
oakley sunglasses wholesale
michael kors uk
rolex watches
gucci sunglasses uk
michael kors outlet store
michael kors outlet
fitflops sale clearance
prada sunglasses for women
michael kors outlet
dior sunglasses
rolex uk
ralph lauren outlet
michael kors outlet
lululemon outlet
longchamp pas cher
camisetas futbol baratas
ferragamo outlet
air force 1 shoes
louis vuitton sunglasses for women
police sunglasses
futbol baratas
reebok shoes
louis vuitton handbags outlet
cartier watches
cheap nba jerseys
prada handbags
versace sunglasses
polo ralph lauren
jordan pas cher
ray-ban sunglasses
20160531zhenhong

6:25 PM, May 30, 2016

 
Anonymous Anonymous said...

toms outlet
tiffany jewellery
beats headphones
oakley sunglasses wholesale
ray-ban sunglasses
oakley sunglasses wholesale
michael kors uk
rolex watches
gucci sunglasses uk
michael kors outlet store
michael kors outlet
fitflops sale clearance
prada sunglasses for women
michael kors outlet
dior sunglasses
rolex uk
ralph lauren outlet
michael kors outlet
lululemon outlet
longchamp pas cher
camisetas futbol baratas
ferragamo outlet
air force 1 shoes
louis vuitton sunglasses for women
police sunglasses
futbol baratas
reebok shoes
louis vuitton handbags outlet
cartier watches
cheap nba jerseys
prada handbags
versace sunglasses
polo ralph lauren
jordan pas cher
ray-ban sunglasses
20160531zhenhong

6:26 PM, May 30, 2016

 
Anonymous شركة تخزين اثاث بالرياض said...

sank you

8:56 AM, July 28, 2016

 
Blogger chenlina said...

chenlina20161018
ralph lauren polo
michael kors handbags
gucci uk
michael kors outlet
polo ralph lauren
uggs sale
coach outlet
moncler uk
coach outlet online
ray ban wayfarer

7:48 PM, October 17, 2016

 
Blogger Nancy Eyre said...

It is quite inspirational that you have the courage to laugh at yourself. But keep sharing posts like this since we are learning from here. Just don't be underestimated.buy dissertation online is the best help that any student can get for completing the dissertation on time

11:18 PM, December 01, 2016

 
Blogger chenmeinv0 said...

louboutin femme
ed hardy outlet
cheap ugg boots
the north face
hilfiger jeans
retro jordan shoes
cheap ray ban sunglasses
gucci outlet online
coach factory outlet
air jordan shoes
2017.1.4xukaimin

10:28 PM, January 03, 2017

 
Blogger hijabkuh said...

great post baby room ideas
teenage bedroom
pink girl bedroom
bedroom wallpaper

10:56 PM, January 12, 2017

 
Blogger Unknown said...

pittsburgh steelers jersey
new orleans saints jerseys
rolex replica
jets jersey
michael kors handbags
canada goose
carolina jerseys
air jordan uk
nike huarache
ecco shoes

12:32 AM, March 16, 2017

 
Blogger shengda xu said...

longchamp sale
yeezy shoes
tods outlet
coach factory outlet online
true religion jeans outlet
michael kors
ralph lauren
polo ralph lauren outlet
kate spade purses
louis vuitton
xushengda0418

6:44 AM, April 18, 2017

 
Blogger shengda xu said...

pandora soldes
yeezy boost
yeezy shoes
mulberry bags
clarks shoes outlet
coach factory outlet online
toms outlet
cheap rolex replica watches
nike sneakers
coach factory outlet online
xushengda0714

2:45 AM, July 14, 2017

 
Blogger shengda xu said...

michael kors outlet store
jordan shoes
pandora charms outlet
yeezy 350 boost
mont blanc pen
adidas running shoes
michael kors outlet
red bottom
michael kors outlet clearance
coach outlet store online
xushengda0714

2:46 AM, July 14, 2017

 
Blogger Quang Đào Duy said...

Tọa lạc tại vị trí đắc địa Chung cư 378 Minh Khai được mệnh danh là " Viên Ngọc" giữa lòng thành phố với vị trí không thể thuận tiện hơn tại nút giao thông huyết mạch nối liền các trung tâm Chính trị, Văn hóa,Trường học, Bệnh viện, Sân gold của thủ đô Hà Nội

Dự án Green Pearl tọa lạc tại số 378 Minh Khai , Hai Bà Trưng, Hà Nội là một vị trí đắc địa thuộc cửa ngõ phía Nam của Thủ đô, nơi tập trung các tuyến giao thông huyết mạch của TP Hà Nội (Đường Minh Khai - Cầu Vĩnh Tuy). Dự án bao gồm nhà ở liền kề, biệt thự và căn hộ cao cấp Green Pearl mở ra không gian sống xanh, trong lành, KHÔNG ồn ào, KHÔNG khói bụi ngay trong lòng đô thị.

Dự án 378 Minh Khai hiểu rõ vui chơi là hình thức tốt nhất giúp trẻ phát triển tư duy. Từ đó chúng tôi xây dựng trường học hiện đại, khu vui chơi trẻ em với những trò vui chơi giải trí, trí tuệ giúp trẻ phát triển cả tư duy sáng tạo và thể chất.

Chung cư Green Pearl
Chung cư Green Pearl Không chỉ là giải pháp đáp ứng nhu cầu nhà ở chất lượng cao, sứ mệnh của Green Pearl là cùng bạn trải nghiệm cuộc sống" bình yên trọn vẹn "tách biệt với cuộc sống ngột ngạt ồn ào bên ngoài. Không gian yên tĩnh trong lành được khai thác tối đa trên những con đường tản bộ xanh mát sẽ giúp bạn cảm thấy thư thái nhằm tái tạo năng lượng hiệu quả nhất.
Mặt bằng chung cư Green Pearl
Tiện ích chung cư Green Pearl
Giá bán chung cư Green Pearl
Chung cư Green Pearl

8:18 PM, January 12, 2018

 
Anonymous Pengobatan Radang Paru-Paru Basah (Pneumonia) said...

Thanks for sharing the information

Pengobatan Fistula Ani
Obat Kista Rahim
Cara Mengangkat Kista Payudara
Obat Atasi Ginjal Bengkak
Pengobatan Untuk Sembuhkan Radang Amandel
Cara Mengobati Vertigo Sampai Tuntas

12:02 AM, March 12, 2018

 
Blogger Rino Febri said...

thank you information, very helpful, do not forget to visit also kunju our page to berai information about health
obat kesrek di selangkangan
Cara Mengobati Tukak Lambung
cara menghilangkan budug
Obat Luka Lebam

9:22 PM, May 22, 2018

 
Anonymous cara menyembuhkan pembengkakan limpa said...

It is imperative that we read blog post very carefully. I am already done it and find that this post is really amazing.
cara menghilangkan darah kotor
pengobatan lipoma

9:44 PM, July 10, 2018

 
Blogger Stjsrty Xtjsrty said...

zzzzz2018.7.16
coach factory outlet
polo ralph lauren
adidas shoes
michael kors outlet online
pandora charms outlet
oakley sunglasses wholesale
yeezy
nfl jerseys
canada goose jackets
nike outlet

7:10 PM, July 15, 2018

 
Anonymous Cara menyembuhkan penebalan dinding rahim said...

thank you very useful information admin, and pardon me permission to share articles here may help :

Cara mengatasi katarak
Suplemen untuk mengencangkan kulit wajah
Obat ruam popok ampuh
Obat ablasio retina tanpa operasi
Walatra berry jus
Cara menyembuhkan mastitis
Obat angin duduk herbal

7:28 PM, July 16, 2018

 
Anonymous Obat Luka Lambung said...

It is imperative that we read blog post very carefully. I am already done it and find that this post is really amazing.
obat ginjal bengkak
obat keloid
obat benjolan di payudara
obat infeksi saluran kencing
obat patah tulang

12:37 AM, July 21, 2018

 
Anonymous Cara Mengatasi Mata Bengkak said...

This article is interesting and useful. Thank you for sharing. And let me share an article about health that God willing will be very useful. Thank you :)

Cara Mengobati Panu
Obat Penghenti Batuk Berdarah
Biaya dan Efek Samping Operasi Kista Bartholin
Penyebab dan cara Mengatasi jantung Berdebar
Cara Mengobati Pengapuran Tulang
Obat Pengapuran Otak pada Anak

6:48 PM, August 04, 2018

 
Blogger Ahmad Jamal said...

i read a lot of stuff and i found that the way of writing to clearifing that exactly want to say was very good so i am impressed and ilike to come again in future.
cara menyembuhkan iritasi mata
obat luka gangren
obat benjolan di tangan
obat tumor
obat ginjal bocor

9:10 PM, August 24, 2018

 
Blogger 5689 said...

zzzzz2018.9.2
christian louboutin sale
off white shoes
pandora outlet
ugg boots clearance
canada goose outlet
canada goose jackets
moncler outlet
christian louboutin outlet
soccer boots
christian louboutin shoes

6:22 PM, September 01, 2018

 
Blogger Unknown said...

https://www.codecademy.com/sattaking12

https://issuu.com/sattaking1226

https://speakerdeck.com/sattaking1226

https://500px.com/sattaking1226

http://www.folkd.com/user/sattaking12

https://www.keepcalm-o-matic.co.uk/user/sattaking/

https://disqus.com/by/disqus_Sn4XHu5pRl/

https://www.edocr.com/user/sattaking1226

http://www.pbase.com/profile/sattaking12

https://www.shapeways.com/designer/sattaking

4:38 AM, September 04, 2018

 
Blogger 洛噷 said...

2018906 leilei3915
pandora charms
canada goose outlet
ralph lauren outlet
ugg outlet online clearance
true religion jeans sale
canada goose jackets
cheap ugg boots
michael kors outlet online
moncler jackets
canada goose jackets

12:00 AM, September 06, 2018

 
Blogger 洛噷 said...

2018906 leilei3915
ugg boots outlet
louboutin shoes
coach outlet
polo ralph lauren
michael kors outlet online
pandora jewelry
coach factory outlet
ray ban sunglasses
fitflops sale
uggs outlet

12:01 AM, September 06, 2018

 
Blogger Reka Nurhanisa said...

the website you manage is very good and in my opinion this is the best wesbite
obat leukosit tinggi

11:07 PM, September 06, 2018

 
Blogger Quang Đào Duy said...

Vincity Gia Lâm
Vincity Ocean Park
Vincity Ocean Park Gia Lâm
Tiện ích Vincty ocean Park
Mặt bằng vincity Ocean Park
Giá bán Vincity Ocean Park
Thiết kế vincity Ocean Park Gia Lâm
Vinhomes Ocean Park
Liền kề Oean Park
Biệt thự Vinhomes Ocean Park
Shophouse Vinhomes Ocean Park
Biệt thự Ocean Park
Vincity Đại Mỗ
Vincity Tây Mỗ

2:08 AM, September 13, 2018

 
Blogger risa icha said...

I would like to thanks for sharing high value and informative article with us. And i hope you'll sharing more idea's and keep writing more like this one.

obat biduran
obat asam lambung
obat herpes
obat liver
obat keputihan gatal
obat ginjal rusak

6:20 PM, October 15, 2018

 
Anonymous Anonymous said...

http://talk-staging.plesk.com/members/linabelaidi.183969/
http://talk-staging.plesk.com/members/ericapellossi.183917/
http://talk-staging.plesk.com/members/chris-piettre.183966/
https://www.themehorse.com/support-forum/users/piettrechris

9:43 PM, October 26, 2018

 
Anonymous Anonymous said...

http://profile.hatena.ne.jp/linabelaidi/

10:27 PM, October 26, 2018

 
Anonymous Anonymous said...

http://djerbabelle.strikingly.com/

1:07 AM, October 27, 2018

 
Anonymous Anonymous said...

http://beaute-faite-femme.e-monsite.com/pages/chirurgie-esthetique-tunisie.html

12:00 AM, November 02, 2018

 
Anonymous Anonymous said...

http://location-voiture-tunisie.angelfire.com/

10:13 AM, November 03, 2018

 
Anonymous Anonymous said...

https://location-voiture.hatenablog.com/entry/2018/11/04/175615

1:17 AM, November 07, 2018

 
Blogger Lutfi Kurniawan said...

Great thank to upload, hopefully it will be useful for writers and those who read :) Obat Kelenjar Getah Bening Bahaya Benjolan Dileher Dan Cara Mengatasinya Cara Mengobati Luka Bernanah Obat Kista Paling Ampuh Cara Menyembuhkan Penyakit Maag Cara Mengatasi Sakit Pinggang

12:38 AM, December 08, 2018

 
Blogger نور الهدى said...


مكافحة حشرات بالخبر مكافحة حشرات بالخبر
مكافحة حشرات بمكة مكافحة حشرات بمكة
مكافحة حشرات بالمدينة المنورة مكافحة حشرات بالمدينة المنورة
مكافحة حشرات بالدمام مكافحة حشرات بالدمام
مكافحة حشرات بالاحساء مكافحة حشرات بالاحساء

7:56 AM, December 08, 2018

 
Blogger Zihan Amelia said...

Ijin share ya, semoga bermanfaat Cara Mengobati Esofagitis Cara Menyembuhkan Bintitan Obat Radang Usus Kronis Cara Mengobati Fistula Ani

10:16 PM, December 09, 2018

 

Post a Comment

Subscribe to Post Comments [Atom]

<< Home

 

If you like this blog, you might also like top photography schools.