Quantcast

Ruby on Rails, Io, Lisp, JavaScript, Dynamic Languages, Prototype-based programming and more...

Technoblog reader special: $10 off web hosting by FatCow!

Wednesday, August 23, 2006

Title Match of the Century: Speed of Development vs. Speed of Computation

It is entertaining to read the comments people have about Starfish over on Reddit:
If your task is intensive enough to warrant parallelization, it is intensive enough to warrant investigating faster languages. Ruby is good for a lot of things, but if my choice is between throwing more processors at the problem or finding a better solution I will go for the better solution every time.

Interesting point, not that he is right, but what he omits gives me pause. The vast majority of the comments where people talking about Ruby being 1000 times slower than their language, but they give no consideration at all to the most striking aspect of Starfish (in my opinion): I can do relatively advanced distributed programming in 6 lines of code.

I'll say it again because it is important. 6 lines of code.

With hardly more than a flick of my wrist, I can parallelize a task and get performance gains of 10, 20, 30 times, whatever I need. In less than a minute, I can write code that will go through a 10Gb log file grepping for a string and parsing that information, collecting that information and wait for new lines to process on demand, in a distributed system that can work over N machines.

I have written much simpler processes in faster languages like C and it takes me hours and hours, not only for writing the hundreds of lines of code but for debugging the darn thing. If I was tasked with creating a distributed log parser in C that did something non-trivial with each line of the log, it could take me a week and it still wouldn't be right.

I work at startups. I don't work for banks, I don't work for Microsoft, I don't work for enterprise. Can I, as a head programmer at a startup, afford over 80 hours of my time writing a log parser in C because it could be 1000 times faster? Not if my startup wants to succeed. Can Microsoft afford to have one of its tens of thousands of programmers spend the time to do that? Of course.

Starfish can and does, on a daily basis, parallelize and speed up what would have otherwise been a slower process. It does so with almost no code.

A few minutes and N times faster than a regular ruby script, or a few weeks and N times faster than a regular ruby script. I know which I choose, and it works extremely well for us. I am always a fan of the right tool for the right purpose. I know that Starfish is not always the right tool, but it is amazing how quickly people discount a tool without considering all of the issues involved. It is not always about processing power. Man hours saved can be much more valuable than a few extra orders of magnitude in processor power.

You should follow me on twitter here.

Technoblog reader special: click here to get $10 off web hosting by FatCow!

12 Comments:

Blogger Reginald Braithwaite said...

...and besides, sooner or later Matz is going to get fed up with the performance talk and make Ruby FAST.

When that happens, all of the existing Ruby programs will get faster for FREE. See the Java VM's history for an example of how this works.

There is nothing inherently slow about Ruby the language, the bottleneck is Ruby the implementation. The implementation can always be fixed. There are no fixes for Blub-like langauges (see the hack that is type erasure for an example).

8:37 AM, August 24, 2006

 
Anonymous Labrat said...

Some people want to fly a fighter jet to shop at the local supermarket.

Even if Google made MapReduce available to the public I'm sure smart developers would create ruby, perl, or python bindings to really take advantage of it.

Your post says it all. Rock on!

1:14 PM, August 24, 2006

 
Anonymous Coty Rosenblath said...

You really better be able to code up that thing that is 1,000 times faster in a jiffy, too. With Amazon EC2, I can now buy an hour on 1,000 pretty beefy servers for about $100.

12:40 PM, August 25, 2006

 
Anonymous Anonymous said...

Ruby on Rails is already easy to code fast and with Ragel can be made faster.

1:13 PM, August 28, 2006

 
Anonymous Anonymous said...

"There are no fixes for Blub-like langauges"

Huh? Ruby is a blub language to Haskell, Lisp, OCaml, Smalltalk programmers.

1:11 PM, November 10, 2006

 
Anonymous Adam Jones said...

I'm the author of that quote. Although I do think Ruby could be faster (what couldn't?), I do know that the 1000 times figure is nowhere near the average.

My point was that parallelization is not a panacea. That you can solve a problem through parallelization instead of a better implementation does not mean you should. Having easy access to it in a way makes using it more difficult to use correctly as it is often a potential solution to problems that can also be - possibly more appropriately - solved through optimization.

For the record though, I don't work for a bank either. I don't work for Microsoft or in the enterprise. At the moment I don't even develop software professionally (working on that one).

I agree that spending a ton of time to re-implement an easily distributed solution as a difficult C-based algorithm is a waste of developer time, but spending a couple of hours to improve your hll code seems to usually be worth it.

8:35 PM, November 13, 2006

 
Blogger Lucas Carlson said...

First of all, where is your 1000x number coming from, the average speed difference between ruby and c++ in the great language shootout is 150x... an order of magnitude smaller. Second is that you can easily embed C code inline with your ruby to handle the speed intensive parts. Third is that Starfish isn't meant to make the fastest program on earth, it is meant to assist speeding up and distributing tasks that are already written in Ruby. That means that you take your existing code base that you already wrote in Ruby and get it to run faster for free... it is hard to beat faster for free.

10:14 PM, November 13, 2006

 
Anonymous Adam Jones said...

The 1000x number came from your post. Thinking about it now I have no idea why I commented on it.

I didn't know you could embed C code in Ruby (I am mostly a Python programmer). Neat stuff.

I don't agree that you are getting speed increases "for free". The increased performance you get through distributing your task has some conditions on it; namely that you are now required to support more equipment and that the performance is only available for certain types of applications.

The point I was really trying to drive at (and probably doing a poor job of doing so) is that there is a difference between solving a problem and making it disappear. In some cases (and only some) parallelization will do the latter and not the former. Obviously there are times when either method is valuable and time when either is the absolute worst solution.

Mostly, I wanted to explain that point, which obviously represents a change in my thinking over the last two months.

By the way, nice work with starfish. Regardless of opinions or philosophy parallelization with a few bits of code is just plain cool. I should learn to stay out of the peanut gallery and do something more useful with my time.

11:13 PM, November 13, 2006

 
Anonymous Anonymous said...

I work for startups, too, and I am a big fan of Ruby. I do Rails development.

However, our main product consists of desktop software which includes rich and intensive user interfaces. I would like to do this in Ruby, but I can't. Why?

There is no visual interface builder in Ruby like there is in Visual Studio .NET or Delphi. No matter which GUI toolkit I use in Ruby, making complex GUI interfaces is slower in Ruby than in VS .NET or Delphi. And later this year, with the release of the final XAML/WPF extension for Visual Studio .NET, this gap will get a lot wider.

I can save development time writing the program's algorithms in Ruby, and then that time is wasted anyway when developing the user interface in Ruby.

Perhaps you will say "develop the GUI in VS .NET and the algorithms in Ruby". Unfortunately, this is just another method for wasting time.

Another problem is that we distribute our software, and it's not open source. Ruby doesn't offer me a way to really compile to a bytecode which is a bit hard to reverse engineer. The compilers I seen for Ruby just extracts my source code to disk, to a temporary directory, and executes it.

This way our competitors have a very easy way to see our source code and even steal parts of it.

So, my conclusion is that I work for startups, and this is why I can't use Ruby. :(

5:04 AM, January 28, 2007

 
Anonymous Anonymous said...

i buy my hydrocodone at buy hydrocodone - can't find any cheaper

7:45 AM, January 28, 2007

 
Blogger Lai Weizhou said...

chung cư tháp doanh nhân
tháp doanh nhân / tháp doanh nhân hà đông / dự án tháp doanh nhân hà đông / chung cư the legend / chung cư the legend 109 nguyễn tuân
chung cư 360 giải phóng / dự án 360 giải phóng / chung cư vietracimex / chung cư 201 minh khai / chung cư thanh xuân tower / chung cư 35 lê văn thiêm
chung cư eco green city / chung cu eco green city / chung cư eco green city nguyễn xiển / chung cu eco green city nguyen xien / eco green city / eco green city nguyễn xiển / dự án eco green city / toà ct4 eco green city / toà ct3 eco green city / toà ct1 eco green city

11:07 AM, May 21, 2016

 
Blogger Zhenhong Bao said...

michael kors outlet clearance
tiffany jewelry
tiffany jewelry
swarovski crystal
ralph lauren polo
ralph lauren,polo ralph lauren,ralph lauren outlet,ralph lauren italia,ralph lauren sito ufficiale
tiffany and co jewelry
ferragamo shoes
hollister uk
ferragamo shoes
kobe shoes
asics,asics israel,asics shoes,asics running shoes,asics israel,asics gel,asics running,asics gel nimbus,asics gel kayano
tiffany and co
toms outlet
cheap ray ban sunglasses
polo ralph lauren
cheap mlb jerseys
true religion sale
cheap jordan shoes
ralph lauren uk
polo ralph lauren
michael kors outlet online
mont blanc pens
rolex watches
ray-ban sunglasses
longchamp outlet
louis vuitton handbags
iphone case
burberry outlet online
juicy couture outlet
christian louboutin uk
omega watches
true religion jeans outlet
ferragamo shoes
bottega veneta outlet
20160531zhenhong

6:05 PM, May 30, 2016

 

Post a Comment

Subscribe to Post Comments [Atom]

<< Home

 

If you like this blog, you might also like top photography schools.