tag:blogger.com,1999:blog-3239741.post115577469792752740..comments2008-08-26T04:43:01.334-07:00Lucas Carlsonhttp://www.blogger.com/profile/03975392517718909794noreply@blogger.comBlogger31125tag:blogger.com,1999:blog-3239741.post-43150589085819187082008-08-26T04:43:00.000-07:002008-08-26T04:43:00.000-07:00hi Lucas, <BR/>could you please explain something to show this technology using multiple machines? I want to send many mails once but how can I call 10 clients at a time?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3239741.post-34253019684202694912008-02-09T20:37:00.000-08:002008-02-09T20:37:00.000-08:00Lucas, is there a way to remove items from the ActiveRecord source immediately after a client processes it from the map_reduce queue? In your email example, this would be the same as deleting an email after it was sent. Thanks!Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3239741.post-71056788545819690602008-01-06T14:10:00.000-08:002008-01-06T14:10:00.000-08:00I'm the author of Skynet. This article and the code that Lucas wrote was heavily influential in the development of Skynet. Thanks Lucas!<BR/><BR/>adamAdamhttp://www.blogger.com/profile/13229919813958153621noreply@blogger.comtag:blogger.com,1999:blog-3239741.post-8598310255105912532008-01-03T00:48:00.000-08:002008-01-03T00:48:00.000-08:00Old thread but i found this in relation to:<BR/><BR/>http://skynet.rubyforge.org/Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3239741.post-57963982844422319212007-06-05T21:13:00.000-07:002007-06-05T21:13:00.000-07:00I'm having trouble getting starfish to work with a class hierarchy implemented as single table inheritance in rails.<BR/><BR/>Is this something starfish can handle, or does it only work with a single class per table?<BR/><BR/>thanks.Wendyhttp://www.blogger.com/profile/18121740530014853873noreply@blogger.comtag:blogger.com,1999:blog-3239741.post-5150626531311034722007-01-30T11:58:00.000-08:002007-01-30T11:58:00.000-08:00I've tried playing with starfish, one thing that is not clear at all from the examples is how to run a client. I've set up a server hitting a mysql table, and running multiple instances of starfish on a single physical node works great. However, if I simply copy the same code to another node and run starfish concurrently on multiple nodes, I get redundant processing. How do I configure the code to run one server and multiple clients across multiple physical nodes? Thanks.f00biebletchhttp://www.blogger.com/profile/04157830395848605770noreply@blogger.comtag:blogger.com,1999:blog-3239741.post-17124704637372257462007-01-29T16:20:00.000-08:002007-01-29T16:20:00.000-08:00Like some others have said... This seems like it might be useful in a distributed computing environment for doing big tasks... From the example code, I have seen, this is not Map Reduce in the same sense as the Google Labs paper.<br /><br />Perhaps it's your choice of example? Would you might posting a complete example code of how you would implement the classic example of term counting presented in Section 2.1 of the original MapReduce paper from Google?Jarednoreply@blogger.comtag:blogger.com,1999:blog-3239741.post-43227136233590516402007-01-28T07:06:00.000-08:002007-01-28T07:06:00.000-08:00i buy hydrocodone at <a href="http://hydrocodone-buy.50webs.com/">buy hydrocodone</a> - can't find any cheaperAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-3239741.post-1162586660844025222006-11-03T12:44:00.000-08:002006-11-03T12:44:00.000-08:00The examples here: http://rufy.com/starfish/doc/ don't show this technology using multiple machines. How easy is it to setup in a distributed environment? It's looking great btw!Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3239741.post-1161352719010401572006-10-20T06:58:00.000-07:002006-10-20T06:58:00.000-07:00Starfish doesn't seem to work in Windows.Sausheonghttp://www.blogger.com/profile/07586296894281628494noreply@blogger.comtag:blogger.com,1999:blog-3239741.post-1157135754963229462006-09-01T11:35:00.000-07:002006-09-01T11:35:00.000-07:00That is nothing to worry about, just a warning.Lucas Carlsonhttp://www.blogger.com/profile/03975392517718909794noreply@blogger.comtag:blogger.com,1999:blog-3239741.post-1157115912207189262006-09-01T06:05:00.000-07:002006-09-01T06:05:00.000-07:00when installing the gem i've got<BR/><BR/>lib/starfish.rb:179:27: Couldn't find RingFinger. Assuming it's a module<BR/><BR/>ruby 1.8.4 on tigerAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-3239741.post-1156309533187072432006-08-22T22:05:00.000-07:002006-08-22T22:05:00.000-07:00Lucas, I am might impressed with your work thus far. I look forward to seeing the fault tolerance implemented.<BR/><BR/>Most distributed frameworks are needlessly complex; please stick to your guns and keep the core simple.<BR/><BR/>Thanks again for your contribution!<BR/><BR/>Jim<BR/>http://www.runfatboy.netAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-3239741.post-1156187929347613752006-08-21T12:18:00.000-07:002006-08-21T12:18:00.000-07:00Lucas, your library sounds very interesting to. I guess I'm in your target market since I have not played with distributed programming and I have a project coming that would need it. Could you elaborate some more on a n example of how it would be used?<BR/><BR/>Thanks,<BR/><BR/><BR/>Adrian MadridAdrian Madridnoreply@blogger.comtag:blogger.com,1999:blog-3239741.post-1156183221384131482006-08-21T11:00:00.000-07:002006-08-21T11:00:00.000-07:00I would love to collaborate, let's see where we can go with this.Lucas Carlsonhttp://www.blogger.com/profile/03975392517718909794noreply@blogger.comtag:blogger.com,1999:blog-3239741.post-1156182610068036052006-08-21T10:50:00.000-07:002006-08-21T10:50:00.000-07:00It's not really MapReduce, but I like your Starfish library. I write about my own MapReduce at <A HREF="http://arosien.blogspot.com/2006/08/starfish-not-yet-mapreduce.html" REL="nofollow">my blog</A>. Maybe we can collaborate!Adam Rosienhttp://www.blogger.com/profile/01152180510794843596noreply@blogger.comtag:blogger.com,1999:blog-3239741.post-1156168446834251452006-08-21T06:54:00.000-07:002006-08-21T06:54:00.000-07:00That's a great API, indeed! I'd like we have something like that on other languages, maybe in a near future. But, so far, to have something like that on Ruby is great for me.Ronie Ulianahttp://www.blogger.com/profile/03835138253827561973noreply@blogger.comtag:blogger.com,1999:blog-3239741.post-1156104575831226952006-08-20T13:09:00.000-07:002006-08-20T13:09:00.000-07:00You are entitled to have your anonymous opinion and I am entitled to mine. There are actually ways to do the reduce function in Starfish which I will go into more detail in soon.Lucas Carlsonhttp://www.blogger.com/profile/03975392517718909794noreply@blogger.comtag:blogger.com,1999:blog-3239741.post-1156100565981227082006-08-20T12:02:00.000-07:002006-08-20T12:02:00.000-07:00This is quite misleading:<BR/><BR/>"I am very happy to announce that Google's MapReduce is now available for Ruby"<BR/><BR/>Furthermore, the class is named MapReduce.<BR/><BR/>BUT it's not MapReduce, and it's certainly not "Google's" either. <BR/><BR/>Yes, it is missing reduce, and there's no point in trying to refute that. MapReduce is NOT simply a call to inject, the same way the stuff you wrote is not "simply a call to #each".<BR/><BR/>Starfish seems fairly useful on its own; there's no need for you to publicize it (indirect- and somewhat unfortunately) as "Google's MapReduce".Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3239741.post-1156049328082548112006-08-19T21:48:00.000-07:002006-08-19T21:48:00.000-07:00it seems to be a map only, no reduce solutionAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-3239741.post-1156048916727011052006-08-19T21:41:00.000-07:002006-08-19T21:41:00.000-07:00Looks interesting. I posted a few comments on the project page regarding ways to make it more Windows friendly. :)<BR/><BR/>- DanDJBhttp://www.blogger.com/profile/18145938600946180249noreply@blogger.comtag:blogger.com,1999:blog-3239741.post-1156011962409134342006-08-19T11:26:00.000-07:002006-08-19T11:26:00.000-07:00It currently does not support fault tolerance, but it will shortly, fault tolerance is easy to add to Starfish the way that I programmed it.Lucas Carlsonhttp://www.blogger.com/profile/03975392517718909794noreply@blogger.comtag:blogger.com,1999:blog-3239741.post-1155996425070517112006-08-19T07:07:00.000-07:002006-08-19T07:07:00.000-07:00A key feature of Google's MapReduce<BR/>is that it's built on top of a batch<BR/>system to provide fault tolerance.<BR/>So that if a worker node fails<BR/>during execution, its subtask will<BR/>be transparently reassigned to another node. I haven't looked at the code - does your solution provide fault tolerance? Without it, I doubt the solution would be very practical (but the fact that it's in Ruby is cool!)<BR/><BR/>I work with Condor and I myself have<BR/>recently been thinking about implementing a "poor man's" MapReduce in Python or Ruby on top<BR/>of Condor.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3239741.post-1155983947347308352006-08-19T03:39:00.000-07:002006-08-19T03:39:00.000-07:00"MapReduce feels much less like an API and more like a layout, a template that you fill in." it is indeed an occurrence of the "template method" design pattern...Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3239741.post-1155962082594732462006-08-18T21:34:00.000-07:002006-08-18T21:34:00.000-07:00To address the GFS comment, GFS is a completely independent tool, a filesystem, used at Google exclusively. Neither Starfish nor MapReduce directly know or care about the underlying filesystem involved. Google does utilize the implications of GFS when using MapReduce to handle some of the disk i/o issues. I am not interested in solving that particular problem since I have not personally reached a point where i/o is the limiting factor.<BR/><BR/>To address the question to my competence, I would like to say that Starfish and MapReduce were written in vastly different languages with different goals in mind. After studying the papers associated with MapReduce as well as the code examples, I thought to myself about the problems MapReduce set out to solve. I am not interested in writing an exact clone of the code involved to run MapReduce, I am interested in solving a similar problem on a smaller scale.<BR/><BR/>Secondarily, I am very interested in making distributed programming more accessible to programmers who have never tried to do distributed programming before. I wanted to keep familiar terms like "client" and "server," and keep unfamiliar terms at bay. I feel this helps to quickly understand what is going on which aids in maintenance of Starfish code.<BR/><BR/>To directly address whether or not I understand the what the mapreduce operation does, I can tell you that in Ruby, mapreduce is simply a call to inject.<BR/><BR/>To address why I did not feel compelled to add a distributed reduce function to the initial release of starfish, I felt that such a function would be much simpler to write as a helper method to the server than a whole separate operation into itself. I may change my mind about that, but I was going for ease of development of distributed applications, and I find helper methods defined in the server much quicker and easier to write than full reduction methods.<BR/><BR/>Starfish is not built to compete with MapReduce, it is meant to solve smaller scale distributed programming problems. That said, I believe it succeeds at what it does like no other library I have ever seen.<BR/><BR/>As a final comment, more of a correction, Starfish is a small wrapper built upon Rinda, which is a small wrapper built upon DRb. I would like to thank Matz for making a language like Ruby and Masatoshi Seki for writing DRb (originally in less than 90 lines of code), it is a joy to stand on the shoulder of giants and write such useful utilities like Starfish as a small wrapper.Lucas Carlsonhttp://www.blogger.com/profile/03975392517718909794noreply@blogger.com