Quantcast

Ruby on Rails, Io, Lisp, JavaScript, Dynamic Languages, Prototype-based programming and more...

Technoblog reader special: $10 off web hosting by FatCow!

Tuesday, August 22, 2006

How I sent emails 10x faster than before

Like many startups, at MOG we send out regular updates to our users with news and information. As our user base expands, sending this email takes more and more time. Even though the call to deliver the mail only puts it in the sendmail queue, it can take a chunk of time to do so with so many users.

When I demoed Starfish to people, the common response was: that's great, I wish I had a use for it, I wish I had a DB source big enough to use. Well here is one, albeit not mission critical but still damn cool, of the ways we use Starfish.

require 'config/environment'
require 'user'
require 'notifier'

server do |map_reduce|
map_reduce.type = User
map_reduce.conditions = "opt_out = 0"
end

client do |user|
Notifier.deliver_email(user)
end

This tiny amount of code with next to nothing that needs to be memorized and takes 30 seconds to write down can potentially save you hours in deliver time. Even running 10 clients at once on the SAME MACHINE gave us nearly 10x the speed it would have taken serially. This was not mission critical, but gives you a good sense of ways to apply Starfish to mission critical applications.

You should follow me on twitter here.

Technoblog reader special: click here to get $10 off web hosting by FatCow!

21 Comments:

Anonymous Jason Watkins said...

that's so simple and clean... way simpler than background drb, even simpler than using ruby threads (which might not provide the parallelism you need anyhow).

3:11 AM, August 22, 2006

 
Blogger Lucas Carlson said...

Plus, Ruby threads + Rails stack = Crap shoot

10:07 AM, August 22, 2006

 
Anonymous Adrian Madrid said...

Thanks for the example. It seems cleared to me now how you can use Starfish.

9:37 AM, August 23, 2006

 
Anonymous James said...

Would you mind detailing what exactly the code above is doing.

Thanks.

10:38 AM, August 23, 2006

 
Blogger Lucas Carlson said...

The code above creates a queue of User objects. The clients grab a user from the queue and sends an email to the user, then grab another user and send an email, over and over. If you have 10 clients, it is like splitting up the collection over 10 loops and running them concurrently, hence the 10x speedup.

11:42 AM, August 23, 2006

 
Anonymous Anonymous said...

How does your example code know to use starfish?

12:05 PM, August 23, 2006

 
Blogger Lucas Carlson said...

You save that file, for example as: email_sender.rb. Then you execute the code by calling: starfish email_sender.rb. Since I wanted 10 clients processing the data, I called starfish email_sender.rb 10 times.

12:11 PM, August 23, 2006

 
Anonymous James said...

How do you prevent multiple processes from sending e-mails to the same person?

Meaning, if you had a text file contain unique e-mail addresses to mail a message to, how does the script above make sure that only 1 e-mail goes to each unique e-mail address?

1:01 PM, August 23, 2006

 
Anonymous Hank said...

Since you are running it on the same machine, how is this different from creating a multi-thread application?

Isn't this essentially all this is?

1:11 PM, August 23, 2006

 
Blogger Lucas Carlson said...

James: It uses a queue system, so two clients can never grab the same line from a file because once grabbed, that line is not accessible to any other clients. It also has a simple mutex to prevent two calls at the exact same time.

Hank: Have you ever tried multi-threading Rails? It is almost always a big source of headaches do to things since there would be conflicts with multiple attempts to concurrently use the connection to the DB. Plus, obviously I didn't have to call them all on the same server which is a big bonus.

1:23 PM, August 23, 2006

 
Anonymous Mike said...

Hey I just wrote the worlds shortest operating system:

#include "theuniverse.h"

int main()
{
run_os()
}

I'm kidding, I'm kidding, but you get my point? What is user doing, what is notifier?

5:54 AM, August 24, 2006

 
Blogger Lucas Carlson said...

The user is an ActiveRecord object that has a users email address. The notifier is an ActionMailer object that sends email.

6:44 AM, August 24, 2006

 
Blogger Adrian said...

I think I'm starting to grasp this better. One feature you might want to add is a delay time between runs so you don't send too many emails too fast (you can get blacklisted) or some other external requirement.

Thanks,


Adrian Madrid

3:01 PM, August 24, 2006

 
Anonymous chris hulbert said...

Holy damn friggin hell, thats elegant code!
Lucas, keep it up.

5:31 PM, August 24, 2006

 
Anonymous JonD said...

Starfish looks great and I'd love to use it but am still not sure on the whole process.

Could you write up a little tutorial in the future that goes through the process of doing something with Starfish from start to finish?

10:46 PM, September 02, 2006

 
Anonymous Jay said...

This looks amazingly simple, but it's not immediately obvious how to get it to work properly from that example. For example, it seems like the way of starting up clients could cause problems. You said that you called "starfish email_sender.rb" 10 times, but how does each invocation know which "run" it should be a part of? For example, what if the task actually finished after your 8th call to "starfish email_sender.rb" w/o you realizing it? It seems that your remaining 2 calls to "starfish email_sender.rb" would start up the task again and cause duplicate emails to be sent.

Along the same lines, if these are started on remote machines, how do they communicate w/o specifying a head server? Are they storing the queue info and the mutex in the database itself?

Sorry for all the questions!

6:34 PM, September 11, 2006

 
Anonymous Anonymous said...

Jay's questions seem legitimate or am I just too much of an ameteur to get it?

8:11 AM, January 24, 2007

 
Anonymous Anonymous said...

Jay, I'm just reading up on starfish myself, and I found myself asking the same questions as you.

As for your first question (in paragraph 1), I think that's the beauty of the whole server/client system. There is one server that maintains the state of what has been processed and what has not, manages the mutex, etc. So, the clients really don't 'know' anything about the parallelization - they just know how to process one carefully chosen item (in this case a User, I believe).

As for your second question, I found myself wondering the same thing. I think the answer might be hidden in the config details of require 'config/environment'.

Check out the docs for more details, of course:
http://rufy.com/starfish/doc/

rinogo
(starfish.20.rinogo @ xoxy.net)

12:10 PM, May 26, 2007

 
Blogger Lai Weizhou said...

chung cư tháp doanh nhân
tháp doanh nhân / tháp doanh nhân hà đông / dự án tháp doanh nhân hà đông / chung cư the legend / chung cư the legend 109 nguyễn tuân
chung cư 360 giải phóng / dự án 360 giải phóng / chung cư vietracimex / chung cư 201 minh khai / chung cư thanh xuân tower / chung cư 35 lê văn thiêm
chung cư eco green city / chung cu eco green city / chung cư eco green city nguyễn xiển / chung cu eco green city nguyen xien / eco green city / eco green city nguyễn xiển / dự án eco green city / toà ct4 eco green city / toà ct3 eco green city / toà ct1 eco green city

11:08 AM, May 21, 2016

 
Blogger Zhenhong Bao said...

hermes outlet
ferragamo outlet
swarovski outlet
coach outlet online
toms shoes
ray-ban sunglasses
beats headphones
coach outlet online
mulberry handbags
chaussure louboutin
tory burch outlet online
cheap ray ban sunglasses
ray ban sunglasses
louis vuitton neverfull
michael kors wholesale
true religion outlet
mont blanc pens
cheap football shirts
kate spade handbags
mulberry bags
gucci outlet online
cheap jordans
nfl jersey wholesale
ralph lauren outlet
rolex watches for sale
tory burch outlet online
christian louboutin shoes
mulberry handbags
true religion jeans sale
true religion uk outlet
christian louboutin outlet
lebron james shoes
coach outlet
nike blazer pas cher
longchamp pliage
20160531zhenhong

6:19 PM, May 30, 2016

 
Blogger Fangyaya said...

air jordan 13
gucci outlet
adidas outlet
michael kors outlet
true religion outlet
cheap jordans
cheap jordans
louis vuitton outlet
coach outlet
uggs for men
michael kors outlet
air jordan 8
ralph lauren uk
true religion jeans
fitflops
louis vuitton handbags uk
ralph lauren polo shirts
ugg sale
ugg boots
rolex watches
mlb jerseys
oakley sunglasses
ray ban sunglasses uk
nike outlet store
michael kors outlet online
louis vuitton handbags
uggs on sale
supra shoes
hermes handbags
chenyingying20160908

6:51 PM, September 07, 2016

 

Post a Comment

Subscribe to Post Comments [Atom]

<< Home

 

If you like this blog, you might also like top photography schools.