Ruby on Rails, Io, Lisp, JavaScript, Dynamic Languages, Prototype-based programming and more...

Technoblog reader special: $10 off web hosting by FatCow!

Tuesday, August 22, 2006

How I sent emails 10x faster than before

Like many startups, at MOG we send out regular updates to our users with news and information. As our user base expands, sending this email takes more and more time. Even though the call to deliver the mail only puts it in the sendmail queue, it can take a chunk of time to do so with so many users.

When I demoed Starfish to people, the common response was: that's great, I wish I had a use for it, I wish I had a DB source big enough to use. Well here is one, albeit not mission critical but still damn cool, of the ways we use Starfish.

require 'config/environment'
require 'user'
require 'notifier'

server do |map_reduce|
map_reduce.type = User
map_reduce.conditions = "opt_out = 0"

client do |user|

This tiny amount of code with next to nothing that needs to be memorized and takes 30 seconds to write down can potentially save you hours in deliver time. Even running 10 clients at once on the SAME MACHINE gave us nearly 10x the speed it would have taken serially. This was not mission critical, but gives you a good sense of ways to apply Starfish to mission critical applications.

You should follow me on twitter here.

Technoblog reader special: click here to get $10 off web hosting by FatCow!


Anonymous Anonymous said...

that's so simple and clean... way simpler than background drb, even simpler than using ruby threads (which might not provide the parallelism you need anyhow).

3:11 AM, August 22, 2006

Blogger Lucas Carlson said...

Plus, Ruby threads + Rails stack = Crap shoot

10:07 AM, August 22, 2006

Anonymous Anonymous said...

Thanks for the example. It seems cleared to me now how you can use Starfish.

9:37 AM, August 23, 2006

Anonymous Anonymous said...

Would you mind detailing what exactly the code above is doing.


10:38 AM, August 23, 2006

Blogger Lucas Carlson said...

The code above creates a queue of User objects. The clients grab a user from the queue and sends an email to the user, then grab another user and send an email, over and over. If you have 10 clients, it is like splitting up the collection over 10 loops and running them concurrently, hence the 10x speedup.

11:42 AM, August 23, 2006

Anonymous Anonymous said...

How does your example code know to use starfish?

12:05 PM, August 23, 2006

Blogger Lucas Carlson said...

You save that file, for example as: email_sender.rb. Then you execute the code by calling: starfish email_sender.rb. Since I wanted 10 clients processing the data, I called starfish email_sender.rb 10 times.

12:11 PM, August 23, 2006

Anonymous Anonymous said...

How do you prevent multiple processes from sending e-mails to the same person?

Meaning, if you had a text file contain unique e-mail addresses to mail a message to, how does the script above make sure that only 1 e-mail goes to each unique e-mail address?

1:01 PM, August 23, 2006

Anonymous Anonymous said...

Since you are running it on the same machine, how is this different from creating a multi-thread application?

Isn't this essentially all this is?

1:11 PM, August 23, 2006

Blogger Lucas Carlson said...

James: It uses a queue system, so two clients can never grab the same line from a file because once grabbed, that line is not accessible to any other clients. It also has a simple mutex to prevent two calls at the exact same time.

Hank: Have you ever tried multi-threading Rails? It is almost always a big source of headaches do to things since there would be conflicts with multiple attempts to concurrently use the connection to the DB. Plus, obviously I didn't have to call them all on the same server which is a big bonus.

1:23 PM, August 23, 2006

Anonymous Anonymous said...

Hey I just wrote the worlds shortest operating system:

#include "theuniverse.h"

int main()

I'm kidding, I'm kidding, but you get my point? What is user doing, what is notifier?

5:54 AM, August 24, 2006

Blogger Lucas Carlson said...

The user is an ActiveRecord object that has a users email address. The notifier is an ActionMailer object that sends email.

6:44 AM, August 24, 2006

Blogger Adrian said...

I think I'm starting to grasp this better. One feature you might want to add is a delay time between runs so you don't send too many emails too fast (you can get blacklisted) or some other external requirement.


Adrian Madrid

3:01 PM, August 24, 2006

Anonymous Anonymous said...

Holy damn friggin hell, thats elegant code!
Lucas, keep it up.

5:31 PM, August 24, 2006

Anonymous Anonymous said...

Starfish looks great and I'd love to use it but am still not sure on the whole process.

Could you write up a little tutorial in the future that goes through the process of doing something with Starfish from start to finish?

10:46 PM, September 02, 2006

Anonymous Anonymous said...

This looks amazingly simple, but it's not immediately obvious how to get it to work properly from that example. For example, it seems like the way of starting up clients could cause problems. You said that you called "starfish email_sender.rb" 10 times, but how does each invocation know which "run" it should be a part of? For example, what if the task actually finished after your 8th call to "starfish email_sender.rb" w/o you realizing it? It seems that your remaining 2 calls to "starfish email_sender.rb" would start up the task again and cause duplicate emails to be sent.

Along the same lines, if these are started on remote machines, how do they communicate w/o specifying a head server? Are they storing the queue info and the mutex in the database itself?

Sorry for all the questions!

6:34 PM, September 11, 2006

Anonymous Anonymous said...

Jay's questions seem legitimate or am I just too much of an ameteur to get it?

8:11 AM, January 24, 2007

Anonymous Anonymous said...

Jay, I'm just reading up on starfish myself, and I found myself asking the same questions as you.

As for your first question (in paragraph 1), I think that's the beauty of the whole server/client system. There is one server that maintains the state of what has been processed and what has not, manages the mutex, etc. So, the clients really don't 'know' anything about the parallelization - they just know how to process one carefully chosen item (in this case a User, I believe).

As for your second question, I found myself wondering the same thing. I think the answer might be hidden in the config details of require 'config/environment'.

Check out the docs for more details, of course:

(starfish.20.rinogo @ xoxy.net)

12:10 PM, May 26, 2007

Blogger Unknown said...

chung cư tháp doanh nhân
tháp doanh nhân / tháp doanh nhân hà đông / dự án tháp doanh nhân hà đông / chung cư the legend / chung cư the legend 109 nguyễn tuân
chung cư 360 giải phóng / dự án 360 giải phóng / chung cư vietracimex / chung cư 201 minh khai / chung cư thanh xuân tower / chung cư 35 lê văn thiêm
chung cư eco green city / chung cu eco green city / chung cư eco green city nguyễn xiển / chung cu eco green city nguyen xien / eco green city / eco green city nguyễn xiển / dự án eco green city / toà ct4 eco green city / toà ct3 eco green city / toà ct1 eco green city

11:08 AM, May 21, 2016

Anonymous Anonymous said...

hermes outlet
ferragamo outlet
swarovski outlet
coach outlet online
toms shoes
ray-ban sunglasses
beats headphones
coach outlet online
mulberry handbags
chaussure louboutin
tory burch outlet online
cheap ray ban sunglasses
ray ban sunglasses
louis vuitton neverfull
michael kors wholesale
true religion outlet
mont blanc pens
cheap football shirts
kate spade handbags
mulberry bags
gucci outlet online
cheap jordans
nfl jersey wholesale
ralph lauren outlet
rolex watches for sale
tory burch outlet online
christian louboutin shoes
mulberry handbags
true religion jeans sale
true religion uk outlet
christian louboutin outlet
lebron james shoes
coach outlet
nike blazer pas cher
longchamp pliage

6:19 PM, May 30, 2016

Blogger Fangyaya said...

air jordan 13
gucci outlet
adidas outlet
michael kors outlet
true religion outlet
cheap jordans
cheap jordans
louis vuitton outlet
coach outlet
uggs for men
michael kors outlet
air jordan 8
ralph lauren uk
true religion jeans
louis vuitton handbags uk
ralph lauren polo shirts
ugg sale
ugg boots
rolex watches
mlb jerseys
oakley sunglasses
ray ban sunglasses uk
nike outlet store
michael kors outlet online
louis vuitton handbags
uggs on sale
supra shoes
hermes handbags

6:51 PM, September 07, 2016

Blogger chenlina said...

ugg australia
oakley sunglasses
coach outlet
tods shoes
oklahoma city thunder jerseys
fitflops uk
michael kors outlet
ed hardy clothing
designer handbags
coach outlet store online

7:51 PM, October 17, 2016

Blogger chenmeinv0 said...

uggs outlet
louis vuitton
christian louboutin shoes
moncler soldes
cheap nhl jerseys
christian louboutin sale
moncler coats
steelers jerseys
uggs on sale
replica rolex watches

10:33 PM, January 03, 2017

Blogger chenmeinv0 said...

air jordan shoes
coach outlet online
chaussures ugg
christian louboutin
mcm handbags
louis vuitton outlet online
nike roshe run pas cher
montblanc pens
adidas outlet
canada goose outlet

10:34 PM, January 03, 2017

Blogger Unknown said...

jimmy choo
nike trainers
replica watches
michael kors
nike air huarache
moncler outlet
new balance shoes
gucci sale
michael kors handbags

12:35 AM, March 16, 2017

Blogger Unknown said...

polo ralph lauren outlet
discount oakley sunglasses
kate spade outlet store
adidas nmd runner
cheap rolex replica watches
cardinals jerseys
mlb jerseys
fred perry polo shirts
true religion
pandora jewelry

6:36 AM, April 18, 2017

Blogger Unknown said...

I know there will be many difficulties and challenges but I am determined to do it. If it does not succeed then it will be a lesson for me as well

8:02 PM, November 27, 2017

Blogger Đào Duy Quang Blog said...

Tọa lạc tại vị trí đắc địa Chung cư 378 Minh Khai được mệnh danh là " Viên Ngọc" giữa lòng thành phố với vị trí không thể thuận tiện hơn tại nút giao thông huyết mạch nối liền các trung tâm Chính trị, Văn hóa,Trường học, Bệnh viện, Sân gold của thủ đô Hà Nội

Dự án Green Pearl tọa lạc tại số 378 Minh Khai , Hai Bà Trưng, Hà Nội là một vị trí đắc địa thuộc cửa ngõ phía Nam của Thủ đô, nơi tập trung các tuyến giao thông huyết mạch của TP Hà Nội (Đường Minh Khai - Cầu Vĩnh Tuy). Dự án bao gồm nhà ở liền kề, biệt thự và căn hộ cao cấp Green Pearl mở ra không gian sống xanh, trong lành, KHÔNG ồn ào, KHÔNG khói bụi ngay trong lòng đô thị.

Dự án 378 Minh Khai hiểu rõ vui chơi là hình thức tốt nhất giúp trẻ phát triển tư duy. Từ đó chúng tôi xây dựng trường học hiện đại, khu vui chơi trẻ em với những trò vui chơi giải trí, trí tuệ giúp trẻ phát triển cả tư duy sáng tạo và thể chất.

Chung cư Green Pearl
Chung cư Green Pearl Không chỉ là giải pháp đáp ứng nhu cầu nhà ở chất lượng cao, sứ mệnh của Green Pearl là cùng bạn trải nghiệm cuộc sống" bình yên trọn vẹn "tách biệt với cuộc sống ngột ngạt ồn ào bên ngoài. Không gian yên tĩnh trong lành được khai thác tối đa trên những con đường tản bộ xanh mát sẽ giúp bạn cảm thấy thư thái nhằm tái tạo năng lượng hiệu quả nhất.
Mặt bằng chung cư Green Pearl
Tiện ích chung cư Green Pearl
Giá bán chung cư Green Pearl
Chung cư Green Pearl

8:19 PM, January 12, 2018

Blogger Joya said...

may be useful for all, helpful article once and pardon me permission to share also here :

Cara mengatasi daya ingat menurun
Manfaat pisang ambon untuk kesehatan
Bahaya amandel
Cara mengatasi alzheimer

8:24 PM, June 07, 2018

Blogger 5689 said...

christian louboutin sale
off white shoes
pandora outlet
ugg boots clearance
canada goose outlet
canada goose jackets
moncler outlet
christian louboutin outlet
soccer boots
christian louboutin shoes

6:22 PM, September 01, 2018

Blogger نور الهدى said...

نقل عفش من الدمام الى مكة نقل عفش من الدمام الى مكة
نقل عفش من الرياض الى الدمام نقل عفش من الرياض الى الدمام

نقل عفش من الرياض الى جدة نقل عفش من الرياض الى جدة
نقل عفش بحفر الباطن نقل عفش بحفر الباطن
نقل عفش الجبيل نقل عفش الجبيل
نقل عفش بالخبر نقل عفش بالخبر

3:21 PM, December 08, 2018

Blogger Sarah Betty said...

This post is really informative and you have posted such precious and informative article which gave me lot of information. I hope that you will keep it up and we will have more informative and helping news from you.

"The Friv games therefore are really exciting and are outstanding. The consumer interests and therefore are designed in ways that fascinates and captivates the people enjoying it. It is a great globe you leak and creates your gambling experience better still into, the game is pleased to back up you with exciting articles and improvements you using the greatest. Also you play kilka friv gry online and when you are fixed for your chair, you will not discover once the period flew away enjoying the game, yes it’s exemplary and that fascinating to play."

2:00 AM, February 03, 2019

Blogger skan81 said...

clinique de chirurgie plastique
reduction mammaire
chirurgien esthetique tunisie
chirurgie plastique
chirurgie esthetique tunisie
chirurgie esthetique en tunisie

6:35 AM, May 21, 2019

Blogger skan81 said...

voiture occasion
discotheque a Paris
discotheque a Paris
voiture vintage
bet paris
entreprise genie energetique
renovation appartement paris
prix ouverture mur porteur
ouverture mur porteur prix

6:36 AM, May 21, 2019

Blogger skan81 said...

entreprise generale de batiment
Agence Marketing Digital
Entreprise de revetement
ouvrage d'art

6:38 AM, May 21, 2019

Blogger skan81 said...

diagnostic de structure
bet structure idf
Maçonnerie paris
bureau d’etudes techniques pluridisciplinaire
Ouvrir un mur porteur

6:38 AM, May 21, 2019

Blogger skan81 said...

chirurgien esthetique tunisie
photo chirurgie esthetique tunisie
aesthetic tunisia
video chirurgie esthetique tunisie
blog tunisie
referenceur Tunisie

6:40 AM, May 21, 2019

Blogger skan81 said...

جراحة التجميل تونس
chirurgie esthetique
greffe de cheveux en tunisie
changer les protheses mammaire
reduction des seins tunisie
operation esthetique tunisie

6:42 AM, May 21, 2019

Blogger Hằng VT said...

I hope this post of yours will be more appreciated by it really excellent, i enjoyed it, thanks for posting it. I do not feel sorry for taking the time to read this post, it is really nice and useful to me, thanks for posting it.
abcya3 hot games, friv for school for kids 2019, cá koi mini, io jogos for kids

7:51 AM, August 06, 2019


Post a Comment

Subscribe to Post Comments [Atom]

<< Home


If you like this blog, you might also like top photography schools.