Quantcast

Ruby on Rails, Io, Lisp, JavaScript, Dynamic Languages, Prototype-based programming and more...

Technoblog reader special: $10 off web hosting by FatCow!

Wednesday, August 16, 2006

MapReduce for Ruby: Ridiculously Easy Distributed Programming


Digg
del.icio.us
FURL
Yahoo! My Web 2.0
Reddit


I am very happy to announce that Google's MapReduce is now available for Ruby (via gem install starfish). MapReduce is the technique used by Google to do monstrous distributed programming over 30 terabyte files. I have been reading about MapReduce recently and thought that it was very exciting for Google to have laid out the ideas that ran Google. I also wondered how they could be applied to everyday applications.

Recently, I gave a talk on Ridiculously easy ways to distribute processor intensive tasks using Rinda and DRb. This talk came from my work with Rinda recently at MOG. We use distributed programming to handle real-time processor intensive needs for over 1 million requests a day. We also use it to make large changes or clean up our database. I realized that the plumbing I wrote in Rinda to accomplish these tasks could be abstracted and easily conform to the MapReduce technique.

Before I move on, I will provide a little more background of Google's MapReduce. MapReduce is a C++ library written by Google. There are about 12 MapReduce programs used to create the inverted index of the www that Google uses for searching. The term MapReduce itself refers to map and reduce functions. Joel recently wrote an article that explains what map a reduce do, so I will refrain from repeating him. One of the parts Joel unfortunately messed up on was this sentence though:

[...] you only have to get one supergenius to write the hard code to run map and reduce on a global massively parallel array of computers, and all the old code that used to work fine when you just ran a loop still works only it's a zillion times faster which means it can be used to tackle huge problems in an instant [...]

Google, nor anyone I know, has written a map function that will "replace" your existing calls to map, like a plugin. In fact, here is some real world MapReduce example code that is used to provide a word count on an arbitrarily sized document:

#include "mapreduce/mapreduce.h"

// User's map function
class WordCounter : public Mapper {
public:
virtual void Map(const MapInput& input) {
const string& text = input.value();
const int n = text.size();
for (int i = 0; i < n; ) {
// Skip past leading whitespace
while ((i < n) && isspace(text[i]))
i++;
// Find word end
int start = i;
while ((i < n) && !isspace(text[i]))
i++;
if (start < i)
Emit(text.substr(start,i-start),"1");
}
}
};

REGISTER_MAPPER(WordCounter);

// User's reduce function
class Adder : public Reducer {
virtual void Reduce(ReduceInput* input) {
// Iterate over all entries with the
// same key and add the values
int64 value = 0;
while (!input->done()) {
value += StringToInt(input->value());
input->NextValue();
}
// Emit sum for input->key()
Emit(IntToString(value));
}
};

REGISTER_REDUCER(Adder);

int main(int argc, char** argv) {
ParseCommandLineFlags(argc, argv);
MapReduceSpecification spec;

// Store list of input files into "spec"
for (int i = 1; i < argc; i++) {
MapReduceInput* input = spec.add_input();
input->set_format("text");
input->set_filepattern(argv[i]);
input->set_mapper_class("WordCounter");
}

// Specify the output files:
// /gfs/test/freq-00000-of-00100
// /gfs/test/freq-00001-of-00100
// ...
MapReduceOutput* out = spec.output();
out->set_filebase("/gfs/test/freq");
out->set_num_tasks(100);
out->set_format("text");
out->set_reducer_class("Adder");

// Optional: do partial sums within map
// tasks to save network bandwidth
out->set_combiner_class("Adder");

// Tuning parameters: use at most 2000
// machines and 100 MB of memory per task
spec.set_machines(2000);
spec.set_map_megabytes(100);
spec.set_reduce_megabytes(100);

// Now run it
MapReduceResult result;

if (!MapReduce(spec, &result)) abort();
// Done: 'result' structure contains info
// about counters, time taken, number of
// machines used, etc.
return 0;
}

MapReduce takes a large data set (in this case a large file), divides the file into many different pieces, and lets 2000 machines each count words and aggregate statistics for a small part of that file, aggregating the result together in the end.

One of the parts that stood out to me is how there is a clear separation of how to do the call to map and how to do the call to reduce. The other part is all the set calls like spec.set_machines(2000);. I love the simplicity: you tell the system how to map, you tell it how to reduce, you set some options, and run it. Notice specifically that you are not writing network code... this is obviously a very network intensive task, but that is all hidden behind #include "mapreduce/mapreduce.h". This is much like Rinda for Ruby where you do not have to write any network code to distribute objects over the network. You do however have to learn an API to use either Rinda or DRb. MapReduce feels much less like an API and more like a layout, a template that you fill in.

I took the lessons from MapReduce, injected my background of Ruby and came up with what I call Starfish. The backend implementation of Starfish is vastly different than Google's MapReduce: MapReduce is highly optimized for speed and best use of 2000 computer resources at a time, Starfish is highly optimized for speed of development and ease of use. That said, the goal of Starfish is the same as MapReduce.

Starfish takes a large data set (in this case a database), divides the table into many different sections, and lets machines each do work on sections of the database in parallel, aggregating the result together in the end.

Here is some example code:

class Item < ActiveRecord::Base; end

server do |map_reduce|
map_reduce.type = Item
end

client do |item|
logger.info item.some_processor_intensive_task
end

You will notice a few major differences quite quickly. First, you do not need to require any libraries, if this file was called item.rb you would run
starfish item.rb
on the command line on as many servers as you want and it will do everything it needed to start working and distributing the work. Next, you do not specify map and reduce functions, rather you specify a client and a server. I loved the simplicity and clarity of defining the two most important parts to Google's MapReduce, but in Ruby it would have been silly to do so because it is not C++ and mapping and reducing is too easy. So I gave it some thought and came up with what I thought was the most important part of distributed programming: what does the server serve and how do the client process the served objects.

Aside from the differences, you will notice the similarity, in the server you are setting options, setting map_reduce.type = Item much like input->set_format("text"); in MapReduce. In the near future, you will be able to tell Starfish that the type is File and let Starfish process files the same way we saw MapReduce do it in the example. Also, logger.info sends some information back to the server that logs it to a file much the same way that out->set_filebase("/gfs/test/freq"); works.

However the biggest major difference is that Starfish is open-source and easy to use. Performing distributed tasks is now a ridiculously easy reality for programmers that may not have been steeped enough in CORBA or some other library to accomplish before.

I hope that you find this library helpful, please tell me how you use it and how I can make it work better for you. There any many options I didn't cover, so if you do use it, please read the documentation.

UPDATE: I wrote an example of how I sent emails 10x faster than before using Starfish.

You should follow me on twitter here.

Technoblog reader special: click here to get $10 off web hosting by FatCow!

80 Comments:

Blogger Amr Malik said...

Lucas, this sounds very interesting indeed! Are you at liberty to discuss the architecture at MOG?

I am wondering why it bacame necessary (from an architecture perspective) to read huge datasets (30gb?) from the database and process in memory?

Aren't DBMS systems optimized for this sort of processing or is it something that the DBMS does not offer (horizontally partitioned tables or federated views etc) and therefore necessary to do it in this manner.

I would be very interested in your thoughts on this.

thanks,

1:31 PM, August 18, 2006

 
Anonymous Joe Martinez said...

very nice sir!

2:28 PM, August 18, 2006

 
Anonymous Anonymous said...

The beauty of google's system is that the data was distributed as well. Aren't you going to have bottleneck issues with database disk i/o?

5:06 PM, August 18, 2006

 
Blogger Lucas Carlson said...

Certainly, but Google built their own distributed file system to deal with that issue. Starfish is not ideal for super-large scale distributed efforts, but it certainly helps for many of the distributed systems you regularly encounter.

5:23 PM, August 18, 2006

 
Anonymous Anonymous said...

Looking at your reference to GFS ("/gfs/test/freq"), does it mean that your library works on GFS? I didn't know Google had relased this piece of technology. And the same for Google's clusters of commodity Linux PCs. Without that infrastructure, how your library works? Thanks.

5:55 PM, August 18, 2006

 
Anonymous Anonymous said...

mmm... i have a feeling you don't understand the mapreduce operation well. by spliting into server and clients, your abstraction not only leaks the underlaying network, but you completely forgot the reduce part. certainly you made a distributed system but not a map reduce operation. it looks to me like a small wrapper on top of drb.

6:04 PM, August 18, 2006

 
Blogger Lucas Carlson said...

To address the GFS comment, GFS is a completely independent tool, a filesystem, used at Google exclusively. Neither Starfish nor MapReduce directly know or care about the underlying filesystem involved. Google does utilize the implications of GFS when using MapReduce to handle some of the disk i/o issues. I am not interested in solving that particular problem since I have not personally reached a point where i/o is the limiting factor.

To address the question to my competence, I would like to say that Starfish and MapReduce were written in vastly different languages with different goals in mind. After studying the papers associated with MapReduce as well as the code examples, I thought to myself about the problems MapReduce set out to solve. I am not interested in writing an exact clone of the code involved to run MapReduce, I am interested in solving a similar problem on a smaller scale.

Secondarily, I am very interested in making distributed programming more accessible to programmers who have never tried to do distributed programming before. I wanted to keep familiar terms like "client" and "server," and keep unfamiliar terms at bay. I feel this helps to quickly understand what is going on which aids in maintenance of Starfish code.

To directly address whether or not I understand the what the mapreduce operation does, I can tell you that in Ruby, mapreduce is simply a call to inject.

To address why I did not feel compelled to add a distributed reduce function to the initial release of starfish, I felt that such a function would be much simpler to write as a helper method to the server than a whole separate operation into itself. I may change my mind about that, but I was going for ease of development of distributed applications, and I find helper methods defined in the server much quicker and easier to write than full reduction methods.

Starfish is not built to compete with MapReduce, it is meant to solve smaller scale distributed programming problems. That said, I believe it succeeds at what it does like no other library I have ever seen.

As a final comment, more of a correction, Starfish is a small wrapper built upon Rinda, which is a small wrapper built upon DRb. I would like to thank Matz for making a language like Ruby and Masatoshi Seki for writing DRb (originally in less than 90 lines of code), it is a joy to stand on the shoulder of giants and write such useful utilities like Starfish as a small wrapper.

9:34 PM, August 18, 2006

 
Anonymous Anonymous said...

"MapReduce feels much less like an API and more like a layout, a template that you fill in." it is indeed an occurrence of the "template method" design pattern...

3:39 AM, August 19, 2006

 
Anonymous Anonymous said...

A key feature of Google's MapReduce
is that it's built on top of a batch
system to provide fault tolerance.
So that if a worker node fails
during execution, its subtask will
be transparently reassigned to another node. I haven't looked at the code - does your solution provide fault tolerance? Without it, I doubt the solution would be very practical (but the fact that it's in Ruby is cool!)

I work with Condor and I myself have
recently been thinking about implementing a "poor man's" MapReduce in Python or Ruby on top
of Condor.

7:07 AM, August 19, 2006

 
Blogger Lucas Carlson said...

It currently does not support fault tolerance, but it will shortly, fault tolerance is easy to add to Starfish the way that I programmed it.

11:26 AM, August 19, 2006

 
Blogger DJB said...

Looks interesting. I posted a few comments on the project page regarding ways to make it more Windows friendly. :)

- Dan

9:41 PM, August 19, 2006

 
Anonymous Anonymous said...

it seems to be a map only, no reduce solution

9:48 PM, August 19, 2006

 
Anonymous Anonymous said...

This is quite misleading:

"I am very happy to announce that Google's MapReduce is now available for Ruby"

Furthermore, the class is named MapReduce.

BUT it's not MapReduce, and it's certainly not "Google's" either.

Yes, it is missing reduce, and there's no point in trying to refute that. MapReduce is NOT simply a call to inject, the same way the stuff you wrote is not "simply a call to #each".

Starfish seems fairly useful on its own; there's no need for you to publicize it (indirect- and somewhat unfortunately) as "Google's MapReduce".

12:02 PM, August 20, 2006

 
Blogger Lucas Carlson said...

You are entitled to have your anonymous opinion and I am entitled to mine. There are actually ways to do the reduce function in Starfish which I will go into more detail in soon.

1:09 PM, August 20, 2006

 
Blogger Ronie Uliana said...

That's a great API, indeed! I'd like we have something like that on other languages, maybe in a near future. But, so far, to have something like that on Ruby is great for me.

6:54 AM, August 21, 2006

 
Blogger Adam Rosien said...

It's not really MapReduce, but I like your Starfish library. I write about my own MapReduce at my blog. Maybe we can collaborate!

10:50 AM, August 21, 2006

 
Blogger Lucas Carlson said...

I would love to collaborate, let's see where we can go with this.

11:00 AM, August 21, 2006

 
Anonymous Adrian Madrid said...

Lucas, your library sounds very interesting to. I guess I'm in your target market since I have not played with distributed programming and I have a project coming that would need it. Could you elaborate some more on a n example of how it would be used?

Thanks,


Adrian Madrid

12:18 PM, August 21, 2006

 
Anonymous Anonymous said...

Lucas, I am might impressed with your work thus far. I look forward to seeing the fault tolerance implemented.

Most distributed frameworks are needlessly complex; please stick to your guns and keep the core simple.

Thanks again for your contribution!

Jim
http://www.runfatboy.net

10:05 PM, August 22, 2006

 
Anonymous Anonymous said...

when installing the gem i've got

lib/starfish.rb:179:27: Couldn't find RingFinger. Assuming it's a module

ruby 1.8.4 on tiger

6:05 AM, September 01, 2006

 
Blogger Lucas Carlson said...

That is nothing to worry about, just a warning.

11:35 AM, September 01, 2006

 
Blogger Sausheong said...

Starfish doesn't seem to work in Windows.

6:58 AM, October 20, 2006

 
Anonymous Anonymous said...

The examples here: http://rufy.com/starfish/doc/ don't show this technology using multiple machines. How easy is it to setup in a distributed environment? It's looking great btw!

12:44 PM, November 03, 2006

 
Anonymous Anonymous said...

i buy hydrocodone at buy hydrocodone - can't find any cheaper

7:06 AM, January 28, 2007

 
Anonymous Jared said...

Like some others have said... This seems like it might be useful in a distributed computing environment for doing big tasks... From the example code, I have seen, this is not Map Reduce in the same sense as the Google Labs paper.

Perhaps it's your choice of example? Would you might posting a complete example code of how you would implement the classic example of term counting presented in Section 2.1 of the original MapReduce paper from Google?

4:20 PM, January 29, 2007

 
Blogger f00biebletch said...

I've tried playing with starfish, one thing that is not clear at all from the examples is how to run a client. I've set up a server hitting a mysql table, and running multiple instances of starfish on a single physical node works great. However, if I simply copy the same code to another node and run starfish concurrently on multiple nodes, I get redundant processing. How do I configure the code to run one server and multiple clients across multiple physical nodes? Thanks.

11:58 AM, January 30, 2007

 
Blogger Wendy said...

I'm having trouble getting starfish to work with a class hierarchy implemented as single table inheritance in rails.

Is this something starfish can handle, or does it only work with a single class per table?

thanks.

9:13 PM, June 05, 2007

 
Anonymous Anonymous said...

Old thread but i found this in relation to:

http://skynet.rubyforge.org/

12:48 AM, January 03, 2008

 
Blogger Adam said...

I'm the author of Skynet. This article and the code that Lucas wrote was heavily influential in the development of Skynet. Thanks Lucas!

adam

2:10 PM, January 06, 2008

 
Anonymous Anonymous said...

Lucas, is there a way to remove items from the ActiveRecord source immediately after a client processes it from the map_reduce queue? In your email example, this would be the same as deleting an email after it was sent. Thanks!

8:37 PM, February 09, 2008

 
Anonymous شركة بن طامى said...

ارخص شركة تنظيف بالرياض

5:30 AM, May 21, 2016

 
Anonymous شركة بن طامى said...


شركة تنظيف منازل شمال الرياض

افضل شركة جلي بلاط بالرياض

ارخص شركة تنظيف موكيت

افضل شركة تنظيف كنب بالرياض

شركة نظافة

5:31 AM, May 21, 2016

 
Anonymous شركة بن طامى said...


افضل شركة مكافحة نمل ابيض بالرياض

مكافحة نمل ابيض بالرياض

افضل شركة نقل اثاث بالرياض

ارخص شركة نقل عفش بالرياض

افضل شركة تخزين اثاث بالرياض

شركة شراء اثاث

افضل شركة كشف تسربات بالرياض

ارخص شركة كشف تسربات

افضل شركة ترميمات بالرياض

ارخص شركة دهانات بالرياض

شركة شفط بيارات بالرياض

5:40 AM, May 21, 2016

 
Blogger Lai Weizhou said...

chung cư tháp doanh nhân
tháp doanh nhân / tháp doanh nhân hà đông / dự án tháp doanh nhân hà đông / chung cư the legend / chung cư the legend 109 nguyễn tuân
chung cư 360 giải phóng / dự án 360 giải phóng / chung cư vietracimex / chung cư 201 minh khai / chung cư thanh xuân tower / chung cư 35 lê văn thiêm
chung cư eco green city / chung cu eco green city / chung cư eco green city nguyễn xiển / chung cu eco green city nguyen xien / eco green city / eco green city nguyễn xiển / dự án eco green city / toà ct4 eco green city / toà ct3 eco green city / toà ct1 eco green city

11:08 AM, May 21, 2016

 
Blogger Zhenhong Bao said...

toms outlet
michael kors online
ralph lauren uk
coach factory outlet
tory burch outlet
replica watches
toms shoes
fitflops sale
coach outlet
michael kors factory outlet
oakley sunglasses wholesale
longchamp handbags
air max 90
michael kors handbags wholesale
louis vuitton sunglasses
michael kors outlet online
louis vuitton bags
oakley sunglasses
ray ban sunglasses
fitflops clearance
chaussure louboutin
swarovski crystal
michael kors outlet store
michael kors outlet sale
chrome hearts outlet
louboutin pas cher
coach outlet online
louis vuitton outlet
louis vuitton neverfull sale
ralph lauren outlet
rolex watches
swarovski outlet
nike huarache
oakley sunglasses
ralph lauren outlet
20160531zhenhong

6:22 PM, May 30, 2016

 
Blogger Guu cà phê said...

Mua cây kim tiền ở đâu
Cây kim tiền

2:13 AM, May 31, 2016

 
Anonymous viliolo said...

This is very amazing to see such work over here. Slitherio | Agario | Happy Wheels | happy fathers day 2016 quotes Carry on the discussions over night.

9:23 PM, May 31, 2016

 
Blogger rabaa bel said...

This is the Truth About how The Venus Factor can help losing weight without starving yourself. watch Venus Factor Diet Program

6:02 AM, June 05, 2016

 
Blogger Guu cà phê said...

bơ đậu phộng

6:56 PM, June 07, 2016

 
Blogger rabaa bel said...

Survive In Bed Review Learn This is my Story With Survive In Bed Program

6:12 AM, June 08, 2016

 
Blogger Kelly Anni said...

I want you to thank for your time of this wonderful read!!! I definately enjoy every little bit of it and I have you bookmarked to check out new stuff of your blog a must read blog!
slither io | wings io | science kombat | tank trouble 4

10:13 AM, June 09, 2016

 
Blogger Regina Hilary said...


The war between humans, orcs and elves continues. Lead your race through a series of epic battles, using your crossbow to fend off foes and sending out units to destroy castles. Researching and upgrading wisely will be crucial to your success! There are 5 ages total and each one will bring you new units to train to fight in the war for you cause.
earn to die game vip | earn to die play game | unfair mario | slitherio | tanh trouble | age of war 6
The game controls are shown just under . Movement mechanisms primarily include acceleration and tilting controls.
It consists of a total of 17 levels and the challenge you face in each level increases as you go up. The game basically has a red ball that has to be moved across the various obstacles in its path to the goal.
tank trouble | age of war | age of war 6 | gold miner | tank trouble 3 | age of war

3:21 AM, June 14, 2016

 
Blogger Hossein Khojasteh said...

***شرکت نوين گيت***

با سالها سابقه در طراحي ،اجرا و خدمات سيستمهاي درب اتوماتيک و راهبند
فروش،نصب و راه اندازي و خدمات انواع درب اتوماتيک (درب سکشنال - کرکره اتوماتيک - کرکره پنجره - درب ريلي اتوماتيک - جک پارکينگي - کرکره شفاف - رول گيتر - انواع راهبند و ... )
خدمات 24 ساعته و شبانه روزي حتي در تعطيلات رسمي
مشاوره رايگان در تمامي مراحل

http://www.novingate.com/

http://www.novingate.ir/

http://www.novinwebsite.ir/website-design

طراحي سايت

نوين گيت

تاجران آريا دلفين

http://www.fa.adt-co.ir/

Aria Dolphin

1:02 AM, June 29, 2016

 
Blogger Giải Trí Tổng Hợp said...

Welcomes to google terminal from Austin Texas! google snake
Amazing insight you have on this, Happy wheels it's nice to find a website that details so much information about different artists... Age of war 2 This article always blew me... Earn to die For how many times I have read this.
slither io Hi! I’ve been reading your blog for a while slitherio it's nice to find a website that details so much big farm

12:47 AM, July 01, 2016

 
Blogger rokn elbeet said...

ما تهتم به شركة كشف تسربات المياه بالرياض هو البحث عن كيفة علاج مشاكل تسريبات المياه التي تطرأ علي المكان فجأة بواسطة اجهزة الكشف الحدية التي تستخدمها شركة كشف تسربات بالرياض والتي تسعي للوصول الي افضل النتائج المثالية القادرة علي حل هذه المشكلة بدون تدمير فالاعتماد علي الاساليب الحديثة يساعدكم في الحصول علي نتيجة مثالية في مصلحة العميل فنحن لا نكتفي بتقديم هذه الاعمال في مدينة الرياض فقط بلا لدينا الفنين المتميزة الذي يقدمون شركة كشف تسربات المياه بالدمام التي تعمل علي حل مشكلة البيت بدون الاعتماد علي ا اساليب تقليدية التي تستخدما بعض مقدمي خدمة شركة كشف تسربات بالدمام فلا تتكايل بشأن هذا العمل بالذات لانه يحل لك الكثير من المشاكل

8:28 AM, July 15, 2016

 
Anonymous al3ab-7arb said...



العاب الحرب اول موقع عربي متخصص في العاب الحرب بشكل عام ومجانا
العاب حرب حربية
العاب حرب
العاب فلاش حرب
al3ab 7arb

9:03 PM, July 15, 2016

 
Blogger Mạc Huệ said...

I like the news that you have written in a detail.
Thank you for the information you provide. I had a look to start my own. Papas Games | Papas Game | Papa Games | Papas Game | Papas Games | Papas Game | Stick RPG 2 | Stick RPG 2 | Stick RPG | StickRPG2 | Stick RPG | Happy Wheels | Happy Wheel | happy wheels game | Happy Wheels | Happy Wheel | happy wheels game | unblocked games online free | Run 2 unblocked | Run 2 unblocked
Diep.io | Diepio | Diep io | Diep.io | Diepio |

7:28 PM, July 19, 2016

 
Blogger Linda Rose said...

Very helpful advice in this particular post! It’s the little changes that make the largest changes. Thanks for sharing!
Run 3 | Stick Run 2 | Return Man 2
Run 2 | Stick Run | Return Man | run 3 game | run 3 unblocked | Tank Trouble | Tank Trouble 2
Papa's Bakeria | Papa Bakeria

9:03 PM, July 28, 2016

 
Blogger Hua Cai said...

asics,asics israel,asics shoes,asics running shoes,asics israel,asics gel,asics running,asics gel nimbus,asics gel kayano
ralph lauren uk
michael kors outlet
christian louboutin uk
nike air max 90
mlb jerseys
nfl jersey wholesale
swarovski outlet
michael kors handbags
rolex watches
nike free run
nba jerseys
nike trainers uk
ugg boots
ray ban sunglasses
rolex watches
mont blanc pens
nike uk store
coach outlet clearance
nike blazer pas cher
michael kors uk
mulberry handbags
vans outlet
ralph lauren outlet
nike store uk
cheap oakley sunglasses
canada goose jackets
nhl jerseys
louis vuitton pas cher
kobe shoes
coach outlet
nba jerseys
air max 90
nike foamposite
ralph lauren pas cher
20160804caihuali

10:44 PM, August 03, 2016

 
Blogger Utorrent Game said...

Great post and straight to the point. I am not sure if this is truly the best place to ask but do you guys have any ideea where to hire some professional writers? Thank you :)
Just Cause 3 Torrent
Rise of the Tomb Raider Torrent
Homefront The Revolution Torrent
doom Torrent
No Man’s Sky Torrent

6:01 PM, August 09, 2016

 
Blogger Apple Shooter said...

Wonderful blog! I found it while searching on Yahoo News. Do you have any suggestions on how to get listed in Yahoo News? I’ve been trying for a while but I never seem to get there! Many thanks.
2048 games | five nights at freddys 4 | plants vs zombies | five nights at freddys 3 | fireboy and watergirl | fireboy and watergirl 4

6:50 PM, August 19, 2016

 
Blogger Tran Hien said...

Introducing to you some of the current hot game
red ball 3 || age of war
or
tank trouble || happy wheels
or
earn to die
this is one of the cult game now, a lot of people enjoy playing them :
slitherio || wingsio
gmail sing in

12:12 AM, August 24, 2016

 
Blogger dong dong23 said...

nike nfl jerseys
mont blanc pens for sale
fitflops uk
oklahoma city thunder jerseys
coach outlet online
coach outlet
kate spade outlet
ugg boots
cartier watches
coach outlet store online clearances
michael kors handbags
tods outlet
polo ralph shirts
canada goose pas cher
lebron james shoes 12
canada goose coats
oakley store
kobe 9 shoes
coach factory outlet
fitflops
coach outlet
polo ralph lauren outlet
ugg store
cheap ray ban sunglasses
oakley sunglasses
nike factory outlet
ugg boots
ugg canada
ray ban sunglasses
mbt shoes
coach factory outlet
beats headphones
replica rolex watches
christian louboutin chaussures
coach outlet store online
gucci handbags
uggs for women
jordan retro 3
louis vuitton outlet
christian louboutin sale
20169.8wengdongdong

7:13 PM, September 07, 2016

 
Blogger Đồ Gỗ Mạnh Cường said...

Sàn Đất Vàng Hà Nội
Chung cư Yên Nghĩa
Điện lạnh Mạnh Dũng
Khu đô thị thanh hà
Đồ gỗ Mạnh Cường
Thế giới cây và hoa

2:17 AM, September 12, 2016

 
Blogger Kiên Văn Đình said...

I was working and suddenly I visits your site frequently and recommended it to me to read also. Age Of War 2
Big Farm | Slitherio
Tank Trouble | Happy Wheels
Goodgeme Empire | Slither.io

1:00 AM, September 26, 2016

 
Blogger John said...

ugg outlet
burberry bags
cheap ugg boots
cheap oakley sunglasses
ugg pas cher
polo ralph lauren outlet online
ysl handbags
oakley sunglasses outlet
ed hardy outlet
michael kors outlet canada
2016927yuanyuan

6:23 PM, September 26, 2016

 
Blogger MINHTRANG said...

I think you'll love it!Thanks!

snapchat

baixar snapchat

baixar snapchat gratis

como usar o snapchat

download snapchat

snapchatdownload

snapchat online

baixar google

baixar google play

baixar google play gratis

baixar googleplay

download google play

googleplay

baixar googleplay gratis

baixar play store

play store


google play download



playstore baixar



play store download


6:59 AM, September 28, 2016

 
Blogger Phu Nguyen said...

I think you'll love it!Thanks!

snapchat

baixar snapchat

baixar snapchat gratis

como usar o snapchat

download snapchat

snapchatdownload

snapchat online

baixar google

baixar google play

baixar google play gratis

baixar googleplay

download google play

googleplay

baixar googleplay gratis

baixar play store

play store


google play download



playstore baixar



play store download



6:10 AM, September 29, 2016

 
Blogger Phu Nguyen said...

I think you'll love it!Thanks!

snapchat

baixar snapchat

baixar snapchat gratis

como usar o snapchat

download snapchat

snapchatdownload

snapchat online

baixar google

baixar google play

baixar google play gratis

baixar googleplay

download google play

googleplay

baixar googleplay gratis

baixar play store

play store


google play download



playstore baixar



play store download



8:49 PM, October 05, 2016

 
Blogger Lê Hạnh said...

chung cư ct2 yên nghĩa hà đông
chung cư thanh hà cienco 5
chung cư ct2 yên nghĩa
chung cư the golden an khánh

6:51 PM, October 07, 2016

 
Blogger Phu Nguyen said...

I think you'll love it!Thanks!

snapchat

baixar snapchat

baixar snapchat gratis

como usar o snapchat

download snapchat

snapchatdownload

snapchat online

baixar google

baixar google play

baixar google play gratis

baixar googleplay

download google play

googleplay

baixar googleplay gratis

baixar play store

play store


google play download



playstore baixar



play store download


2:19 AM, October 10, 2016

 
Anonymous Convert youtube videos into mp3 said...

Thanks for sharing

12:48 AM, October 14, 2016

 
Anonymous Green henry said...

amazing info. http://itsgreenhenry.tumblr.com/

1:30 AM, October 14, 2016

 
Blogger chenlina said...

coach outlet store online
denver broncos jerseys
ray-ban sunglasses
ugg boots uk
ray ban sunglasses
uggs outlet
coach factory outlet
coach outlet store online
longchamp outlet
louis vuitton outlet
chenlina20161018

7:53 PM, October 17, 2016

 
Blogger Surfers Subway said...

Amazing .. Thanks for giving me the useful information. I think I need it!
Subway Surfers
Subway Surf
Subway Surfers download

7:04 PM, October 18, 2016

 
Blogger Outlet Wade said...

Nike Air Max 90
Nike Air Max 95
Nike Huarache
Nike Outlet
Nike Roshe Run
Nike Air Max Thea
Nike Air Max Tavas
Nike Air Max 90
Nike Air Max 95
Nike Shoes
Nike Flyknit
Nike Flyknit Racer
Nike Air Max
Nike Shoes
Nike Outlet
Nike Store
Nike Factory Store
Nike Outlet Store
Nike Air Max 2016
Nike Air Max 2017
Giuseppe Zanotti Sneakers
Giuseppe Zanotti Shoes
Valentino Shoes
Jimmy Choo Shoes
Yeezy Boost 350
Adidas Yeezy

5:15 AM, October 25, 2016

 
Blogger Candy Sim said...

The blog or and best that is extremely useful to keep I can share the ideas
of the future as this is really what I was looking for, I am very comfortable and pleased to come here. Thank you very much.
gold miner|stick war 3| pokemongo
| stick man|animal jam login

9:42 PM, October 31, 2016

 
Blogger Ellis Amy said...

Hotmail is also known as Outlook mail or an e-mail services provided by Microsoft that is probably popular after Google’s Gmail. With a Hotmail account, you can send and receive emails quickly and easily as well as login and use all Microsoft services.
hotmail login
sign into hotmail
hotmail sign in
recover hotmail password
create hotmail account

1:30 AM, November 01, 2016

 
Blogger Giải Trí Tổng Hợp said...

The blog or and best that is extremely useful to keep I can share the ideas. Age Of War 2
Big Farm | Slitherio | Tank Trouble
Of the future as this is really what I was looking for, I am very comfortable and pleased to come here. Thank you very much.
Happy Wheels | Goodgeme Empire | Slither.io


7:12 PM, November 01, 2016

 
Blogger بدر ابو المعاطى said...

شركة ركن الاصالة تقدم غروض سهله لعملئها بئرخص الاسعار بلا منافس فنحن شركة ركن الاصالة شركه كبيره جدا في التنظيف ولكن هل سئلت يوما ان المواد التي تستخدم في عالم النظافه انهاامنه ام لااقول انهاامنه وفعاله بنسبه مئه في المئه ولكن شركتنا تقدم وتحطم الاسعاربسب ارتفاع الاسعار وبسب اوضاع البلد بمايحدث من ظروف سياسيه واجتتماعي واقتصاديه فشركتنا تقد ارخص الاسعار في عالم النظافه من اجل اطفالك ومن اجل نفسك ومن اجل كل عئلتك ان تكون دقيقا في اختيار الشركه التي تقوم بعمليه التنظيف وعليك ايضا ان تكون علي قدر كبير من الزكاء والفطانه في علمك لئختيار الشركه ومواد الشركه فهل هياامنه ام لاعالم النظافه عالم واسع وكبير فكن حزرا وكن فاهما فعليك ان تري بنفسك الفرق شركة شركة تنظيف بالمدينة المنورة
شركة مكافحة حشرات بالمدينة المنورة
شركة تنظيف بجدة

4:59 AM, November 02, 2016

 
Blogger Dash Geometry said...

great post. i like it. feeling great when reading your post. Keep on posting!
fireboy watergirl
Geometry Dash

2:14 AM, November 03, 2016

 
Blogger Jenny Daring said...

Such a very useful article. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article.

Discover our website bounty of free online games now!
Our website has the biggest collection of free online games. Totally new games are added every day!

age of war 2
gold Miner 2
unfair Mario 2
cubefield 2
tanki Online 2

9:37 PM, November 03, 2016

 
Blogger yoob games said...

تعتبر العاب تلبيس من اشهر الانواع في هذا المجال وهي بدورها تتضمن عدة اصناف جميلة ويعشقها الكتير وخاصة البنات منها العاب تلبيس ومكياج التي تمزج بين التلبيس وكذلك الميك اب في آن واحد هذا الامر الدي يزيد من جمالها وتجعل كل من يلعبها يستمتع بذلك زد على ذلك العاب تلبيس باربي التي تعرف شعبية كبيرة لانها شخصية مشهورة ويعرفها الصغير والكبير ولهم ذكريات جميلة معها لانها اشتهرت في عالم الكارتون والان اصبح الامر كذلك في مجال الالعاب وغير هذا هناك كذلك نوع آخر مميز ايضا وهو العاب تلبيس عرائس فالجميع يحلم ان يقوم بتلبيسهما لانها تذكرهم بهذه المناسبة الجميلة الا وهي الزواج التي تعتبر اهم مرحلة في حياة الانسان وهناك انواع مغايرة لها جمهور كبير في كل انحاء العالم وهي العاب قص الشعر ليس هي فقط بل توجد ايضا العاب طبخ التي يمكن للجميع لعبها سواء كانوا اولادا او بناتا وهي الاكتر طلبا في النت ويحبها الجميع ومعها ايضا العاب باربي التي تكلمنا عليها بكل انواعها
العاب فلاش تعرف انتشارا مميزا وغير عادي في الايام الاخيرة وكل الفئات تلعبها لانها مسلية وممتعة جدا وبعد ان اصبح لها معجبين كتر في نفس الوقت كترت المواقع الخاصة بالالعاب وتتميز الالعاب بكترة اصنافها فهناك انواع خاصة بالاولاد واخرى بالبنات وايضا هناك انواع يلعبها الكل متل العاب طبخ متلا فيمكن للاولاد لعبها والبنات ايضا وهناك اصناف يلعبها الاولاد فقط على سبيل المتال العاب سيارات و العاب سباق زد على ذلك العاب كرة القدم و العاب فرايف وللبنات انواع ايضا يلعبونها بكترة واكترها العاب تلبيس بنات و ايضا العاب مكياج و العاب قص الشعر والعديد من الاصناف الاخرى
طبخ كيك
طبخ سارة
لعبة طبخ 01السباغيتي
لاعداد الكاب كيك

6:40 PM, November 08, 2016

 
Blogger Ellis Amy said...

With a Hotmail account, you can send and receive emails quickly and easily as well as login and use all Microsoft services.
hotmail login | create new hotmail account
Creating a Gmail account is quick and easy. You can follow the tutorial to sign up Gmail here.
gmail login | gmail sign in | install google drive

1:16 AM, November 14, 2016

 
Blogger hường lê said...

Such a very useful article. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article.

Discover our website bounty of free online games now!
Our website has the biggest collection of free online games. Totally new games are added every day!

age of war 2| gold Miner 2| unfair Mario 2 | cubefield 2 |tanki Online 2

8:31 PM, November 17, 2016

 
Anonymous gun mayhem said...

This is the game so players feel very fierce , adventurous . You try to join and play this game , you will feel great like.
| tank trouble | learn to fly
gun mayhem | can your pet
| happ wheels

12:37 AM, November 23, 2016

 
Blogger Mark Smith said...

Ladies Rolex Watches JavyEstrella.com is a Los Angeles-based wholesaler and online retailer of authentic luxury watches. Their experts have a combined expertise of over 75 years of selling, buying and distributing authentic Rolex and other luxury watches worldwide.

3:30 PM, November 23, 2016

 
Blogger bui loan said...

Thanks for sharing your post. Thanks to this I can understand more about the role of education. This article is easy to understand and useful. Do you like Piano Tiles 2, Sushi Cat 2, B612, fancy pants 3, baixar mobogenie, Happy Wheels 4?

6:09 PM, November 25, 2016

 
Blogger Elnaz Ghajar said...

Thank you very much. Great information. Best regards…
تست جوش
درب اتوماتیک
زومر
زومر
کرکره برقی
راهبند اتوماتیک
درب پارکینگ
پرده زبرا
راهبند اتوماتیک
کرکره برقی

12:53 AM, December 01, 2016

 
Anonymous Anonymous said...

Cara Menghilangkan Bau Tidak Sedap Saat Berhubungan Intim
Cara Menghilangkan Bekas Jahitan Yang Menonjol
Penyebab Ibu Hamil Kekurangan Air Ketuban
Cara Mengeluarkan Silikon dan Minyak Kemiri di Alat Vital
Cara Mengobati Penis Akibat Minyak Kemiri

11:54 PM, December 02, 2016

 

Post a Comment

Subscribe to Post Comments [Atom]

<< Home

 

If you like this blog, you might also like top photography schools.