Angel \”Java\” Lopez on Blog

July 28, 2014

SharpDoop Implementing Map Reduce in C# (1) The Project

Filed under: .NET, C Sharp, MapReduce, Open Source Projects, SharpDoop — ajlopez @ 4:51 pm

I was experimenting with map-reduce in Node.js / JavaScript. But it is also something interesting to implement the algorithm in C #, to practice TDD (Test-Driven Development) and to learn more about what it takes to implement the algorithm. The project that I started is:

https://github.com/ajlopez/SharpDoop

The current status:

It has a class library project and the test project. I came to think of something not yet distributed. All I want for now is to specify the map/reduce algorithm map, and run it in the same process.

See that there is then a MapReduceJob. The base class that is responsible for running a map (lambda function) and reduce (takes a key,  a list of values for that key, and output the result). With C #, these two operations can be expressed as lambdas or delegate methods inside a more complex object. For now, I pose the simple cases in the tests, and all them are running.

I also put together a MapProcessJob, which is a variant of map / reduce to discuss. Instead of processing all keys first, then roll them into the reduce phase, try to do everything together. In some use cases it may be more efficient, but is an issue to discuss in more detail.

Upcoming topics: more in-depth review and implementation ideas, such as map / process.

Stay tuned!

Angel “Java” Lopez
http://www.ajlopez.com
http://twitter.com/ajlopez

1 Comment »

  1. I delight in, cause I found exactly what I was having a look for.
    You have ended my four day long hunt! God Bless you
    man. Have a nice day. Bye

    Comment by Alethea — August 23, 2014 @ 3:57 pm


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Blog at WordPress.com.

%d bloggers like this: