Angel \”Java\” Lopez on Blog

July 28, 2014

SharpDoop Implementing Map Reduce in C# (1) The Project

Filed under: .NET, C Sharp, MapReduce, Open Source Projects, SharpDoop — ajlopez @ 4:51 pm

I was experimenting with map-reduce in Node.js / JavaScript. But it is also something interesting to implement the algorithm in C #, to practice TDD (Test-Driven Development) and to learn more about what it takes to implement the algorithm. The project that I started is:

https://github.com/ajlopez/SharpDoop

The current status:

It has a class library project and the test project. I came to think of something not yet distributed. All I want for now is to specify the map/reduce algorithm map, and run it in the same process.

See that there is then a MapReduceJob. The base class that is responsible for running a map (lambda function) and reduce (takes a key,  a list of values for that key, and output the result). With C #, these two operations can be expressed as lambdas or delegate methods inside a more complex object. For now, I pose the simple cases in the tests, and all them are running.

I also put together a MapProcessJob, which is a variant of map / reduce to discuss. Instead of processing all keys first, then roll them into the reduce phase, try to do everything together. In some use cases it may be more efficient, but is an issue to discuss in more detail.

Upcoming topics: more in-depth review and implementation ideas, such as map / process.

Stay tuned!

Angel “Java” Lopez
http://www.ajlopez.com
http://twitter.com/ajlopez

Blog at WordPress.com.