Angel \”Java\” Lopez on Blog

June 15, 2015

Spark: Links and Resources (1)

Filed under: Distributed Computing, Java, Links, MapReduce, Spark — ajlopez @ 10:09 am

There are two project named Spark in Java: a web framework, and a distributed map reduce runner.

Scalable Machine Learning | edX

Spark Programming Guide – Spark 1.3.1 Documentation

Overview – Spark 1.3.1 Documentation



Top Best Tools for Java Programmers | Devzum – Its all about Design & Development

Sparkling, A Clojure API for Apache Spark

Getting Started with Sparkling

Developing Single Page Web Applications using Java 8, Spark, MongoDB, and AngularJS – OpenShift Blog


Spark Framework – A tiny Java web framework

Apache Spark with Scala


Windows on Devices

MapReduce and Spark | Cloudera VISION

Mesosphere · Learn how to use Apache Mesos

Got a Minute? Spin up a Spark cluster on your laptop with Docker. | AMPLab – UC Berkeley

Spark Streaming Programming Guide – Spark 0.7.3 Documentation

Spark: Open Source Superstar Rewrites Future of Big Data | Wired Enterprise |

Spark | Lightning-Fast Cluster Computing

Learning Spark – O’Reilly Media

My Links

Stay tuned!

Angel “Java” Lopez

June 13, 2015

Computer History: Links And Resources (17)

Filed under: Computer History, Links — ajlopez @ 8:21 pm

Previous Post

Happy 100th birthday Hedy Lamarr, the inventor who made the wireless internet possible

The Hypercard Legacy

Oracle founder Larry Ellison resigns after 35 years as CEO | Technology |

Object Oriented Programming: A Critical Approach

(507) What is the most complex/significant program created by a single programmer? – Quora

(496) Who is considered to be the best programmer of all time? – Quora

Best Business Books To Read Fall 2014 – Business Insider

Randy Pausch – Wikipedia, la enciclopedia libre

(492) What is a Lisp machine and what is so great about them? – Quora

Why Should I Care What Color the Bikeshed Is?

GoRuCo 2014 – What We Can Learn From COBOL by Andrew Turley – YouTube

Twitter ‘Buy Now’ Button Appears for First Time

Waterfall | Martín Alaimo | Agile Coach & Trainer

AKKA 5 Year Anniversary

The Mouse Trap: Raising Lazarus – The 20 Year Old Bug that Went to Mars

Guido van Rossum on the History of Python – YouTube

Google Glass and the Future of Technology –

My Links

Stay tuned!

Angel “Java” Lopez

June 9, 2015

Computer History: Links And Resources (16)

Filed under: Computer History, Links — ajlopez @ 8:55 pm

Previous Post
Next Post

Talking with Mikel Evins about the Lisp-based Newton OS from Apple

I, Cringely The Decline and Fall of IBM – I, Cringely

You won’t believe how old TDD is | Arialdo Martini

20 Years of Beowulf Workshop Issues Call for Papers – insideHPC

The Golden Age of Basic – IEEE Spectrum

General Magic – Wikipedia, the free encyclopedia

Scientific computing’s future: Can any coding language top a 1950s behemoth? | Ars Technica

Injected, Inspected, Detected, Infected, Neglected and Selected | The Weekly Squeak

The Origins of Linux – Linus Torvalds – YouTube

xerox :: parc :: techReports :: CSL-79-13 WFS A Simple Shared File System for a Distributed Environment : Free Download & Streaming : Internet Archive

Preliminary discussion of the logical design of an electronic computing instrument

In less than a decade, phone memory cards have grown from 128MB to 128GB | The Verge

Let Over Lambda

Un argentino ayudó a nacer al lenguaje Basic – 03.05.2014 –

How Steve Wozniak Wrote BASIC for the Original Apple From Scratch

Happy 50th Birthday, BASIC! – The Visual Basic Team – Site Home – MSDN Blogs

My Links

Stay tuned!

Angel “Java” Lopez

June 6, 2015

Docker: Links, News And Resources (2)

Filed under: DevOps, Docker, Links — ajlopez @ 8:29 pm

Previous Post

3 reasons for using Docker in your open cloud architecture – Thoughts on Cloud

Initial thoughts on the Rocket announcement | Docker Blog

CoreOS is building a container runtime, Rocket

Why Docker and CoreOS’ split was predictable – Daniel With Music

What is Tutum? | Tutum Blog

Gartner Panel Reveals Stark Differences in Container Based PaaS Options | Pivotal P.O.V.

Docker Hosting – Run Docker Containers in the Cloud – Tutum

Tutum | CrunchBase

Tutum, a startup building on Docker’s hip tech, reels in $2.6M | VentureBeat | Deals | by Jordan Novet

Docker: Present and Future

Technophilia: Distrubuted JMeter testing using Docker

The beginner’s guide to Docker | Application virtualization – InfoWorld

Getting started with Docker |

Systemd: Harbinger of the Linux apocalypse

How to Docker Workshop, Step-by-Step

Docker: The first true devops tool? | Application development – InfoWorld

The March Towards Go –

Docker Gets Ignore, Auto-Pauses Containers On Commit

My Links

Stay tuned!

Angel “Java” Lopez

June 4, 2015

New Month’s Resolutions: June 2015

Filed under: .NET, C Sharp, Java, JavaScript, Liqueed, NodeJs, Open Source Projects — ajlopez @ 9:48 am

It’s the time for review my May’s resolutions and write the new ones:

– Improve ClojJS [pending]
– Add NPM support to ClojJS [pending]
– Write posts about JavaScript and Artificial Intelligence [pending]
– Give a talk about Meteor [partial]
– Prepare a talk about Clojure or ClojureScript [partial]
– Improve BScript [pending]
– Improve AjErl, distributed features [partial] see repo
– Improve Liqueed Project, kudos features [complete] see repo

I also worked on:

– Start SparkSharp, Apache Spark-like in C# [complete] see repo
– Improve SharpMongo, MongoDB-like in C# [complete] see repo
– Improve OStore, object store in memory, JavaScript/NodeJS [complete] see repo
– Improve PythonSharp, Python interpreter in C# [complete] see repo
– Improve RedPython, compile Python to C using JavaScript/NodeJS [complete] see repo

My new month’s resolutions:

– Give a talk about Meteor
– Write posts about JavaScript and Artificial Intelligence
– Improve ClojJS
– Add NPM support to ClojJS
– Continue work on SparkSharp
– Start Message project in C#, a la Apache Camel
– Improve Liqueed Project, kudos features
– Improve Templie, template engine in Java
– Improve BScript, Basic-like interpreter in C#
– Continue work on OStore
– Continue work on SharpMongo

More fun is coming ;-)

Stay tuned!

Angel “Java” Lopez

June 3, 2015

Computer History: Links And Resources (15)

Filed under: Computer History, Links — ajlopez @ 10:27 am

Previous Post
Next Post

Celebrating 50 years of BASIC

Smalltalk??? (2013/1/31)?????? – YouTube

PARC Movies – YouTube

Before Silicon Valley got nasty, the Pirates of Analog Alley fought it out | Ars Technica

Visual Programming Languages – Snapshots

The Birth & Death of JavaScript — Destroy All Software Talks

Memories of Steve

RailsConf 09: Robert Martin, “What Killed Smalltalk Could K – YouTube

Monad Manifesto

The Man Behind the Google Brain: Andrew Ng and the Quest for the New AI | Enterprise | WIRED

The Oberon-07 language report is 17 pages

50 years of BASIC: Celebrating the programming language’s long, eventful life

Xv6, a simple Unix-like teaching operating system

Objects are Just Objects, Aren’t they? – Rick DeNatale – Ruby Conference 2010

How I Came to Write D | Dr Dobb’s

BBC News – Half-century milestone for IBM mainframes

S-expression – Wikipedia, the free encyclopedia

Visual Smalltalk Brochure – a set on Flickr

My Links

Stay tuned!

Angel “Java” Lopez

June 1, 2015

Computer History: Links And Resources (14)

Filed under: Computer History, Links — ajlopez @ 10:05 am

Previous Post
Next Post

Contributions of Individual Programming Languages to Software Development | Javalobby

The Art of Atari: A celebration of game packaging’s golden age | Polygon

Conway’s Game Of Life in APL – YouTube

Why Language Designers Tolerate Undefined Behavior | Dr Dobb’s

Microsoft makes source code for MS-DOS and Word for Windows available to public – The Official Microsoft Blog – Site Home – TechNet Blogs

Emails From Eric Schmidt And Sergey Brin On Hiring Apple Workers – Business Insider

Systems Past: the only 8 software innovations we actually use – Technical Journal

Self: The Movie; – YouTube

Ruby Turns 21: 5 Major Milestones Of The Programming Language – ReadWrite

Douglas Lenat

artificial intelligence – How To Design Eurisko – Stack Overflow

Eurisko, The Computer With A Mind Of Its Own | Alicia Patterson Foundation

EURISKO – Lesswrongwiki

Let’s reimplement EURISKO! – Less Wrong

Eurisko – Wikipedia, the free encyclopedia

James Iry’s history of programming languages (illustrated with pictures and large fonts) | The Quick Word

Robot Odyssey: The hardest computer game of all time.

_why: La historia de un genio post-moderno

Apple ‘1984’ Commercial Is 30 Years Old Today (VIDEO)

My Links

Stay tuned!

Angel “Java” Lopez

May 30, 2015

Computer History: Links And Resources (13)

Filed under: Computer History, Links — ajlopez @ 10:02 am

Previous Post
Next Post

Remembering Apple’s “1984” Super Bowl ad – O Say Can You See?

Starting January 24, 2014

The Mac Turns 30

ANNOUNCEMENT: The Scala Programming Language

Rasmus Lerdorf: A look at PHP 5.4 and 5.5 – YouTube – Open Source HyperCard-related stuff

25 years of HyperCard—the missing link to the Web | Ars Technica

Origins of Common UI Symbols

Screen shots of computer code: Photo

When Apple drops the ball: the gear that flopped | News | TechRadar


Alan Turing, Enigma Code-Breaker and Computer Pioneer, Wins Royal Pardon –

launch – Wikipedia’s forgotten founder Larry Sanger


PHP Manual Masterpieces

ComputerHistory – YouTube

Oral History of Adele Goldberg – YouTube


blog dds: 2013.12.11 – The Birth of Standard Error

Grace Hopper: Google doodle for creator of Cobol computer language – Mirror Online

Fast and Dynamic

DEFCON 19 (2011) – The History and Evolution of Computer Viruses – YouTube

My Links

Stay tuned!

Angel “Java” Lopez

May 28, 2015

SparkSharp, Spark in C# (2) Implementing Map and Reduce

Filed under: .NET, C Sharp, Open Source Projects, Spark, SparkSharp — ajlopez @ 10:15 am

Previous Post

The project repo is at:

I’m using TDD (Test-Driven Development) workflow to write the application, so the code is evolving with the use cases I added as tests. I only wrote the code needed to pass the tests. When I had new use cases, I will add new functionality. As an example, the original project has an Spark Context, with factory methods to create Datasets. I don’t need that class, yet. So in the current conde, the Datasets are created as public objects using new operator..

Born after a refactor, the abstract class for all Datasets is BaseDataset. Partial code:

public abstract class BaseDataset<T> : IEnumerable<T>
    public abstract IEnumerable<T> Elements { get; }

    public BaseDataset<S> Map<S>(Func<T, S> map)
        return new EnumDataset<S>(this.ApplyMap(map));

    public S Reduce<S>(Func<S, T, S> reduce)
        S result = default(S);

        foreach (var elem in this)
            result = reduce(result, elem);

        return result;
    // ...

    private IEnumerable<S> ApplyMap<S>(Func<T, S> map)
        foreach (var elem in this)
            yield return map(elem);
    // ...

The enumeration of the dataset elements should be implemented by the concrete subclass. The implementation of Map and Reduce is general, for all datasets. Those methods are defined in the abstract class. Thanks to C#, those methods can receive a lambda or a Func (a function).

In the ApplyMap method I’m using the C# yield operator to return an element suspending the executiong of the foreach. That command will resume when the consumer needs the next element of the enumerable collection. In this way, the generation of the elements is lazy, each element is produced only under demand. A note: C# has lambdas and delegate functions, and they are examples of good and useful features added to a programming language. In contrast, Java world has Scala, that in my opinion, it a bit “too much”. I prefer the evolution of C# instead of Scala.

There are no tests using the abstract class (it was born as a refactor), but they are tests on concrete ones. A test of Map method using EnumDataset (a Dataset that is a wrapper around an IEnumerable collection):

public void MapIncrement()
    EnumDataset<int> ds = new EnumDataset<int>(new int[] { 1, 2, 3 });
    BaseDataset<int> mapds = ds.Map(i => i + 1);
    var enumerator = mapds.GetEnumerator();

    for (int k = 1; enumerator.MoveNext(); k++)
        Assert.AreEqual(k + 1, enumerator.Current);

    Assert.AreEqual(3, mapds.Count());

And a Reduce test:

public void ReduceSum()
    EnumDataset<int> ds = new EnumDataset<int>(new int[] { 1, 2, 3 });
    var result = ds.Reduce<int>((x, y) => x + y);

    Assert.AreEqual(6, result);

Next topics: more BaseDataset methods, concrete classes, datasets with keys, etc…

Stay tuned!

Angel “Java” Lopez

May 23, 2015

SparkSharp, Spark in C# (1) First Ideas

Filed under: .NET, C Sharp, Open Source Projects, Spark, SparkSharp — ajlopez @ 11:20 pm

Next Post

In these days, I visited Apache Spark project:

And started to think about implementing some of its ideas in C#.

The original project has Datasets, that can be consumed, item by item, and processed by methods like map and reduce. A dataset can consume a text file, local files or distributed ones. The jobs to run over datasets, applying transformations, can be launched in many distributed nodes (I should review the consolidation of results).

I started a new C# project:

To me, it is important to start with small steps, using TDD (Test-Driven Development) workflow. So, in my first commits, I wrote datasets that implement IEnumerable. They have methods like Map, Reduce, Split, Take, Skip. Those methods were implemented writing the tests that express the expected API and behavior.

A dataset can be a simple wrapper of any IEnumerable, or it can read a text file, reading lines.

All these datasets are local, reside in the same machine. My idea is to implement a dataset wrapper, to expose the dataset content to other machines, and write a client wrapper that runs in each machine. The client wrapper looks like a regular dataset, but when the client program needs the next item of the dataset, that item come from the remote original machine.

The remote dataset gives the next item to any client. Each item is delivered only to ONE client. So, the items can be consumed and processed by n remote clients, without having an item processed twice.

To implement such pair server/client, I should implement serialization/deserialization of an arbitrary type T. I will use my previous work in AjErl and Aktores to have such feature. Using TDD, I could assert the expected behavior of the serializaction/deserialization process. If in the future, I have a better idea for such process, like using an external robust open source serialization library, all the TDD tests will help me to make the switch.

But, baby steps. Next steps: improve current local datasets, maybe add a new variant of dataset, and write keyed datasets, created using MapToKey method (to implement)

Stay tuned!

Angel “Java” Lopez

« Newer PostsOlder Posts »

The Shocking Blue Green Theme. Blog at


Get every new post delivered to your Inbox.

Join 68 other followers