Distributed Computing: Links, News And Resources (5)

Previous Post

Amazon Redshift
http://aws.amazon.com/redshift/
Amazon Redshift is a fully managed, petabyte-scale data warehouse service in the cloud.

hgrecco / pizco
https://github.com/hgrecco/pizco
Pizco is Python module/package that allows python objects to communicate via ZMQ. Objects can be exposed to other process in the same computer or over the network, allowing clear separation of concerns, resources and permissions.

Ricon2012
http://basho.com/community/ricon2012/
A distributed systems conference for developers

causeway
http://code.google.com/p/causeway/
Debugger for distributed communicating event loops programs

ActiveMQ or RabbitMQ or ZeroMQ or
http://stackoverflow.com/questions/731233/activemq-or-rabbitmq-or-zeromq-or

Building Healthy Distributed Systems
http://www.infoq.com/presentations/Distributed-Systems-Basho
Mark Phillips discusses 3 types of distributed systems and how they run them at Basho: Computer Systems, Communities, and Companies.

Ken: A Platform for Fault-Tolerant Distributed Computing
http://ai.eecs.umich.edu/~tpkelly/Ken/
Ken is a lightweight C implementation of a rollback-recovery protocol that provides crash-restart resilience to distributed applications.

Embracing Concurrency at Scale
http://www.infoq.com/presentations/Concurrency-Scale-Distributed
Justin Sheehy discusses designing reliable distributed systems that can scale in order to deal with concurrency problems and the tradeoffs required by such systems.

c9 / smith
https://github.com/c9/smith
Smith is an RPC agent system for Node.JS used in vfs

Project Voldemort
http://www.project-voldemort.com/voldemort/
Voldemort is a distributed key-value storage system

Amazon’s Dynamo
http://www.allthingsdistributed.com/2007/10/amazons_dynamo.html

How to get NetLogo/BehaviorSpace up and running on a Linux cluster
http://groups.yahoo.com/group/netlogo-users/message/11210
Communication Patterns in Cloud Haskell
http://www.well-typed.com/blog/73

Concurrent and Distributed Applications with Akka, Java and Scala
http://www.slideshare.net/frodriguezolivera/concurrent-and-distributed-applications-with-akka-java-and-scala

Cluster Computing with Node.js
http://architects.dzone.com/articles/cluster-computing-nodejs

Distributed Algorithms in NoSQL Databases
http://highlyscalable.wordpress.com/2012/09/18/distributed-algorithms-in-nosql-databases/

Doing redundant work to speed up distributed queries
http://www.bailis.org/blog/doing-redundant-work-to-speed-up-distributed-queries/

A Peek At Facebook’s Insanely Awesome Monitoring Tool Claspin
http://techcrunch.com/2012/09/19/a-peak-at-facebooks-insanely-awesome-monitoring-tool-claspin/
Beyond the Higgs: Training PanDA to Tackle Astrophysics, Biology
http://www.quantumdiaries.org/2012/09/18/higgs-hunting-software/
With the help of a system called PanDA, or Production and Distributed Analysis, researchers at CERN’s Large Hadron Collider (LHC) in Geneva, Switzerland discovered such a particle by slamming protons together at relativistic speeds hundreds of millions of times per second. The data produced from those trillions of collisions—roughly 13 million gigabytes worth of raw information—was processed by the PanDA system across a worldwide network and made available to thousands of scientists around the globe.

Spanner: Google’s Globally-Distributed Database
http://research.google.com/archive/spanner.html

Spanner: Google’s Globally-Distributed Database
http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en//archive/spanner-osdi2012.pdf

Bacterial Foraging Optimization
http://msdn.microsoft.com/en-us/magazine/hh882453.aspx

Joxa
http://joxa.org/
Joxa is a small semantically clean, functional lisp. It is a general-purpose language encouraging interactive development and a functional programming style. Joxa runs on the Erlang Virtual Machine. Like other Lisps, Joxa treats code as data and has a full (unhygienic) macro system.

Drill
http://wiki.apache.org/incubator/DrillProposal
Drill is a distributed system for interactive analysis of large-scale datasets, inspired by Google’s Dremel.

What is Memcached?
http://memcached.org/

Riemann monitors distributed systems
http://aphyr.github.com/riemann/
Redis Sentinel beta released
http://oldblog.antirez.com/post/redis-sentinel-beta-released.html

janm399 / akka-patterns
https://github.com/janm399/akka-patterns
Demonstration of the common Akka patterns

On Distributed Memory Systems
http://blog.paralleluniverse.co/post/26909672264/on-distributed-memory

My Links
http://delicious.com/ajlopez/distributedcomputing

Distributed Computing: Links, News And Resources (4)

Previous Post
Next Post

Distributed RPC in Storm
https://github.com/nathanmarz/storm/wiki/Distributed-RPC

High Performance Network Applications in the Capital Markets
http://www.infoq.com/presentations/High-Performance-Network-Applications-in-the-Capital-Markets
Twitter Storm: Open Source Real-time Hadoop
http://www.infoq.com/news/2011/09/twitter-storm-real-time-hadoop

Bridge is a new RPC framework for building modular services and scalable cross-language applications.
https://www.getbridge.com/

Hadoop Beyond MapReduce, Part 1: Introducing Kitten
http://www.cloudera.com/blog/2012/06/hadoop-beyond-mapreduce-introducing-kitten/
This week, a team of researchers at Google will be presenting a paper describing a system they developed that can learn to identify objects, including the faces of humans and cats, from an extremely large corpus of unlabeled training data.

Google’s ‘brain simulator’: 16,000 computers to identify a cat
http://www.smh.com.au/technology/sci-tech/googles-brain-simulator-16000-computers-to-identify-a-cat-20120626-20zmd.html

ZooKeeper
http://zookeeper.apache.org/doc/trunk/zookeeperOver.html
A Distributed Coordination Service for Distributed Applications
Building a distributed concurrent queue with Apache ZooKeeper
http://www.cloudera.com/blog/2009/05/building-a-distributed-concurrent-queue-with-apache-zookeeper/
Celery: Distributed Task Queue
http://celeryproject.org/
Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well.

Storm
http://storm-project.net/
Storm is a free and open source distributed realtime computation system.

Stratosphere
https://www.stratosphere.eu/
Stratosphere is a DFG-funded research project and investigates “Information Management on the Cloud”.

Storm Tutorial
https://github.com/nathanmarz/storm/wiki/Tutorial

AtomizeJS
http://atomizejs.github.com/
A Distributed Software Transactional Memory implementation in JavaScript

pysage
http://code.google.com/p/pysage/
lightweight high-level message passing library supporting actor based concurrency

Life beyond Distributed Transactions:
an Apostate’s Opinion
http://www.ics.uci.edu/~cs223/papers/cidr07p15.pdf

Building a Hybrid Cloud at Canadian Pacific
http://www.infoq.com/presentations/Building-a-Hybrid-Cloud-at-Canadian-Pacific
Stuart Charlton presents Canadian Pacific’s strategy for a modern IT: adopting hybrid cloud, introducing Agile/Lean development, automating everything, distributed data systems, RESTful integration.

NetApp Case Study
http://www.infoq.com/presentations/NetApp-Case-Study
Kumar Palaniapan and Scott Fleming present how NetApp deals with big data using Hadoop, HBase, Flume, and Solr, collecting and analyzing TBs of log data with Think Big Analytics.

Caching and shared data
http://cprieto.com/2011/08/caching-and-shared-data/

Skynet: A Scalable, Distributed Service Mesh in Go
http://www.infoq.com/presentations/Skynet-A-Scalable-Distributed-Service-Mesh-in-Go
Brian Ketelsen introduces Skynet, a platform for polyglot, distributed and composable services that communicate with each other over RPC/JSON.

Distributed Systems with ZeroMQ and gevent
http://www.infoq.com/presentations/Distributed-Systems-with-ZeroMQ-and-gevent

Erlang Distribution Protocol
http://www.erlang.org/doc/apps/erts/erl_dist_protocol.html

External Term Format
http://www.erlang.org/doc/apps/erts/erl_ext_dist.html

Vert.x vs node.js simple HTTP benchmarks
http://vertxproject.wordpress.com/2012/05/09/vert-x-vs-node-js-simple-http-benchmarks/

Introducing DCell: actor-based distributed objects for Ruby
http://www.unlimitednovelty.com/2012/04/introducing-dcell-actor-based.html

davidfowl / CacheR
https://github.com/davidfowl/CacheR
A distributed cache implemented on top of SignalR

My Links
http://delicious.com/ajlopez/distributedcomputing

Keep tuned!

Angel “Java” Lopez
http://www.ajlopez.com
http://twitter.com/ajlopez

Programming Languages, Distributed Computing and Artificial Intelligence

It’s time to write a post, explaining my personal interest in some topics, like programming languages, messaging, distributed computing and artificial intelligence. English is not my mother tongue, so I’m afraid of not to be capable to express all what I want to transmit, but I hope this post will shade some light on my development activities.

I’m sure that most of you are interested in such topics. They are interesting, and studying and exploring them is a lot of fun. As software developers, we usually like to play with: writing a new programming language, implementing an existing one, building distributed applications, etc.

But in my case, there is a common ground for all these works. For example, programming languages. The current programming languages should be extended to support a better, with less friction, serialization of simple objects (without behavior, simple messages), to have distributed applications. There are two ways of doing it: extending the language with new syntax, or adding new libraries (see Scala and Akka as examples). So, in order to have a better understanding of what it is needed,  I wrote my own interpreted language (AjSharp), extended Smalltalk (AjTalk) to run distributed objects, writing agents and actors-like demos (AjAgents, and agents in AjSharp), distributed messaging (as in AjFabriq) and exploring JavaScript with Node.js (SimpleMessages, SimpleQueue, MultiNodes, SimpleRemote, SimpleActors… ), to have all in place to have an easy way of writing and running distributed applications. Ok, there are plenty of other projects that I could use. I wrote my own ones to practice and to better understanding of the underlying problems and opportunities. And to have my own fun

But, why distributed computing? I’m convinced that is the path to explore to have new kind of applications. Fundamentally, applications that solve complex problem, that can be tackled using parallel computation, and horizontal scalability. Not only Big Data processing. There are life beyond analyze tweets in real-time. The idea is to attack artificial intelligence problems, using commodity hardware.

That is the big background picture on my development landscape: flexible programming languages; messaging to support distributed application communications; distributed applications for the next generation of problems to solve.

Yes, the next frontier. Where no man has gone before

Keep tuned!

Angel “Java” Lopez
http://www.ajlopez.com
http://twitter.com/ajlopez

Distributed Computing: Links, News And Resources (3)

Previous Post
Next Post

http://en.wikipedia.org/wiki/Distributed_computing

Distributed computing is a field of computer science that studies distributed systems. A distributed system consists of multiple autonomous computers that communicate through a computer network. The computers interact with each other in order to achieve a common goal. A computer program that runs in a distributed system is called a distributed program, and distributed programming is the process of writing such programs

MilkyWay@Home
http://milkyway.cs.rpi.edu/milkyway/
Milkyway@Home uses the BOINC platform to harness volunteered computing resources, creating a highly accurate three dimensional model of the Milky Way galaxy using data gathered by the Sloan Digital Sky Survey. This project enables research in both astroinformatics and computer science.

Distributed Computing
http://distributedcomputing.info/

Our Grid
http://www.ourgrid.org/
The OurGrid Community comprises all users and developers of the OurGrid middleware. This middleware enables the creation of peer-to-peer computational grids, and since its first release, in December 2004, it has been used by hundreds of users to speed up the execution of Bag-of-Tasks applications¹.

BOINC
http://boinc.berkeley.edu/index.php
Open-source software for volunteer computing and grid computing.

Choosing BOINC projects
http://boinc.berkeley.edu/projects.php

About Einstein@Home
http://einstein.phys.uwm.edu/
Einstein@Home uses your computer’s idle time to search for weak astrophysical signals from spinning neutron stars (also called pulsars) using data from the LIGO gravitational-wave detectors, the Arecibo radio telescope, and the Fermi gamma-ray satellite.

Hadoop + HBase + Cygwin + Windows 7 x64
http://alans.se/blog/2010/hadoop-hbase-cygwin-windows-7-x64/

tim8dev / play-akka-pi
https://github.com/tim8dev/play-akka-pi
Pi Approximation distributed over the network using AKKA 2.0 Remote Actors

Migrating from Scala Actors to Lift Actors
http://blog.goodstuff.im/?/archives/96-Migrating-from-Scala-Actors-to-Lift-Actors.html

Avout
http://avout.io/
Distributed state in Clojure

Distributed concurrent applications in Clojure?
http://groups.google.com/group/clojure/browse_thread/thread/38924bdb1ab63c60/4a7a866c45dc2101?pli=1

Bloom: Disorderly Programming for a Distributed World
http://channel9.msdn.com/Events/Lang-NEXT/Lang-NEXT-2012/Bloom-Disorderly-Programming-for-a-Distributed-World
I will present Bloom, a programming language targeted at developers of complex cloud computing and distributed systems.  Bloom is a ‘disorderly’ language: it differentiates itself from most common programming languages by embracing rather than resisting the disorderly realities of distributed computing architectures.  Building on recent theoretical results including the CALM Theorem, Bloom supports a powerful new programming analysis framework for analyzing the correctness and consistency of distributed programs.

twitter / flockdb
https://github.com/twitter/flockdb
A distributed, fault-tolerant graph database

utajio / related
https://github.com/sutajio/related
Related is a Redis-backed high performance distributed graph database.

Distributed Evolutionary Algorithms in Python
http://pypi.python.org/pypi/deap/0.8

The Game of Distributed Systems Programming. Which Level Are You?
http://blog.incubaid.com/2012/03/28/the-game-of-distributed-systems-programming-which-level-are-you/

Getting Real About Distributed System Reliability
http://blog.empathybox.com/post/19574936361/getting-real-about-distributed-system-reliability

Erlang: Distributed OTP Applications
http://learnyousomeerlang.com/distributed-otp-applications

My Links
http://delicious.com/ajlopez/distributedcomputing

Keep tuned!

Angel “Java” Lopez
http://www.ajlopez.com
http://twitter.com/ajlopez

Distributed Computing: Links, News And Resources (2)

Previous Post
Next Post

Distributed OTP Applications | Learn You Some Erlang for Great Good!
http://learnyousomeerlang.com/distributed-otp-applications
Hey there, it appears your Javascript is disabled. That’s fine, the site works without it. However, you might prefer reading it with syntax highlighting, which requires Javascript!Although Erlang leaves us with a lot of work to do, it still provided a few solutions. One of these is the concept of …

Inaka Networks: Scaling Erlang
http://inakanetworks.com/blog/2011/10/07/scale-test-plan-simple-erlang-appl...
One of the most common reasons why people choose Erlang is to build highly scalable systems. And Erlang does a great job helping developers reach those goals. But creating a scalable system is not a matter of just writing it in Erlang. Here at Inaka we usually have complex systems written in Erlang …

Made of Bugs » Why node.js is cool (it’s not about performance)
http://blog.nelhage.com/2012/03/why-node-js-is-cool/
For the past N months, it seems like there is no new technology stack that is either hotter or more controversial than node.js. node.js is cancer! node.js cures cancer! node.js is bad ass rock star tech!. I myself have given node.js a lot of shit, often involving the phrase “explicit …

Robust Composition: Towards a Unified Approach to Access Control and Concurrency…
http://erights.org/talks/thesis/index.html
A dissertation submitted to Johns Hopkins University in conformity with the requirements for the degree of Doctor of Philosophy. Baltimore, Maryland, May, 2006Copyright © 2006, Mark Samuel Miller. All rights reserved. Permission is hereby granted to make and distribute verbatim copies of this …

Open Source
Distributed Capabilities
http://erights.org/index.html

Cloudjs
http://dan.harabagiu.net/cloudjs/
This module is build upon Node JS and provides for the user the following features
A network distributed event system. Similar to node JS standard event system
A process pool, where objects can be added and ran at a periodic interval a predefined functions.
An auto-balancing system, that migrate objects in the process pool, from one running instance to another, based on the load of each instance.

Sensei DB
http://www.senseidb.com/
Open-source, distributed, realtime, semi-structured database

Amazon DynamoDB – a Fast and Scalable NoSQL Database Service Designed for Intern…
http://www.allthingsdistributed.com/2012/01/amazon-dynamodb.html
Today is a very exciting day as we release Amazon DynamoDB, a fast, highly reliable and cost-effective NoSQL database service designed for internet scale applications. DynamoDB is the result of 15 years of learning in the areas of large scale non-relational databases and cloud services. Several …

Galois Tech Talk (2 of 3 next week!): Model-based Code Generation and Debugging …
http://calagator.org/events/1250461800
Design and implementation of distributed systems often involve many subtleties due to their complex structure, non-determinism, and low atomicity as well as occurrence of unanticipated physical events such as faults. Thus, constructing correct distributed systems has always been a challenge and …

How hadoop works.
http://dwellman.tumblr.com/post/15460756943/how-hadoop-works

Welcome to the Jungle « Sutter’s Mill
http://herbsutter.com/welcome-to-the-jungle/
In the twilight of Moore’s Law, the transitions to multicore processors, GPU computing, and HaaS cloud computing are not separate trends, but aspects of a single trend – mainstream computers from desktops to ‘smartphones’ are being permanently transformed into heterogeneous supercomputer clusters. …

Richard Jones | Anti-RDBMS: A list of distributed key-value stores
http://www.metabrew.com/article/anti-rdbms-a-list-of-distributed-key-value-stores
Whatever your reasons, there are a lot of options to chose from. At Last.fm we do a lot of batch computation in Hadoop, then dump it out to other machines where it’s indexed and served up over HTTP and Thrift as an internal service (stuff like ‘most popular songs in London, UK this week’ etc).

YOW! 2011: Steve Vinoski – Riak Core, Erlang and Frisbee Freestyle | Charles | C…
http://channel9.msdn.com/Blogs/Charles/YOW-2011-Steve-Vinoski-Riak-Core-Erl...
Steve Vinoski is an architect at Basho Technologies in Cambridge, MA, USA. He is a senior member of the IEEE and a member of the ACM. Steve is regarded as an expert in the areas of middleware and distributed computing systems, topics for which he has authored or co-authored over 80 articles, …

Fractals in Clojure – Distributed Buddhabrot Fractal Using ClojureScript (by Nur…
http://nakkaya.com/2011/12/15/fractals-in-clojure-distributed-buddhabrot-fractal-using-clojurescript/
This one got started because I wanted a large Buddhabrot image on my wall. A large good looking image takes a long time to render, now that we have ClojureScript I thought easiest way to distribute the calculation among machines in the house would be to compile to JavaScript since I’ve already …

Models for distributed parallelism | Lambda the Ultimate
http://lambda-the-ultimate.org/node/4405
I’ve been reading left and right (including on this forum) looking for models of parallel computing that are relevant to what I’m dealing with, and frankly not finding much. I get the impression that most theory of parallel computing is about shared memory, and so a bunch of old ideas for SMPs gets …

Awelon.org
http://awelon.org/
Awelon aims to be a secure programming language for open, distributed systems programming – a domain where challenges include disruption, latency, adversaries and security concerns, network partitioning, efficiency, scalability, activity spikes (i.e. the so-called slashdot-effect and DDOS attacks), …

Clojure/core — Introducing Avout: Distributed State in Clojure
http://clojure.com/blog/2011/11/29/avout.html
Today we are releasing Avout, which brings Clojure’s in-memory model of state to distributed application development by providing a distributed implementation of Clojure’s Multiversion Concurrency Control (MVCC) STM along with distributable, durable, and extendable versions of Clojure’s Atom and …

InfoQ: Distributed STM – A New Programming Model for the Cloud
http://www.infoq.com/presentations/Distributed-STM
In case you are having issues watching this video, please follow these simple steps to help us investigate the issue: 1. Right click on the video player and select Copy log2. Paste the copied information in an email to video-issue@infoq.com (clicking this link will fill in the default details in …

devdazed/partition.io – GitHub
https://github.com/devdazed/partition.io
P2P Distributed Workload for NodeJS

InfoQ: Distributed Cache as a NoSQL Data Store?
http://www.infoq.com/news/2011/11/distributed-cache-nosql-data-sto#.TrgCGIy...
NoSQL data stores offer alternative data storage options for non-relational data types like document based, object graphs, and key-value pairs. Can a distributed cache be used as a NoSQL store? Greg Luck from Ehcache wrote about the similarities between a distributed cache and a NoSQL data store. ..

Distributed Podcast
http://distributedpodcast.com/
In this episode we interviewed David Fowler and Damian Edwards who have created a great project called SignalR. This project is a signaling (or messaging) library which can be used to establish long-running connections between the browser and web server.Side note: This episode has been a long time …

My Links
http://delicious.com/ajlopez/distributedcomputing

Distributed Computing: Links, News and Resources (1)

Next Post

I just realized I never published a list of links of one of my preferred topic. This is the first post:

Storm – a real time Hadoop like system in Clojure
http://pseudofish.com/blog/2011/09/26/storm-a-real-time-hadoop-like-system-in-clojure/

Hadoop Programming Challenge
http://bigdatauniversity.com/web/hadoop-programming-challenge.php

The Design of Distributed Applications
http://groups.google.com/group/the-design-of-distributed-applications

Thoughts around REST, DDD, and CQRS: Models, Queries, and Commands
http://groups.google.com/group/the-design-of-distributed-applications/t/f2295ec60ef87c77

Akka 2.x roadmap…
https://docs.google.com/document/pub?id=1CMz_MEQA8oPcGw9oaFdq_KYYFB_5qZjsDYYwuXfZhBU&pli=1

Chess@home
http://chessathome.org/

Welcome to Apache Pig
http://pig.apache.org/
Apache Pig is a platform for analyzing large data sets that consists of a high-level language for expressing data analysis programs, coupled with infrastructure for evaluating these programs. The salient property of Pig programs is that their structure is amenable to substantial parallelization, which in turns enables them to handle very large data sets.

Welcome to Hama project
http://incubator.apache.org/hama/
Apache Hama is a distributed computing framework based on BSP (Bulk Synchronous Parallel) computing techniques for massive scientific computations, e.g., matrix, graph and network algorithms. It was inspired by Google’s Pregel, but different in the sense that it’s purely BSP and common model, not just for graph.

InfoQ: Things Break, Riak Bends
http://www.infoq.com/presentations/Things-Break-Riak-Bends

HPCC Systems | Open-source. Fast. Scalable. Simple
http://hpccsystems.com/
HPCC (High Performance Computing Cluster) is a massive parallel-processing computing platform that solves Big Data problems. The platform is now Open Source!

SmartFrog
http://wiki.smartfrog.org/wiki/display/sf/SmartFrog+Home
SmartFrog is a powerful and flexible Java-based software framework for configuring, deploying and managing distributed software systems.

Mesos: Dynamic Resource Sharing for Clusters
http://www.mesosproject.org/
Mesos is a cluster manager that provides efficient resource isolation and sharing across distributed applications, or frameworks. It can run Hadoop, MPI, Hypertable, Spark (a new framework for low-latency interactive and iterative jobs), and other applications. Mesos is open source in the Apache Incubator.

Dryad – Microsoft Research
http://research.microsoft.com/en-us/projects/Dryad/

InfoQ: Secure Distributed Programming on ECMAScript 5 + HTML5
http://www.infoq.com/presentations/Secure-Distributed-Programming

Ceph as a scalable alternative to the Hadoop Distributed File System
http://www.usenix.org/publications/login/2010-08/openpdfs/maltzahn.pdf

Data-driven Apps With Microsoft Velocity Distributed Caching
http://msdn.microsoft.com/en-us/magazine/dd861287.aspx

Spark
http://www.spark-project.org/
Spark is an open source cluster computing system that aims to make data analytics fast — both fast to run and fast to write.

Distributed computing fallacies and REST
http://lostechies.com/jimmybogard/2011/05/27/distributed-computing-fallacies-and-rest/

Presentation Schedule // CS 525: Advanced Distributed Systems // Spring 2011
http://www.cs.uiuc.edu/class/sp11/cs525/sched.htm

InfoQ: Francesco Cesarini and Simon Thompson on Erlang
http://www.infoq.com/interviews/cesarini-thompson-erlang

TypeSafe
http://typesafe.com/
Scala and Akka are deployed in production at some of the largest web properties and financial institutions in the world, and run on the battle-tested Java runtime environment. Deploy with confidence.

Introducing Riak Core
http://blog.basho.com/2010/07/30/introducing-riak-core/

Actors: A Model of Concurrent Computation in Distributed Systems
http://dspace.mit.edu/handle/1721.1/6952

The Hadoop Distributed File System
http://storageconference.org/2010/Papers/MSST/Shvachko.pdf

InfoQ: Concurrency Control in Data Replication
http://www.infoq.com/articles/Concurrency-Control-Data-Replication

Build a distributed realtime tweet search system in no time. Part 1/2
http://sna-projects.com/blog/2011/02/build-a-distributed-realtime-tweet-search-system-in-no-time-part-12/

Windows Azure futures: Turning the cloud into a supercomputer
http://www.zdnet.com/blog/microsoft/windows-azure-futures-turning-the-cloud-into-a-supercomputer/8592

Episode 1: Distributed Systems Host Introductions
http://distributedpodcast.com/2010/episode-1-host-introductions

Distributed Podcast
http://distributedpodcast.com

Frangipani: A Scalable Distributed File System
http://www.systemswemake.com/blog/28/frangipani/

Systems We Make
http://www.systemswemake.com

Fault tolerance techniques for distributed systems
http://www.ibm.com/developerworks/rational/library/114.html

Swarm: A true distributed programming language
http://blog.locut.us/2008/10/07/swarm-a-true-distributed-programming-language/

MSDN Magazine: Distributed Apps
http://msdn.microsoft.com/en-us/magazine/gg232773.aspx

Scalable System Design Patterns
http://horicky.blogspot.com/2010/10/scalable-system-design-patterns.html
Load Balancer, Scatter and Gather, Result Cache, Shared Space, Pipe and Filter, Map Reduce, Bulk Synchronous Parallel, Execution Orchestrator

My Links
http://www.delicious.com/ajlopez/distributedcomputing

Angel “Java” Lopez
http://www.ajlopez.com
http://twitter.com/ajlopez

Socket.IO: Links, News And Resources (2)

Previous Post

Ceiboo Software Development
http://www.ceiboo.com/

LearnBoost / engine.io-client
https://github.com/learnboost/engine.io-client

How to deliver realtime web and mobile applications?
http://sebastianconcept.com/brandIt/how-to-deliver-realtime-web-and-mobile-applications

Node.js & WebSocket – Simple chat tutorial
http://martinsikora.com/nodejs-and-websocket-simple-chat-tutorial

Build a Node.js Chat Application with Socket.IO on a Windows Azure Cloud Service
http://www.windowsazure.com/en-us/develop/nodejs/tutorials/app-using-socketio/

nodeGame / nodegame-client
https://github.com/nodeGame/nodegame-client
Node.js and Socket.io part 1 Building a HTML5 game part 6
http://www.youtube.com/watch?v=KnQIkKHrdFY&feature=relmfu
Node.js and Socket.io part 3 Building a HTML5 game part 8
http://www.youtube.com/watch?v=hARvsRQpWVE&feature=relmfu

Node.js and Socket.io part 2 Building a HTML5 game part 7
http://www.youtube.com/watch?v=ptvEn3iYKVY&feature=related
NodeCamp Socket.IO Workshop with Guillermo Rauch
http://www.youtube.com/watch?v=Y1vC7S385x8&feature=related

BogotaJS: Guillermo Rauch – Socket.IO / Engine.IO
http://www.youtube.com/watch?v=e3FdQAiG3-o&feature=youtu.be

icetan / blox
https://github.com/icetan/blox
A terminal based multiplayer tetris clone

How to Build a Real-Time Chat Service with Socket.IO, Express, and the Azure SDK–Part 1: Setting Up
http://www.aaronstannard.com/post/2012/04/16/How-to-Build-a-Real-Time-Chat-Service-with-SocketIO-Express-and-the-Azure-SDKe28093Part-1.aspx

Multiplayer HTML5, Node.js, Socket.IO
http://gamedev.stackexchange.com/questions/13207/multiplayer-html5-node-js-socket-io

Does drupal or joomla allow a combination node.js and socket.io with php?
http://stackoverflow.com/questions/10712648/does-drupal-or-joomla-allow-a-combination-node-js-and-socket-io-with-php

Socket.io and redis store
https://groups.google.com/forum/?fromgroups=&hl=en#!topic/socket_io/gymBTlruYxs
Socket.IO RedisStore and xhr-polling
http://stackoverflow.com/questions/10593616/socket-io-redisstore-and-xhr-polling

Examples in using RedisStore in socket.io
http://stackoverflow.com/questions/9267292/examples-in-using-redisstore-in-socket-io

RedisStore and rooms with Socket.IO
http://www.ranu.com.ar/search/label/socket.io

My Links
http://delicious.com/ajlopez/socketio

Keep tuned!

Angel “Java” Lopez
http://www.ajlopez.com
http://twitter.com/ajlopez

Socket.IO: Links, News and Resources (1)

Next Post

Today, I will present Node.js, Express and Socket.IO at UDADev 2012, Universidad de Azuay, Ecuador. My first list of links about Socket.IO:

Social Chat
http://chat.skychen.com/

Building a Real-Time Location-Based Urban Geofencing Game with Socket.io,
Redis, Node.js and Sinatra Synchrony
https://geoloqi.com/blog/2011/09/building-a-real-time-location-based-urban-geofencing-game-with-socket-io-redis-node-js-and-sinatra-synchrony/

Re-using Backbone.js Models on the server with Node.js and Socket.io to
build real-time apps
http://blog.andyet.com/2011/feb/15/re-using-backbonejs-models-on-the-server-with-node/

LearnBoost / engine.io
https://github.com/LearnBoost/engine.io

Building a multiplayer HTML5 game using Node, Socket.io
http://www.nodejs-news.com/nodejs-tech/Building-Multiplayer-Game-HTML5-Node-Socketio/

fent / socket.io-clusterhub
https://github.com/fent/socket.io-clusterhub

MMO server/client test
http://www.youtube.com/watch?v=uOY0bzuJs6c

NodeJS Pokemon: MMO iPad, iPhone, Chrome, Safari Card Game
http://www.youtube.com/watch?v=Kcsjb1Isweo

HTML5 Canvas MMORPG Games using Node.js and Nowjs
http://nodegames.blogspot.com.ar/

Websockets via Node.js/socket.io
http://social.msdn.microsoft.com/Forums/en-US/windowsazureconnectivity/thread/74308a54-e06b-479d-af7f-7ad0fef498e1

socket.io and Express. Tying it all together
http://www.danielbaulig.de/socket-ioexpress/

ceiboo software development
http://www.ceiboo.com/

Communicating with a socket.io server via c#
http://stackoverflow.com/questions/6499861/communicating-with-a-socket-io-server-via-c-sharp

c# alternative to socket.io for node.js
http://stackoverflow.com/questions/5952042/c-sharp-alternative-to-socket-io-for-node-js

Asynchronous scalable web applications with real-time persistent
long-running connections with SignalR
http://www.hanselman.com/blog/AsynchronousScalableWebApplicationsWithRealtimePersistentLongrunningConnectionsWithSignalR.aspx

gerad / lazeroids-node
https://github.com/gerad/lazeroids-node/

deserat / sock-drawer
https://github.com/deserat/sock-drawer/
Basically a multi-user whiteboard with socket.io, node.js, and canvas. Not
serious just Playing

Speed Limit of PaaS – 64K TCP Ports
http://blog.mudynamics.com/2011/08/26/speed-limit-of-paas-64k-tcp-ports/

Good beginners tutorial to socket.io?
http://stackoverflow.com/questions/4094350/good-beginners-tutorial-to-socket-io

nodechat.js – Using node.js, backbone.js, socket.io, and redis to make a
real time chat app
http://fzysqr.com/2011/02/28/nodechat-js-using-node-js-backbone-js-socket-io-and-redis-to-make-a-real-time-chat-app/

node.js & socket.io fun
http://till.klampaeckel.de/blog/archives/133-node.js-socket.io-fun.html

Getting Your Feet Wet With node.js and socket.io – Part 1
http://thecoffman.com/2011/02/21/getting-your-feet-wet-with-node.js-and-socket.io/

Websockets everywhere with Socket.IO
http://howtonode.org/websockets-socketio

My Links
http://delicious.com/ajlopez/socketio

Keep tuned!

Angel “Java” Lopez
http://www.ajlopez.com
http://twitter.com/ajlopez

AjFabriq on NodeJs (Part 3) A Distributed Simple Application

Previous Post

Lets run our “killer” application (a simple counter) in two node. At the repo, under examples\numbers, I have an appserver.js program:

It’s similar to my local example. The difference is that the top message processor is listening using a port:

/**
 * Host.
 */
var host = ajfabriq.createLocalHost();
/**
 * Application configuration.
 */
 
var app = host.createProcessor('numbers', 'application');
var node = app.createProcessor('processor', 'node');
node.on('decrement', function (message) {
	console.log("Processing number " + message.number);
	
	if (message.number <= 1) {
		console.log("End Processing");
		return;
		}
		
	var number = message.number-1;
	
	this.post({ action: 'decrement', number: number });
});
host.listen(3000);
host.process({ application: 'numbers', node: 'processor', action: 'decrement', number: 10 });

In this code, I’m using ajfabriq.createLocalHost() instead .createProcessor(). And host.listen(3000) to accept messages from other nodes. 
I run another program: appclient.js. It has the same local processors:
/**
 * Application configuration.
 */
 
var app = host.createProcessor('numbers', 'application');
var node = app.createProcessor('processor', 'node');
node.on('decrement', function (message) {
	console.log("Processing number " + message.number);
	if (message.number <= 1)
		return;
		
	var number = message.number-1;
	
	this.post({ action: 'decrement', number: number });
});

But it connects to the first server, and post a new message:
var socket = new net.Socket();
socket.connect(3000, 'localhost',
	function() {
		host.connect(new ajfabriq.Channel(socket), true);
		socket.write(JSON.stringify({name : 'ajfmessage', message: { application: 'numbers', node: 'processor', action: 'decrement', number: 10 }}));
	}
);
	

ajfabriq.Channel is the bidirectional channel between two ajfabriq servers. 
This server output:
 
Note the interchange of message between the two servers, at the beginning. They are informing their local processors, so each server knows if a message could be processed by another server.
The first server reaction:
 
Some of the numbers are processed by the second server, and the others are routed to the first server. The routing is a simple random choice in this demo. LocalHost objects have a new .post message:
LocalHost.prototype.post = function (message) {
	var hosts = [ this ];
	
	for (var remote in this.remotes) {
		if (this.remotes[remote].accepts(message)) {
			hosts.push(this.remotes[remote]);
		}
	}
	var n = Math.floor(Math.random() * hosts.length);
	
	hosts[n].process(message);
};

Next steps: better routing, improve socket communication (large JSON messages detection and split), logging, more sample apps.
Keep tuned!
Angel “Java” Lopez

  
http://www.ajlopez.com 
http://twitter.com/ajlopez
	

AjFabriq on NodeJs (Part 2) A local Simple Application

Previous Post
Next Post

Lets explore how to use AjFabriq on NodeJs. Here is a simple application:

https://github.com/ajlopez/AjFabriqJs/tree/master/examples/numbers

It implements the ultimate killer application: it receives a message with a number, and post a message with the number less one . Let’s see how the application is defined:

/**
 * Module dependencies.
 */
 
var ajf = require('ajfabriq');

I included c:\Git in my NODE_PATH environment variable, and there is a c:\Git\ajfabriq containing my git local repo under development. You can clone the repo in your node_modules folder if you are using NodeJs 5.x (see Playing With NodeJs (1) Running on Windows (and Azure)) .
AjFabriq defines an object that exposes some methods. This is the way to create a message processor that is local, and it is not exposed to other servers (a distributed example in the repo, to review in an upcoming post):
/**
 * Host.
 */
var host = ajf.createProcessor();

Now, a message processor (see previous post) can be a composite. The aprocessor.createProcessor creates a new processor and adds it to the parent processor:
/**
 * Application configuration.
 */
 
var app = host.createProcessor('numbers', 'application');
var node = app.createProcessor('processor', 'node');

The first processor, app, will accept and process messages with property “application” having "numbers” as value. Its child processor node will process message with additional property “node” with value “processor”. In this way, we can define a tree of message processors. The message properties and their values are the routing information, so each message will be send to the appropriate message processor.
But, how to define the leaf processor behavior? To be aligned to NodeJs async processing, each processor inherits from an EventEmitter. Since the previous post, I defined the Processor as a “subclass” of process.EventEmitter in AjFabriq code:
var EventEmitter = process.EventEmitter;
// ...
function Processor(name, kind)
{
	this.name = name;
	this.kind = kind;
	this.processors = [];
}
Processor.prototype.__proto__ = EventEmitter.prototype;

Now, we can define the leaf processor behavior:
node.on('decrement', function (message) {
	console.log("Processing number " + message.number);
	
	if (message.number <= 1) {
		console.log("End Processing");
		return;
		}
		
	var number = message.number-1;
	
	this.post({ action: 'decrement', number: number });
});
The message processing detects an action property, and raise the corresponding event. Note the emit method invocation in AjFabriq code:
Processor.prototype.process = function (message)
{
	if (this.processors == null || this.processors.length == 0) {
		this.emit(message.action, message);
		return;
	}
	
	for (var processor in this.processors)
		if (this.processors[processor].accepts(message)) 
		{
			this.processors[processor].process(message);
		}
}
 
At the end of my “killer” app sample, a message is sent to the top processor:
host.process({ application: 'numbers', node: 'processor', action: 'decrement', number: 10 });

The output:
 
But wait! Do you check the ‘decrement’ code I commented above? There is a:
this.post({ action: 'decrement', number: number });
No application, node properties!! Yes, it's right. If you post a message from a processor (node in this case), the parent processors fill the missing properties. That is, application property is set to “numbers”, and node property is set to “processor”. If you want to sent a message to other application, you can explicitly set the corresponding property in your new message. Note that the recommended practice is: don't change the original message. Maybe it could be processed by other processors.
For upcoming posts: new apps, a distributed example, more fun!  
Keep tuned!
Angel “Java” Lopez

  
http://www.ajlopez.com 
http://twitter.com/ajlopez