Last January, I update my AjMessages project to support communication using DSS/CCR and Windows Communication Foundation. AjMessages is a program that can run in a distributed way, sending asynchronous messages from one logical node to another. The logical nodes can be hosted in one or more physical machines. More info about AjMessages at:
You can download the source code from
I had the idea of write a distributed message processor. Years ago, I wrote my first attempt, named AjServer
Hacia el AjServer (Spanish Post)
After that project, I met Fabriq ideas and implementation, where there are logical message handler, that can be distributed in a transparent way, via configuration:
In 2007, I wrote the first version of AjMessages, as a proof of concept: I could reproduce most of the Fabriq ideas, trying to raise some abstraction, in order to have a little more flexibility. Fabriq was more SOA-oriented. AjMessages is oriented to arbitrary message processing, to have a grid-like application.
This is the current AjMessages solution:
There is core project, and two transport projects: one for WCF support, and another to DSS/CCR (Microsoft Robotics technology). If you don’t have DSS/CCR, you can remove the related projects. The core AjMessages is free of dependencies on such technologies. AjMessages.SampleApp contains simple handler: a decrement handler, that takes a message with an integer, and produces a new message with a new decremented integer. AjMessages.WebCrawler implements the message and handlers that make a distributed web crawler application. You can run it in only one machine or in many hosts.
Each AjMessage message has:
Body: an arbitrary payload
Headers: additional information, key-value.
Action: describing the target of the message, using pattern Application/Node/Action
(more detail in the mentioned post AjMessages- a message processor)
Let’s explore the distributed web crawler example. This application is composed by nodes, that can be orchestrated to visit a link, and gets from that initial page all the links and related pages, up to a level. Logically, it can be described as:
The first messages is send to the node Controller, action Dispatch. It contains the initial page to visit. The message is enriched, and passed to node Controller, action Resolve. It’s in charge to keep a list of visited page, and to control the deep of processing. If the link is aproved, then a message is send to node Downloader, action Download. The content of the page is downloaded, added to the message, and forwarded to node Harvester, action Harvest. This action analyze the content of the page, and emits many message, one for each link found in it. The recipient of the new links is the node Resolver, action Resolve. So, the process continues up to visit all pages (with a limit on the deep of processing).
This application can be described using an XML configuration file, example:
<?xml version="1.0" encoding="utf-8" ?> <AjMessages> <Application Name="WebCrawler"> <Node Name="Dispatcher"> <Handler Name="DispatcherHandler" Type="AjMessages.WebCrawler.Handlers.Dispatcher, AjMessages.WebCrawler"/> <Action Name="Dispatch" Handler="DispatcherHandler"/> </Node> <Node Name="Harvester"> <Handler Name="HarvesterHandler" Type="AjMessages.WebCrawler.Handlers.Harvester, AjMessages.WebCrawler"/> <Action Name="Harvest" Handler="HarvesterHandler"/> </Node> <Node Name="Downloader"> <Handler Name="DownloaderHandler" Type="AjMessages.WebCrawler.Handlers.Downloader, AjMessages.WebCrawler"/> <Action Name="Download" Handler="DownloaderHandler"/> </Node> <Node Name="Controller"> <Handler Name="ResolverHandler" Type="AjMessages.WebCrawler.Handlers.Resolver, AjMessages.WebCrawler"/> <Action Name="Resolve" Handler="ResolverHandler"/> </Node> </Application> </AjMessages>
An application is composed by nodes. Each node can be viewed as a “logical class”. Each node can handle Actions, that are the targets of messages. An action can be composed by one or more steps, of message handler. This is the extensibility point of the application: you must provided the steps, the message handler, and write a configuration file to orchestrate the message processing.
But one thing is the application, and another its distribution over physical machines. You can have two hosts, and install different logical nodes in each one:
In this diagram, the Dispatcher and Resolver are in one host, and the Downloader and Harvester in the other one. But you can put harversters in each host, or in twenty machines. It’s up to you how to distribute the load of the work. When a message is sent to a target, AjMessage forwards it to an appropiate host, that have a node capable of attending the message action.
The distribution of logical nodes into physical host is defined via configuration, an example:
<?xml version="1.0" encoding="utf-8" ?> <AjMessages> <Host Name="Server1" Address="http://localhost:50002/AjMessages" Activate="true"> <Application Name="AjMessages"> <Node Name="Administration"/> </Application> <Application Name="WebCrawler"> <Node Name="Controller"/> <Node Name="Dispatcher"/> <Node Name="Harvester"/> <Node Name="Downloader"/> </Application> </Host> <Host Name="Server2" Address="http://localhost:50003/AjMessages"> <Application Name="AjMessages"> <Node Name="Administration"/> </Application> <Application Name="WebCrawler"> <Node Name="Dispatcher"/> <Node Name="Harvester"/> <Node Name="Downloader"/> </Application> </Host> </AjMessages>
Running the Web Crawler example
You can test the program, launching two host, in the same machine. Run the console program AjMessage.Console. Then enter the command
This command launches a second host. Now, you are ready to configure the two hosts. In the first one, enter:
This file loads the initial AjMessage application, and define a WCF endpoint to listen messages. Then, enter:
This command loads the Web Crawler node definitions (no deploy info yet). The third command define the distribution in hosts:
Now, back to second host console. Enter the corresponding commands:
You are ready to launch the first web crawling. Go to first console, and enter:
send WebCrawler/Dispatcher/Dispatch http://ajlopez.zoomblog.com
(that is my Spanish non technical blog… ;-)
The web crawling begins to work, in both consoles. A typical view on first console:
I should fix some problems in DSS/CCR tranport (I could run a simpler app, but not the web crawler). Two points to implements:
- Remote configuration of nodes
- Distribution of message handlers bits to remote nodes
- One configuration for all the nodes (now, each node is configured separately)
I began to experiment with a more abstract way of processing message, that should implement uses cases like:
- A simple hello world
- Decrement example, as in AjMessages sample app (see AjMessage.SampleApp)
- Distributed web crawler
- Enterprise Service Bus-like app
My first steps at:
But the project is still in its infancy. Keep tuned!