Angel \”Java\” Lopez on Blog

October 8, 2008

Distributed applications with AjMessages using DSS/CCR

Well, it’s only a project. The sample application is minimal. But it’s the evolution of my previous work with AjMessages via Windows Communication Foundation:

AjMessages- a message processor

Last year, I wrote that example, and another one that uses DSS/CCR. Only the first was published, now, I’m reviewing my DSS/CCR implementation, mounted on Microsoft Robotics technology. Currently, you can download the code from CodePlex at:

http://www.codeplex.com/ajmessages

Both examples were based on Fabriq project ideas, trasposed to new technologies. Recently, Arvindra Sehmi posted about a new version of Fabriq, named Fabriq4Dss, that is using DSS/CCR in a more powerful ways. He prepared a presentation for JAOO, read more at:

JAOO 2008 Presentation

Keep tuned!

The solution

It’s composed by three projects:

The first one is a core implementation, that is independent of the transport to use for distributed messages:

The second project contains a simple message handler to use in the demo. The third project is the DSS service component to use as a host node:

 

For a more detailed description of application, message, hosts, read my previous post:

AjMessages- a message processor

The application

A configuration XML file define the applications to run. An application has:

  • Nodes: a collection of actions to process.
  • Handlers: objects to process a message. A handler can be a composite of other handlers.
  • Actions: named entry points to a handler

An example application definition (from AjMessages.SampleApp\Configurations\Server1.xml):

<?xml version="1.0" encoding="utf-8" ?> <AjMessages> <Application Name="AjMessages"> <Node Name="Administration"> <Handler Name="ConfigureHandler" Type="AjMessages.ConfigureHandler, AjMessages"/> <Handler Name="PrintHandler" Type="AjMessages.PrintHandler, AjMessages"/> <Handler Name="PrintMessageHandler" Type="AjMessages.PrintMessageHandler, AjMessages"/> <Handler Name="ConfigurePipeline"> <Handler Name="PrintHandler"> <Property Name="Text" Value="Reconfiguring server..."/> </Handler> <Handler Name="ConfigureHandler"/> </Handler> <Action Name="Configure" Handler="ConfigurePipeline"/> </Node> </Application> <Application Name="App1"> <Node Name="Node1"> <Handler Name="PrintMessageHandler" Type="AjMessages.PrintMessageHandler, AjMessages"/> <Handler Name="PostHandler" Type="AjMessages.PostHandler, AjMessages"/> <Handler Name="DecrementHandler" Type="AjMessages.SampleApp.Handlers.DecrementHandler, AjMessages.SampleApp"/> <Handler Name="Pipeline1"> <Handler Name="DecrementHandler"/> <Handler Name="PostHandler"> <Property Name="Action" Value="App1/Node1/Process"/> </Handler> </Handler> <Action Name="Process" Handler="Pipeline1"/> </Node> <Node Name="Node2"/> </Application> <Host Name="Server1" Activate="true"> <Application Name="AjMessages"> <Node Name="Administration"/> </Application> <Application Name="App1"> <Node Name="Node1"/> <Node Name="Node2"/> </Application> </Host> <Host Name="Server2" Address="http://localhost:50002/ajmessages/node"> <Application Name="AjMessages"> <Node Name="Administration"/> </Application> <Application Name="App1"> <Node Name="Node1"/> <Node Name="Node2"/> </Application> </Host> </AjMessages>

Running the example

Load the solution in Visual Studio 2008 (you must have Microsoft Robotics Developer Studio, a CTP version of this year, and you must configure the output path of the projects to point to your MRDS directory). Running the DSS project lunch a DSS Host that shows a controller window:

The host is running on ports 50000/50001. Configure the host using the AjMessages.SampleApp\Configurations\Server1.xml.

You can launch a second host from DSS prompt:

bin/dsshost /p:50002 /t:50003
m:<pathto>\Source\AjMessages.DssServices\AjMessagesDssServices.manifest.xml

A second form controller appears:

Use the other config file AjMessages.SampleApp\Configurations\Server2.xml and port 50002.

You can send a message to the sample app. The message is only an integer, that is decremented in each node. The decrement node is installed in the two servers, so the message travels from one to another server.

Next steps

I want to support an arbitrary object, serialized in DSS, and uses WCF as a transport, too. The previous post describes the WCF implementation.

Angel “Java” Lopez
http://www.ajlopez.com/en

September 24, 2008

Microsoft Robotics in enterprise applications

I read in Arvindra Sehmi’s blog that Tyco Software House implemented a system using CCR (Concurrent and Coordination Runtime), the library inside Microsoft Robotics. In his article ARCast.TV – Nice Example of CCR Adoption in the Enterprise, Arvindra shows the video where Tyco dev explain how they can process a big load of messages, using asynchronous processing.

ARCast.TV – Stephen Tarmey of Tyco on adopting Robotics Studio CCR for High Performance Async IO
 

I would like something more specific about the implementation of their system, but it’s interesting to see that Microsoft Robotics is used in enterprise applications.

Recently, Arvindra have commented about using DSS/CCR beyond robotics:

DSS Use Cases in the Enterprise

I don’t have more info about those use cases, I hope there will be soon more details on those works.

Last year, I wrote and publish an example, AjMessage, that interchanges async messages, using WCF as the communication between two nodes in a grid:

AjMessages- a message processor

At the same time, I wrote a version that uses CCR and DSS, that I’ve never published. I plan to polish and publish the code, to show that we can use CCR internally, to enqueue messages in process, and DSS to send message to other nodes. The core of the system could be hosted in a WCF application or in a DSS hosts. If we have more nodes in the grid, the application should run without recoding. I could write the web crawler, as a proof of concept app; currently it is using DSS/CCR/VPL:

Web Crawler example using DSS (Decentralized Software Services)

But VPL is limited to distribute jobs to previously known machine. But let imagine we have more available machines  at any moment. We could discover the new machines, and send the messages with job information to any node in the grid, so the message would be processed by any machine that accepts it. I wrote new examples, managing multiple machines in runtime:

Distributed Agents and Fractals using DSS/VPL

Distributed Agents using DSS/VPL

Now, in the latests bits of Microsoft Robotics (July CTP of Microsoft Robotics Developer Studio 2008), there is support to DSS Node discovery using UPnP (Universal Plug and Play).

In my opinion, the async messaging model in a grid implementation can be used in many creative ways. It’s an interesting field to explore.

Angel “Java” Lopez
http://www.ajlopez.com/en

June 22, 2008

Distributed Agents and Fractals using DSS/VPL

Last week I wrote the base of an application running distributed agents over DSS/VPL, interchanging arbitrary messages, with automatic load balancing. You can read the details at:

Distributed Agents using DSS/VPL

Today, I extended the example with a new project, Fractal:

You can download from my Skydrive.

It has two DSS Service Components, one is the Calculator: it calculates a sector of the Mandelbrot fractal. The other component is the Renderer, that has a form to control and show the results of the calculation. The message that has the info of the sector to calculate is:

public class SectorInfo : MessagePayload { public double RealMinimum { get; set; } public double ImgMinimum { get; set; } public double Delta { get; set; } public int FromX { get; set; } public int FromY { get; set; } public int Width { get; set; } public int Height { get; set; } public int MaxIterations { get; set; } public int MaxValue { get; set; } }

Another class is the message that returns the calculation:

public class Sector : MessagePayload { public int FromX { get; set; } public int FromY { get; set; } public int Width { get; set; } public int Height { get; set; } public int[] Values { get; set; } }

The calculator splits the sector to calculate, if it too big. It can calculate all in only one step, but it is interesting to split the sector to generate more messages:

 

private void Calculate(AgentMessage msg) { LogInfo("Entering Calculator with Action: " + msg.Action); SectorInfo sectorInfo = (SectorInfo) msg.Payload; LogInfo(String.Format("X {0} Y {1} Width {2} Height {3}", sectorInfo.FromX, sectorInfo.FromY, sectorInfo.Width, sectorInfo.Height)); if (sectorInfo.Width > 100 && sectorInfo.Height > 100) SplitSector(sectorInfo); else CalculateSector(sectorInfo); }

As in the previous example, you can run this from a VPL diagram, FractalVpl1:

There is only one renderer, and two calculators agents. If you launch this VPL program, a window appears. This is the initial window content (after pressing the Calculate button):

You can drag and move the mouse to select a new region, or use the buttons to zoom in and out. Reset backs to the initial position. You can resize the window and invoke Calculate again. New colors button changes the color palette in use.

There is another VPL program, FractalVpl2, that it can be used to run the same example in a distributed way. It has a diagram with two AgentHosts:

and two nodes:

You must compile the VPL example and run as distributed nodes, using the rundeployer.cmd (see my previous post for details).

These some of the drawings the system produces:

 

Enjoy!

Angel “Java” Lopez
http://www.ajlopez.com/en

June 21, 2008

Messages everywhere

Recently, I wrote a post about Message Passing Interface:

Message Passing Interface, CCR, DSS, and Pure MPI.NET

I used to pass message between agents in my example of a web crawler:

Distributed Agents using DSS/VPL

The passing of messages between components, agents, objects, is a feature that deserves more study. I guess we could write new kinds of applications, using one-way message passing, so we could abandon the call-a-method current way of doing things. Let’s explore first, some message passing applications (not only one-way style).

In the Wikipedia entry about Message Passing:

http://en.wikipedia.org/wiki/Message_passing

we can read Alan Kay opinion:

Alan Kay has suggested that he placed too much emphasis on objects themselves and not enough on the messages being sent between them. Message passing enables extreme late binding in systems.

If you develop systems with Smalltalk, Self or alikes, you’ll notice that the message is a first citizen, in many implementations, a full object, not only a way of call a method.

There is another place for use message passing. For 25 years, QNX operating systems uses the message passing paradigm to run a real time kernel.

I found this interview at Dr. Dobb’s to Sebastien Marineau-Mes, VP of engeneering at QNX:

Real Time OS in Today s World

Sebastien talks about the use of QNX in current market, and the challenge that multi-core machine could create on legacy code.

Remember: all the QNX kernel is based on message passing, although its messages are not one-way, and the passing is optimized to not incurr in loss of performance (more details, see QNX at Wikipedia). I see that many of these challenges and opportunities could be translated to the use, not only to multi-core, but to “multi-machines” in a grid. There many forces that are conspiring to bring these topics to current arena:

- We have a better understanding of agents, message passing and other ideas

- Normal hardware is cheap

- Each year, there are more application with higher level needs of scalabity (the user base will be the whole Internet in the near future, for non-trivial app)

- Many application must interact with other application in the same enterprise or in the net, and messages are an effective way to do that.

- In order to face that challenges, we must begin to abandon n-tier-only apps, to a cloud, grid or “something alike” schema.

I could imagine languages and technologies, based on message passing features. That is one of the reasons I’ve been involved exploring simples and crazy ideas with AjMessages, AjAgents, and Distributed Software Services during last months. I hope to write more about these subjects:

- Another example of distributed agents using DSS/VPL

- AjMessages ported to DSS (I had an implementation last year, but I never published it, I published the WCF only version)

- One-way messages implemented as primitives in AjTalk (Smalltalk-like interpreter)

- Deferred/Concurrent/LazyEvaluation/SomethingAlike implemented in AjBasic (using CCR?)

- Blog about a better-finished app running in a grid of DSS hosts (my team was working hard on this, last months).

Incidentally, you can read more about use cases applied in the real world using CCR/DDS, in Arvindra Sehmi’s post:

CCR/DSS Use Cases in the Enterprise

So many ideas…. only one life…. Should I begin to parallized myself? ajlopez-in-a-grid…. ;-)

Angel “Java” Lopez
http://www.ajlopez.com/en

June 15, 2008

Distributed Agents using DSS/VPL

In this post, I explore some ideas to implement distributed agents, leveraging the features from Decentrilized Software Services (DSS) and Visual Programming Language (VPL), included in Microsoft Robotics Developer Studio (I’m working with CTP 2.0 version). You can download the source code from my Skydrive:

AjDssAgents-0.1.zip

In my post:

Web Crawler example using DSS (Decentralized Software Services)

I wrote DSS service components orchestrated from VPL. In that example, there are a Dispatcher, a Resolver, a Downloader, and a Harvester components.

Let’s suppose we have many machines to run the web crawling process. We want to deploy and run MANY downloaders and harvesters, in a grid of machines, using automatic load balancing. The problem with VPL orchestration is that it doesn’t support load balancing out of the box. Then, I wrote an example where the components communicate each other, as agents, using special messages.

An agent, in this example, is a DSS service component, capable of receiving and processes appropiate incoming messages. It can send outgoing messages to other components. Instead of sending a message to one of the other components, an agent specify in the message the type of agent to which the message is forwarded.

Another specialized component, AgentHost, is in charge of receive such outgoing messages, and it forward them to a local or remote agent, according to its type.

The solution

The solution has three projects:

AjDssAgents contains the generic agent contract and types, and the concrete AgentHost implementation.

DecrementAgent and WebCrawler contain simple agents to use in the example. The web crawler code is similar to the implementation described in the post mentioned above.

The message

Agents interchanges messages, objects of type AgentMessage:

[DataContract] public class AgentMessage { [DataMember] public string From { get; set; } [DataMember] public string To { get; set; } [DataMember] public string Action { get; set; } [DataMember] public object Payload { get; set; } }

The From field indicates the origin of the message (I’m not using this field yet). The To field is the physical address (DSS address) of the target agents, or its logical type. In the current example, I’m using only logical types. Why a logical type? If a message has a To with value “WebCrawler/Dispatcher”, it will be forwarder to one agent that has that logical type.

How an agent knows what other agents are running and what are their logical types? It doesn’t. The component that keeps that information is the local singleton AgentHost. Each agent post their outgoing message to the local AgentHost, so, this components selects a target agent and forwards the message to it.

The Agents

Each agent is a DSS service component, with an address assigned when it is created. During the start of the agent, it sends to its local AgentHost a DSS message, indicating its address and its logical type (i.e., WebCrawler/Dispatcher). This is the way an AgentHost knows the agents that are running in its local DssHost process. See the starting code for the Dispatcher agent in WebCrawler example:

 

protected override void Start() { base.Start(); // Add service specific initialization here. _state.AgentType = "WebCrawler/Dispatcher"; host.NewNode newNode = new host.NewNode(new host.AgentInfo() { Address = this.ServiceInfo.Service, AgentType = _state.AgentType }); _hostPort.Post(newNode); }

The type of the agent is keep in its state.

This is a typical code, from Dispatcher agent, showing the treatment of an incoming message and the production of outgoing messages:

 

[ServiceHandler(ServiceHandlerBehavior.Concurrent)] public IEnumerator<ITask> PostMessageHandler(generic.PostMessage postMessage) { if (postMessage.Body.Action.Equals("Dispatch")) Dispatch(postMessage.Body); else if (postMessage.Body.Action.Equals("Resolve")) Resolve(postMessage.Body); postMessage.ResponsePort.Post(DefaultSubmitResponseType.Instance); yield break; } private void Dispatch(AgentMessage msg) { LogInfo("Entering Dispatcher with Action: " + msg.Action); LogInfo("URL: " + msg.Payload); DownloadTarget target = new DownloadTarget(); target.Uri = (string) msg.Payload; target.Depth = 1; AgentMessage postmsg = new AgentMessage() { Action = "Resolve", To = _state.AgentType, Payload = target }; host.PostMessage post = new host.PostMessage(postmsg); _hostPort.Post(post); } private void Resolve(AgentMessage msg) { LogInfo("Entering Dispatcher with Action: " + msg.Action); DownloadTarget downloadtarget = (DownloadTarget)msg.Payload; LogInfo("URL: " + downloadtarget.Uri + ", Depth: " + downloadtarget.Depth); DownloadTarget target = ProcessUrl(downloadtarget); if (target != null) { AgentMessage agentmsg = new AgentMessage() { To = "WebCrawler/Downloader", Action="Download", Payload = downloadtarget }; host.PostMessage postmsg = new host.PostMessage(agentmsg); _hostPort.Post(postmsg); } }

The AgentHost

There is one and only one AgentHost per running DssHost. The AgentHost receives new agent information (address and logical type), and keeps that information in its state.

It receives messages from local agents, and then it forward them to other local agents, or to a remote AgentHost. In the later case, it serialize the payload to a string, using XML serialization (a generic object cannot be send using the DSS generated proxies). This is the structure of a remote message:

 

[DataContract] public class RemoteAgentMessage { [DataMember] public string From { get; set; } [DataMember] public string To { get; set; } [DataMember] public string Action { get; set; } [DataMember] public string PayloadTypeName { get; set; } [DataMember] public string Payload { get; set; } }

Note that the remote message has an string Payload (XML serialization of the original payload), and its qualified type, so the target host can deserialize the payload to reconstruct the original object.

An AgentHost supports subscription. Other AgentHosts can subscribe to new agent informations. In general, if you have three machine, you start one AgentHost in each machine, and subcribe each agent host to the others. In this way, any AgentHost has all the information about the running agents, local and remote ones.

A distributed Web Crawler VPL example

The WebCrawlerVpl2 VPL example contains the diagram:

There are one Dispatcher, two Downloaders and two Harverters agents. The Dispatcher launch the initial URL to crawl, and keeps a list of downloaded URLs. A Downloader reads the content of each page to crawl. A Harvester examines that content and gets new links to download.

Note that they are two AgentHost, and they are related so each one sends new agent information to the other.

All these agents and components are distributed in two nodes:

Windows node will run on localhost:50000/50001, and Windows0 node uses localhost:50002/50003 as address. You can modify these settings, add more agents and nodes, without changing the code.

To run the distributed app, you must compile using the Build -> Compile as a Service option in VPL menu. Note: you must change the settings in VPL properties, now they are pointing to local directories in my machine:

VPL will show the compiling process:

After compiling the VPL example, go to MRDS DOS prompt, change to the bin directory, and launch the rundeployer.cmd:

I run the deployer in my local machine. If you plan to run the example in remote machines, you must start the deployer in each one.

Now, we are ready to run the web crawler. Select the Run -> Run on distributed nodes option, and the application will start. A dialog prompt for the URL to crawl. After entering a valid URL, the process begins to retrieve the pages in the site. You can see the first AgentHost state at:

http://localhost:50000/agenthost

There are three local agents and two remote ones.

In the other side, there is another AgentHost:

http://localhost:50002/agenthost0

See the difference: here are two local nodes, and three remote ones.

To see the advance of the process, point your explorer to

http://localhost:50000/console/output

Conclusions

With these ideas, we can implement grid-alike applications, running in many nodes. We lost the VPL orchestration, we can’t draw the road of the messages. But we gain load balancing and dynamic deploying. With some additional effort, we can write a controlling service, to starts and deploy the system in a new remote machine, even in the middle of a running process. The serialization of arbitrary objects is possible, but with custom serialization.

I could add subscription to messaging, in a future version. That is, an agent could receive some messages that are not for it, according to some subscription criteria. The suscriptions could be kept by the AgentHosts. When an AgentHost route an outcoming message, it could forward it to any interested local or remote agent.

Angel “Java” Lopez
http://www.ajlopez.com/en

June 9, 2008

Lisp-like interpreter using DSS and VPL

My sunday project was to write some primitives of a Lisp-like interpreter using DSS Service Components. The core interpreter was derived from my previous work on AjLisp, a Lisp interpreter (I had never published the code before). I write it using Visual Studio 2008 and the CTP v2.0 of Microsoft Robotics Developer Studio. I published the code on my Skydrive space as DssLisp.zip.

The solution

It has two projects: a class library named AjLisp (a revamped version of my original interpreter in VB.NET):

and a C# Dss Service project, with has many implemented service components:

It has no manifest. It is designed to use from VPL diagrams.

Most of these components receive or return SymbolicExpression, the base type of my interpreter. See the messages for Rest service Execute operation:

[DataContract()] public class ExecuteRequest { [DataMember] public SymbolicExpression Expression; } [DataContract] public class ExecuteResponse { [DataMember] public SymbolicExpression Result; }

To use this components, you must adjust the references and directories in the DSS project (it points to my local directories, and to my local Microsoft Robotics Developer Studio install).

Using VPL

There are three VPL programs included in the examples. They use the compiled components of the solution. I had a problem with these VPL programs. I couldn’t run them from the menu option Run. You must use Build -> Compile as a Service, and then, use the option Run -> Run Compiled Services. I guess the origin of the problem is that the diagrams maps special messages, not only primitive types or Strings. In order to successful compile, you must adjust the properties of diagrams in each VPL project:

The first one DssLispVpl1 calculate (first ‘(a b c)):

 

There are two components LispParser and LispToString, that help to convert from string to SymbolicExpression and from SymbolicExpression to string.

The second one is DssLispVpl2:

It uses LispAppend component to append two predefined lists.

The third one is a mini-interpreter DsspLispInterpreter:

I must improve the error treatment. Now, if an exception occurs inside the service process, no Fault response is generated, and a causality is raised.

I could implement some generic contracts, and reuse it in many components, for example, a GenericBinaryOperation component.

Problems

I failed to write an Activity, to implement a Reverse function. The activity called itself recursively, but the generated code returns at the last recursive invocation. The problem is caused by my use of a merge to return a value. It’s weird to explain clearly, after some hours trying to do the job, I abandoned the code. Another stone in the way was the fact that activities only accept predefined types in its messages and results. But in this case, I found a solution: generate the code, and modified the generated solution to accept my SymbolicExpression type as inputs and outputs.

One interesting thing I wrote, is the component Uncons, that has TWO expressions in its result response: it takes a list, and returns the first and the rest of that list. Having two members in the response, you can processes each one in parallel branchs of the diagram.

Conclusion

As in previous posts, I found the VPL orchestration as very powerful and interesting, but lacks the support of a generic object, and has problems to manage custom types. I think you can implements any functional languages primitives and make a VPL diagram for each program you want. VPL would have problems with recursion (or I didn’t found a workable solution to manage it). This exercise could be expanded to other ideas:

- Manage a pipeline processing a generic message (I discovered that we can have a generic object as a DataMember)
- Have load balance as a component
- Serialize/Deserialize objects between nodes
- Implement some workflow or application intergration patterns in VPL

Angel “Java” Lopez
http://www.ajlopez.com/en

 

May 25, 2008

Web Crawler example using DSS (Decentralized Software Services)

Some weeks ago, I wrote an example Scribble app using DSS. Today, I wrote another example: a web crawler application managed from Visual Programming Language (VPL). There is a additional VPL program that reads a web pages, using a Text To Speech control. You can download the example from my Skydrive:

DssWebCrawler2008May.zip

The solution

It has three projects: one DSS assembly, one class library with some utilities to parse an HTML content, and a class library to test the HTML parser. The test library uses NUnit: you can remove it from the solution, if you want, the library is only used for testing and it’s not needed in the final solution.

The DSS assembly is named DssWebCrawler. I defined five DSS service components:

Dispatcher: It receives an initial URL to download, and then dispatch it to the resolver.

Resolver: This service mantains a list of visited URLs, and check if the new URLs to download are valid and are in the same domain of the first page. It has a hardcoded max depth of 3 level of links to explore

Downloader: This service performs the download of the content of an URL. The content is returned as part of the response message.

Harvester: It examines the received content and harvest new URLs to examine and download. For each of these URLs it sends a notification to any service interested in that info.

Reader: It uses the simple HTML parse I wrote for this project. It can obtain the title of the page, or the body, discarding HTML tags and scripts.

The VPL Program

There is a VPL program named VPLWebCrawler. It consists of three diagrams. The first one defines the kickoff process. The first URL to download is entered in a dialog window:

 

 The second diagram defines the process of harvester notifications of new URLs:

The third diagram is a plus: it processes downloader notification of new content, extracting the title, and forwarding it to a Text to Speech component:

 

To launch the application, go to Run -> Start menu. A windows appears, prompting to enter the page URL to begin crawling:

Enter a valid URL, then, the crawling process begins:

After some seconds, the titles of the downloaded pages are posted to the Text to Speech service: you can hear the crawling process.

Reading Pages

In another VPL program, VPLWebReader, you can read the content of a web page, using the Text to Speech:

It is interesting that we are using the same service components than in the last example. But using VPL composition, we can use them for another purpose.

You can use it to read my experiments in “Anglish” (Angel’s English) at http://ajlopezen.zoomblog.com.

Conclusions

The service components were written to use with VPL orchestration. They don’t have partners, or direct connections with other service components in the project. This is a new way of programming: you must plan the message request and message response, to use in the communication to draw with VPL. The notification feature is a plus: you can use the same outgoing messages in different target components.

You can play a little more: put some of the components in another node/machine, using VPL new features.

I hope you’ll find this example useful. I had fun writting it.

Thanks to Fernando Tubio, for his initial ideas for a web crawler implementation.

Angel “Java” Lopez
http://www.ajlopez.com/en

April 20, 2008

Distributed Scribble with Microsoft Robotics Developer Studio and DSS

I’m playing withthe new CTP release of Microsoft Robotics Developer Studio. It has new features, as deploying and running a Visual Programming Language diagram in many nodes.

To explore some of this new functionality, I’ve write the dream of any software developer: a distributed multiuser scribble program…;-)… Well, it’s a very simple one, but it works, and it can run in many places, exposing and sharing your most inspired mouse drawings. The code can be downloaded from here.

The project

I have the 1.5 version of the MSRS and the new CTP MRDS (new acronym, guys), running in the same machine, without glitches. I’ve started creating a new Simple Dss Service RDS 2008:

The project contains one service component named DssScribble, and one Windows form, that is in charge of the drawing:

At the start of the service component, it creates the form, and connects to it using ports and its dispatcher queue:

protected override void Start() { base.Start(); // Add service specific initialization here. WinFormsServicePort.Post(new RunForm(CreateForm)); } private Form CreateForm() { return new CanvasForm(_mainPort, linePort, TaskQueue); }

Each line the user draw in the window is posted to service component, to be distributed to any interested subscriber:

[ServiceHandler(ServiceHandlerBehavior.Exclusive)] public IEnumerator<ITask> PostLineHandler(PostLine postLine) { SendNotification<PostLine>(_submgrPort, postLine); yield break; }

The lines are received in the NewLineHandler (the way to connect two components is via VPL, we’ll see later). The linePort is the port to communicate with the WinForm instance:

[ServiceHandler(ServiceHandlerBehavior.Exclusive)] public IEnumerator<ITask> NewLineHandler(NewLine newLine) { linePort.Post(newLine.Body); newLine.ResponsePort.Post(DefaultUpdateResponseType.Instance); yield break; }

The service component has no state. I use a class to transport the information of new lines to draw and distribute:

[DataContract()] public class DssScribbleState { } [DataContract()] public class DssScribbleLine { [DataMember()] public int fromX; [DataMember()] public int fromY; [DataMember()] public int toX; [DataMember()] public int toY; public DssScribbleLine() { } public DssScribbleLine(int fromX, int fromY, int toX, int toY) { this.fromX = fromX; this.fromY = fromY; this.toX = toX; this.toY = toY; } }

Drawing with VPL

The Visual Programming Language is a fascinating tool, with so many features. You can drag and drop service components from the toolbar. When the tool is launched, you can search the DssScrible service component in the left bottom toolbox:

I dropped two instances of DssScribble service component, and renamed them to DssScribble1 and DssScribble2:

I connected the instances, so they listen their notifications. Every line produced in one instance, is transmitted to the other. In this way, what you draw in a window is viewed in the other.

The connection is between the PostLine notification to a NewLine operation:

To launch the application, go to Run -> Start menu. A windows appears, showing the log messages:

Few moments, and the two instances of DssScribble service are created, each one shows a form. You can draw in any form, and the mouse drawn lines are replayed in the other form:

That is my self-portrait: my left eye with a tear, because the emotion of distributed scribbling…. ;-)

Running on distributed nodes

VPL IDE has a node designer. At the right, there is a node branch on the Project tree. Each node represents a DSS node hosting DSS service components. I added two nodes, the first containing the instance and diagram of the current application (diagrams are services that can be viewed during the debug phase in your browser, you’ll love that featuer). I dragged and dropped DssScribble2 instance to the second node:

Note the right bottom panel: I set the properties for Windows0 node, in the same localhost machine, but with other ports (Windows node will run at 50000/50001 ports). In other project, I tried to have a remote machine, and the distribution of the application worked without problems. You can define any number of nodes, each in a machine of your office, put on them DssScribble instances, related each other, and play scribbling with your coworkers (after work, please…. ;-)

In order to run the application in a distribute way, you must compile it. Use the Build -> Compile as a Service menu option. You must set a directory where to generate the code and solution. Then, VPL tool compile the solution and generate packages to deploy. You can open the generated solution with Visual Studio, if you are curious (the code is a very interesting, a very clever one, implementing, I suppose, an state or stack machine, look at Decrement and Increment methods).

Each node must have a package deployer service, running on another port (the same one for all nodes, default value is 55555). Go to run bin directory of Microsoft Robotics Developer Studio, and run the rundeployer.cmd:

 

Note: if you want to compile the service again, you must stop the deployer node: this is using the bin directory and the compilation conflicts with the package deployer host.

Now, we are ready to deploy our killer app: execute Run -> Run on distributed nodes. The system warns about to have deployers running:

And the magic begins: each node is contacted, and the application is deployer to them. A DSS node is started in each node, and its corresponding service instances are created. In this example, you will have two new old-DOS windows, running each DSS node. This is my second node window:

Now, each node show its scribbling window, and the fun begins.

Notes

I’m not using partners in this example. Instead, I’m put notification of events, in this case, new lines, so other components can receive such notification. You can write other components, other than scribble windows, to receive and process the lines. One idea is to use the coordinates of a new line to control a robot motor, or to persists the lines produced with a new persistence component. To control a robot motor, you should play with VPL, without rewriting the service component code.

Conclusions

Using VPL you can compose new applications from preexisting components. You can create your own activities, too. And using node diagrams (in VPL or using the DSS Manifest Editor) with the packager deployer, we can distribute the application in many physical nodes, in any convenient manner. But you must define each node and component: there is no provisioning of have a grid, so we can clone and distribute a working component. A gridified DSS application can be written, but in my opinion, it should be created with custom code. Idea to explore: write a custom control application that, using package deployer, deploys tasks to nodes in a grid.

All these tools opened my brain. Writing DSS service component is a new way of thinking and programming, something I want to explore for years: how to write objects with one-way message processing, in a distributed and parallel way (AjAgents was motivated by that objective). I envision some system that model Minsky ideas about the human mind.

Well, it’s a new brave world. Dorothy: we are not in Kansas anymore… ;-)

Angel “Java” Lopez
http://www.ajlopez.com/en

April 15, 2008

Message Passing Interface, CCR, DSS, and Pure MPI.NET

Filed under: .NET, Concurrency and Coordination Runtime, Grid Computing — ajlopez @ 11:40 am

Recently, during my reseach about grid computing, Microsoft Robotics Studio, DSS and CCR, I found a very interesting paper:

High Performance Multi-Paradigm Messaging Runtime Integrating Grids and Multicore Systems

The authors are Two of the authors are Xiaohong Qiu, Geoffrey C. Fox, Huapeng Yuan, Seung-Hee Bae, from Indiana University Bloomington, and George Chrysanthakopoulos, Henrik Frystyk Nielsen, from Microsoft Research. Nielsen and Chrysanthakopoulos are the “creators” of the Concurrency and Coordination Runtime (CCR) and Decentralized Software Services (DSS), pillar technologies of Microsoft Robotics Studio, that can be used beyond robotics. More on these technologies at:

http://www.microsoft.com/robotics

The paper abstract is:

eScience applications need to use distributed Grid environments where each component is an individual or cluster of multicore machines. These are expected to have 64-128 cores 5 years from now and need to support scalable parallelism. Users will want to compose heterogeneous components into single jobs and run seamlessly in both distributed fashion and on a future “Grid on a chip” with different subsets of cores supporting individual components. We support this with a simple programming model made up of two layers supporting traditional parallel and Grid programming paradigms (workflow) respectively. We examine for a parallel clustering application, the Concurrency and Coordination Runtime CCR from Microsoft as a multi-paradigm runtime that integrates the two layers. Our work uses managed code (C#) and for AMD and Intel processors shows around a factor of 5 better performance than Java. CCR has MPI pattern and dynamic threading latencies of a few microseconds that are competitive with the performance of standard MPI for C.

What is MPI? The acronym refers to Message Passing Interface. According to Wikipedia:

Message Passing Interface (MPI) is both a computer specification and is an implementation that allows many computers to communicate with one another. It is used in computer clusters.

There is a Microsoft Implementation:

Microsoft Message Passing Interface (MS MPI) is an implementation of the MPI2 specification by Microsoft for use in Windows Compute Cluster Server to interconnect and communicate (via messages) between High performance computing nodes. It is mostly compatible with the MPICH2 reference implementation, with some exceptions for job launch and management. MS MPI includes bindings for C and FORTRAN languages. It supports using the Microsoft Visual Studio for debugging purposes.

Oh! FORTRAN….. Those old good days! ;-). I remember working with Gregory Chaitin implementation of Lisp on FORTRAN, last century. But no back to the past, paraphrasing David Hilbert: Out of this paradise that Java and .NET have created nobody will expell us…. ;-). You can read the original cite at this interesting thread.

But I disgress. Back to topic.

The main sites about MPI are:

http://www.mpi-forum.org/
http://www.open-mpi.org/
http://www.lam-mpi.org/

I was thinking of implementing some MPI ideas with .NET or Java, when I visited this site:

http://www.purempi.net/

PureMpi.NET is a completely managed implementation of the message passing interface. The object-oriented API is simple, and easy to use for parallel programming. It has been developed based on the latest .NET technologies, including Windows Communication Foundation (WCF). This allows you to declaratively specify the binding and endpoint configuration for your environment, and performance needs. When using the SDK, a programmer will definitely see the MPI’ness of the interfaces come through, and will enjoy taking full advantage of .NET features – including generics, delegates, asynchronous results, exception handling, and extensibility points.

PureMpi.NET allows you to create high performance production quality parallel systems, with all the benefits of in .NET

It is an implementation that you can download and use, with VS2005 or VS2008. It uses generics to implements typed channels on MPI.

I downloaded the library, and installed it on a machine with Visual Studio 2008. The installation program added a new project template, Mpi.NET:

I created a project, that looks:

I modified the Program.cs to:

 

using System; using System.Collections.Generic; using System.Linq; using System.Text; using Mpi; namespace Mpi.NET1 { class Program { static void Main(string[] args) { ProcessorGroup.Process("MPIEnvironment", delegate(IDictionary<string, Comm> comms) { Comm comm = comms["MPI_COMM_WORLD"]; Console.WriteLine(comm.Rank); IAsyncResult result = comm.BeginSend<string>(0, "", "Rank: " + comm.Rank, TimeSpan.FromSeconds(30), null, null); if (comm.Rank == 0) { for (int i = 0; i < comm.Size; i++) { string receivedMsg = comm.Receive<string>(i, Constants.AnyTag, TimeSpan.FromSeconds(30)); Console.WriteLine(receivedMsg); } } comm.EndSend<string>(result); }); } } }

The ProcessGroup class is in charge of the processes to run. Note the support of delegates to specify the process. A MPI process receives a dictionary of Comm objects, channels to use to communicate with other MPI processes.

The ProcessGroup class has this structure (according to metadata info):

 

namespace Mpi { public class ProcessorGroup : IDisposable { public ProcessorGroup(Environment environment, Processor processor); public ProcessorGroup(string environment, Processor processor); public Environment Environment { get; } public ICollection<IAsyncResult> Results { get; } public void Dispose(); protected virtual void Dispose(bool disposing); public static void Process(string environmentConfigName, Processor processor); public void Start(); public void WaitForCompletion(); } }

The number and configuration of processors could be defined in the App.config file:

 

<?xml version="1.0" encoding="utf-8" ?> <configuration> <configSections> <section name="Mpi" type="Mpi.ConfigurationSection, Mpi"/> </configSections> <Mpi> <Environments> <Environment name="MPIEnvironment"> <Hosts> <Host comms="MPI_COMM_WORLD" client="MpiClient1" service="MpiService1" /> <Host comms="MPI_COMM_WORLD" client="MpiClient2" service="MpiService2"/> <Host comms="MPI_COMM_WORLD" client="MpiClient3" service="MpiService3"/> </Hosts> </Environment> </Environments> </Mpi> <system.serviceModel> <client> <endpoint address="net.tcp://localhost:8080/MpiService" binding="netTcpBinding" bindingConfiguration="" contract="Mpi.IMpiService" name="MpiClient1"> <identity> <userPrincipalName value="" /> </identity> </endpoint> <endpoint address="net.tcp://localhost:8081/MpiService" binding="netTcpBinding" bindingConfiguration="" contract="Mpi.IMpiService" name="MpiClient2"> <identity> <userPrincipalName value="" /> </identity> </endpoint> <endpoint address="net.tcp://localhost:8082/MpiService" binding="netTcpBinding" bindingConfiguration="" contract="Mpi.IMpiService" name="MpiClient3"> <identity> <userPrincipalName value="" /> </identity> </endpoint> </client> <behaviors> <serviceBehaviors> <behavior name="MpiServiceBehavior"> <serviceDebug httpHelpPageEnabled="false" httpsHelpPageEnabled="false" includeExceptionDetailInFaults="true" /> </behavior> </serviceBehaviors> </behaviors> <services> <service behaviorConfiguration="MpiServiceBehavior" name="MpiService1"> <endpoint address="net.tcp://localhost:8080/MpiService" binding="netTcpBinding" bindingConfiguration="" name="MpiServiceEndpoint" contract="Mpi.IMpiService" /> </service> <service behaviorConfiguration="MpiServiceBehavior" name="MpiService2"> <endpoint address="net.tcp://localhost:8081/MpiService" binding="netTcpBinding" bindingConfiguration="" name="MpiServiceEndpoint" contract="Mpi.IMpiService" /> </service> <service behaviorConfiguration="MpiServiceBehavior" name="MpiService3"> <endpoint address="net.tcp://localhost:8082/MpiService" binding="netTcpBinding" bindingConfiguration="" name="MpiServiceEndpoint" contract="Mpi.IMpiService" /> </service> </services> </system.serviceModel> <system.runtime.serialization> <dataContractSerializer> <declaredTypes> </declaredTypes> </dataContractSerializer> </system.runtime.serialization> </configuration>

Oh! They use <host..>…  This remembers me AjMessages;-)

Running the program produces:

Well, it’s not a great program, I must admit: but it’s my first MPI program. There are 3 “ranks”, according to config file above.

You’ll find many running examples include with Pure MPI.NET distribution. For me, it’s an interesting implementation of MPI ideas, with twists adapted from .NET world: generics and delegates are welcome.

¿Grid and MPI? Maybe. I must study the references mentioned in the cited paper. Althought the paper is dedicated to high performance issues, it has a good conceptual discussion of execution model, and relations with MPI, CCR and DSS.

Angel “Java” Lopez
http://www.ajlopez.com/en

April 10, 2008

Genetic Algorithms with AjAgents and Concurrency and Coordination Runtime (CCR)

Filed under: .NET, AjAgents, Concurrency and Coordination Runtime — ajlopez @ 10:19 am

In previous post, I wrote about my C# example AjAgents:

Agents using Concurrency and Coordination Runtime (CCR)

and about some ideas to explore:

Agents in a Grid

Now, I extended my example AjAgentsCCR-0.1.zip implemented two new projects. One is a console project AjAgents.Genetic01, and the other is WinForms onw, AjAgents.WinGenetic01:

The basic idea is to run a genetic algorithm using agents, that send messages using CCR ports. Then, each agents process its incomming messages, and post outcomming message to other agents, in a “parallel” way. The problem I modeled is the classical Travelling Salesman Problem. According to Wikipedia:

If a salesman starts at point A, and if the distances between every pair of points are known, what is the shortest route which visits all points and returns to point A?

The Windows form is simple:

The Genoma class represents the travel (a list of Points and a distance value):

class Genoma { public List<Point> travel = new List<Point>(); public int value; } class Point { public int x; public int y; }

But the interesting point, is the lunch of the run. First, the agents are created and connected:

best = new BestSoFar(); evaluator = new Evaluator(); mutator = new Mutator(); collector = new Collector(); evaluator.Collector = collector; collector.Mutator = mutator; collector.Evaluator = evaluator; collector.BestSoFar = best; mutator.Evaluator = evaluator;

Then, a initial genoma is created:

Genoma genoma = new Genoma(); Random rnd = new Random(); for (int k = 0; k<40; k++) { Point pt = new Point(); pt.x = rnd.Next(12); pt.y = rnd.Next(12); genoma.travel.Add(pt); } genoma.value = evaluator.CalculateValue(genoma);

The BestSoFar agent can raise an event. The form is listening to that event, to refresh the best travel drawing. The program sends the genoma to the mutator agent, many times:

best.NewBest += BestGenoma; for (int k = 0; k < 60; k++) { mutator.Post("Mutate", genoma); }

Note the use of the Post method. Every agent implements that method, that invokes a method in the agent, but using a CCR port to invoke the target method. So, the method is executed in a “parallel way”, possibly in a thread from the CCR thread pool.

A simple diagram to explain the interactions between the agents:

The mutator sends each genoma to the Evaluator. This agent calculate the value assigned to that genoma. Then, it sends the genoma to a Collector. This is a new idea, for this example: instead of having a population class, the collector receives genomas, and then when it has n or more, evaluates the best ones, and post them to a Mutator, so the process continues. There is no population concept, or, at least, not the standard population found in other implementations. The best genoma detected by the Collector is informed to the BestSoFar agent. A typical method, from Collector class:

void ProcessList(List<Genoma> list) { list.Sort(comparer); bestsofar.Post("Process",list[0]); evaluator.Post("Evaluate",list[0]); evaluator.Post("Evaluate",list[1]); for (int k = 0; k < 4; k++) { mutator.Post("Mutate",list[k]); mutator.Post("Mutate",list[k]); } }

I must improve the genetic algorithm. Now, it’s using only mutation, and it has no crossover operator. Then, its result is not optimal: it can reach a local minimum, but it could not be the best solution. But the basis of this example, is to show how AjAgents and CCR can be used in an “proof of concept” example.

Conclusions

One can see each agent as a “little cell” or “little organism” that reacts to incoming messages, and sends outcomming messages to other agents. Each agent is loosly coupled with the others. In these examples, the agents were created by code, but it’s clear that they could be created and related using dependency injection frameworks, as Spring.NET or the new Unity Application Block from Microsoft.

Instead of using only CCR ports, I could write the agents as DSS Service components (see Microsoft Robotics Studio for more information), so the algorithm could be “gridified”, and each agents is located in a transparent way to others.

But this is another story….;-)

Incidentally, yesterday this post announced the new version of Microsoft Robotics Developer Studio 2008:

http://blogs.msdn.com/msroboticsstudio/archive/2008/04/09/microsoft-robotics-developer-studio-2008-ctp-april-available.aspx

The third new feature, mentioned in that post, could be a great feature for these ideas about “miniagents”:

Support for creating applications that run on multiple DSS nodes using Visual Programming Language and DSS Manifest Editor. This makes it much simpler to create applications that run across nodes, either on the same machine or across the network. When an application containing multiple nodes is to be started, VPL creates individual deploy packages for each node and fires them up across the network.

Ooooohhh!…. ;-) ;-)

Angel “Java” Lopez
http://www.ajlopez.com/

Older Posts »

The Shocking Blue Green Theme Blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.

Join 56 other followers