Angel \”Java\” Lopez on Blog

June 22, 2008

Distributed Agents and Fractals using DSS/VPL

Last week I wrote the base of an application running distributed agents over DSS/VPL, interchanging arbitrary messages, with automatic load balancing. You can read the details at:

Distributed Agents using DSS/VPL

Today, I extended the example with a new project, Fractal:

You can download from my Skydrive.

It has two DSS Service Components, one is the Calculator: it calculates a sector of the Mandelbrot fractal. The other component is the Renderer, that has a form to control and show the results of the calculation. The message that has the info of the sector to calculate is:

public class SectorInfo : MessagePayload { public double RealMinimum { get; set; } public double ImgMinimum { get; set; } public double Delta { get; set; } public int FromX { get; set; } public int FromY { get; set; } public int Width { get; set; } public int Height { get; set; } public int MaxIterations { get; set; } public int MaxValue { get; set; } }

Another class is the message that returns the calculation:

public class Sector : MessagePayload { public int FromX { get; set; } public int FromY { get; set; } public int Width { get; set; } public int Height { get; set; } public int[] Values { get; set; } }

The calculator splits the sector to calculate, if it too big. It can calculate all in only one step, but it is interesting to split the sector to generate more messages:

 

private void Calculate(AgentMessage msg) { LogInfo("Entering Calculator with Action: " + msg.Action); SectorInfo sectorInfo = (SectorInfo) msg.Payload; LogInfo(String.Format("X {0} Y {1} Width {2} Height {3}", sectorInfo.FromX, sectorInfo.FromY, sectorInfo.Width, sectorInfo.Height)); if (sectorInfo.Width > 100 && sectorInfo.Height > 100) SplitSector(sectorInfo); else CalculateSector(sectorInfo); }

As in the previous example, you can run this from a VPL diagram, FractalVpl1:

There is only one renderer, and two calculators agents. If you launch this VPL program, a window appears. This is the initial window content (after pressing the Calculate button):

You can drag and move the mouse to select a new region, or use the buttons to zoom in and out. Reset backs to the initial position. You can resize the window and invoke Calculate again. New colors button changes the color palette in use.

There is another VPL program, FractalVpl2, that it can be used to run the same example in a distributed way. It has a diagram with two AgentHosts:

and two nodes:

You must compile the VPL example and run as distributed nodes, using the rundeployer.cmd (see my previous post for details).

These some of the drawings the system produces:

 

Enjoy!

Angel “Java” Lopez
http://www.ajlopez.com/en

June 21, 2008

Messages everywhere

Recently, I wrote a post about Message Passing Interface:

Message Passing Interface, CCR, DSS, and Pure MPI.NET

I used to pass message between agents in my example of a web crawler:

Distributed Agents using DSS/VPL

The passing of messages between components, agents, objects, is a feature that deserves more study. I guess we could write new kinds of applications, using one-way message passing, so we could abandon the call-a-method current way of doing things. Let’s explore first, some message passing applications (not only one-way style).

In the Wikipedia entry about Message Passing:

http://en.wikipedia.org/wiki/Message_passing

we can read Alan Kay opinion:

Alan Kay has suggested that he placed too much emphasis on objects themselves and not enough on the messages being sent between them. Message passing enables extreme late binding in systems.

If you develop systems with Smalltalk, Self or alikes, you’ll notice that the message is a first citizen, in many implementations, a full object, not only a way of call a method.

There is another place for use message passing. For 25 years, QNX operating systems uses the message passing paradigm to run a real time kernel.

I found this interview at Dr. Dobb’s to Sebastien Marineau-Mes, VP of engeneering at QNX:

Real Time OS in Today s World

Sebastien talks about the use of QNX in current market, and the challenge that multi-core machine could create on legacy code.

Remember: all the QNX kernel is based on message passing, although its messages are not one-way, and the passing is optimized to not incurr in loss of performance (more details, see QNX at Wikipedia). I see that many of these challenges and opportunities could be translated to the use, not only to multi-core, but to “multi-machines” in a grid. There many forces that are conspiring to bring these topics to current arena:

- We have a better understanding of agents, message passing and other ideas

- Normal hardware is cheap

- Each year, there are more application with higher level needs of scalabity (the user base will be the whole Internet in the near future, for non-trivial app)

- Many application must interact with other application in the same enterprise or in the net, and messages are an effective way to do that.

- In order to face that challenges, we must begin to abandon n-tier-only apps, to a cloud, grid or “something alike” schema.

I could imagine languages and technologies, based on message passing features. That is one of the reasons I’ve been involved exploring simples and crazy ideas with AjMessages, AjAgents, and Distributed Software Services during last months. I hope to write more about these subjects:

- Another example of distributed agents using DSS/VPL

- AjMessages ported to DSS (I had an implementation last year, but I never published it, I published the WCF only version)

- One-way messages implemented as primitives in AjTalk (Smalltalk-like interpreter)

- Deferred/Concurrent/LazyEvaluation/SomethingAlike implemented in AjBasic (using CCR?)

- Blog about a better-finished app running in a grid of DSS hosts (my team was working hard on this, last months).

Incidentally, you can read more about use cases applied in the real world using CCR/DDS, in Arvindra Sehmi’s post:

CCR/DSS Use Cases in the Enterprise

So many ideas…. only one life…. Should I begin to parallized myself? ajlopez-in-a-grid…. ;-)

Angel “Java” Lopez
http://www.ajlopez.com/en

June 15, 2008

Distributed Agents using DSS/VPL

In this post, I explore some ideas to implement distributed agents, leveraging the features from Decentrilized Software Services (DSS) and Visual Programming Language (VPL), included in Microsoft Robotics Developer Studio (I’m working with CTP 2.0 version). You can download the source code from my Skydrive:

AjDssAgents-0.1.zip

In my post:

Web Crawler example using DSS (Decentralized Software Services)

I wrote DSS service components orchestrated from VPL. In that example, there are a Dispatcher, a Resolver, a Downloader, and a Harvester components.

Let’s suppose we have many machines to run the web crawling process. We want to deploy and run MANY downloaders and harvesters, in a grid of machines, using automatic load balancing. The problem with VPL orchestration is that it doesn’t support load balancing out of the box. Then, I wrote an example where the components communicate each other, as agents, using special messages.

An agent, in this example, is a DSS service component, capable of receiving and processes appropiate incoming messages. It can send outgoing messages to other components. Instead of sending a message to one of the other components, an agent specify in the message the type of agent to which the message is forwarded.

Another specialized component, AgentHost, is in charge of receive such outgoing messages, and it forward them to a local or remote agent, according to its type.

The solution

The solution has three projects:

AjDssAgents contains the generic agent contract and types, and the concrete AgentHost implementation.

DecrementAgent and WebCrawler contain simple agents to use in the example. The web crawler code is similar to the implementation described in the post mentioned above.

The message

Agents interchanges messages, objects of type AgentMessage:

[DataContract] public class AgentMessage { [DataMember] public string From { get; set; } [DataMember] public string To { get; set; } [DataMember] public string Action { get; set; } [DataMember] public object Payload { get; set; } }

The From field indicates the origin of the message (I’m not using this field yet). The To field is the physical address (DSS address) of the target agents, or its logical type. In the current example, I’m using only logical types. Why a logical type? If a message has a To with value “WebCrawler/Dispatcher”, it will be forwarder to one agent that has that logical type.

How an agent knows what other agents are running and what are their logical types? It doesn’t. The component that keeps that information is the local singleton AgentHost. Each agent post their outgoing message to the local AgentHost, so, this components selects a target agent and forwards the message to it.

The Agents

Each agent is a DSS service component, with an address assigned when it is created. During the start of the agent, it sends to its local AgentHost a DSS message, indicating its address and its logical type (i.e., WebCrawler/Dispatcher). This is the way an AgentHost knows the agents that are running in its local DssHost process. See the starting code for the Dispatcher agent in WebCrawler example:

 

protected override void Start() { base.Start(); // Add service specific initialization here. _state.AgentType = "WebCrawler/Dispatcher"; host.NewNode newNode = new host.NewNode(new host.AgentInfo() { Address = this.ServiceInfo.Service, AgentType = _state.AgentType }); _hostPort.Post(newNode); }

The type of the agent is keep in its state.

This is a typical code, from Dispatcher agent, showing the treatment of an incoming message and the production of outgoing messages:

 

[ServiceHandler(ServiceHandlerBehavior.Concurrent)] public IEnumerator<ITask> PostMessageHandler(generic.PostMessage postMessage) { if (postMessage.Body.Action.Equals("Dispatch")) Dispatch(postMessage.Body); else if (postMessage.Body.Action.Equals("Resolve")) Resolve(postMessage.Body); postMessage.ResponsePort.Post(DefaultSubmitResponseType.Instance); yield break; } private void Dispatch(AgentMessage msg) { LogInfo("Entering Dispatcher with Action: " + msg.Action); LogInfo("URL: " + msg.Payload); DownloadTarget target = new DownloadTarget(); target.Uri = (string) msg.Payload; target.Depth = 1; AgentMessage postmsg = new AgentMessage() { Action = "Resolve", To = _state.AgentType, Payload = target }; host.PostMessage post = new host.PostMessage(postmsg); _hostPort.Post(post); } private void Resolve(AgentMessage msg) { LogInfo("Entering Dispatcher with Action: " + msg.Action); DownloadTarget downloadtarget = (DownloadTarget)msg.Payload; LogInfo("URL: " + downloadtarget.Uri + ", Depth: " + downloadtarget.Depth); DownloadTarget target = ProcessUrl(downloadtarget); if (target != null) { AgentMessage agentmsg = new AgentMessage() { To = "WebCrawler/Downloader", Action="Download", Payload = downloadtarget }; host.PostMessage postmsg = new host.PostMessage(agentmsg); _hostPort.Post(postmsg); } }

The AgentHost

There is one and only one AgentHost per running DssHost. The AgentHost receives new agent information (address and logical type), and keeps that information in its state.

It receives messages from local agents, and then it forward them to other local agents, or to a remote AgentHost. In the later case, it serialize the payload to a string, using XML serialization (a generic object cannot be send using the DSS generated proxies). This is the structure of a remote message:

 

[DataContract] public class RemoteAgentMessage { [DataMember] public string From { get; set; } [DataMember] public string To { get; set; } [DataMember] public string Action { get; set; } [DataMember] public string PayloadTypeName { get; set; } [DataMember] public string Payload { get; set; } }

Note that the remote message has an string Payload (XML serialization of the original payload), and its qualified type, so the target host can deserialize the payload to reconstruct the original object.

An AgentHost supports subscription. Other AgentHosts can subscribe to new agent informations. In general, if you have three machine, you start one AgentHost in each machine, and subcribe each agent host to the others. In this way, any AgentHost has all the information about the running agents, local and remote ones.

A distributed Web Crawler VPL example

The WebCrawlerVpl2 VPL example contains the diagram:

There are one Dispatcher, two Downloaders and two Harverters agents. The Dispatcher launch the initial URL to crawl, and keeps a list of downloaded URLs. A Downloader reads the content of each page to crawl. A Harvester examines that content and gets new links to download.

Note that they are two AgentHost, and they are related so each one sends new agent information to the other.

All these agents and components are distributed in two nodes:

Windows node will run on localhost:50000/50001, and Windows0 node uses localhost:50002/50003 as address. You can modify these settings, add more agents and nodes, without changing the code.

To run the distributed app, you must compile using the Build -> Compile as a Service option in VPL menu. Note: you must change the settings in VPL properties, now they are pointing to local directories in my machine:

VPL will show the compiling process:

After compiling the VPL example, go to MRDS DOS prompt, change to the bin directory, and launch the rundeployer.cmd:

I run the deployer in my local machine. If you plan to run the example in remote machines, you must start the deployer in each one.

Now, we are ready to run the web crawler. Select the Run -> Run on distributed nodes option, and the application will start. A dialog prompt for the URL to crawl. After entering a valid URL, the process begins to retrieve the pages in the site. You can see the first AgentHost state at:

http://localhost:50000/agenthost

There are three local agents and two remote ones.

In the other side, there is another AgentHost:

http://localhost:50002/agenthost0

See the difference: here are two local nodes, and three remote ones.

To see the advance of the process, point your explorer to

http://localhost:50000/console/output

Conclusions

With these ideas, we can implement grid-alike applications, running in many nodes. We lost the VPL orchestration, we can’t draw the road of the messages. But we gain load balancing and dynamic deploying. With some additional effort, we can write a controlling service, to starts and deploy the system in a new remote machine, even in the middle of a running process. The serialization of arbitrary objects is possible, but with custom serialization.

I could add subscription to messaging, in a future version. That is, an agent could receive some messages that are not for it, according to some subscription criteria. The suscriptions could be kept by the AgentHosts. When an AgentHost route an outcoming message, it could forward it to any interested local or remote agent.

Angel “Java” Lopez
http://www.ajlopez.com/en

May 25, 2008

Web Crawler example using DSS (Decentralized Software Services)

Some weeks ago, I wrote an example Scribble app using DSS. Today, I wrote another example: a web crawler application managed from Visual Programming Language (VPL). There is a additional VPL program that reads a web pages, using a Text To Speech control. You can download the example from my Skydrive:

DssWebCrawler2008May.zip

The solution

It has three projects: one DSS assembly, one class library with some utilities to parse an HTML content, and a class library to test the HTML parser. The test library uses NUnit: you can remove it from the solution, if you want, the library is only used for testing and it’s not needed in the final solution.

The DSS assembly is named DssWebCrawler. I defined five DSS service components:

Dispatcher: It receives an initial URL to download, and then dispatch it to the resolver.

Resolver: This service mantains a list of visited URLs, and check if the new URLs to download are valid and are in the same domain of the first page. It has a hardcoded max depth of 3 level of links to explore

Downloader: This service performs the download of the content of an URL. The content is returned as part of the response message.

Harvester: It examines the received content and harvest new URLs to examine and download. For each of these URLs it sends a notification to any service interested in that info.

Reader: It uses the simple HTML parse I wrote for this project. It can obtain the title of the page, or the body, discarding HTML tags and scripts.

The VPL Program

There is a VPL program named VPLWebCrawler. It consists of three diagrams. The first one defines the kickoff process. The first URL to download is entered in a dialog window:

 

 The second diagram defines the process of harvester notifications of new URLs:

The third diagram is a plus: it processes downloader notification of new content, extracting the title, and forwarding it to a Text to Speech component:

 

To launch the application, go to Run -> Start menu. A windows appears, prompting to enter the page URL to begin crawling:

Enter a valid URL, then, the crawling process begins:

After some seconds, the titles of the downloaded pages are posted to the Text to Speech service: you can hear the crawling process.

Reading Pages

In another VPL program, VPLWebReader, you can read the content of a web page, using the Text to Speech:

It is interesting that we are using the same service components than in the last example. But using VPL composition, we can use them for another purpose.

You can use it to read my experiments in “Anglish” (Angel’s English) at http://ajlopezen.zoomblog.com.

Conclusions

The service components were written to use with VPL orchestration. They don’t have partners, or direct connections with other service components in the project. This is a new way of programming: you must plan the message request and message response, to use in the communication to draw with VPL. The notification feature is a plus: you can use the same outgoing messages in different target components.

You can play a little more: put some of the components in another node/machine, using VPL new features.

I hope you’ll find this example useful. I had fun writting it.

Thanks to Fernando Tubio, for his initial ideas for a web crawler implementation.

Angel “Java” Lopez
http://www.ajlopez.com/en

April 20, 2008

Distributed Scribble with Microsoft Robotics Developer Studio and DSS

I’m playing withthe new CTP release of Microsoft Robotics Developer Studio. It has new features, as deploying and running a Visual Programming Language diagram in many nodes.

To explore some of this new functionality, I’ve write the dream of any software developer: a distributed multiuser scribble program…;-)… Well, it’s a very simple one, but it works, and it can run in many places, exposing and sharing your most inspired mouse drawings. The code can be downloaded from here.

The project

I have the 1.5 version of the MSRS and the new CTP MRDS (new acronym, guys), running in the same machine, without glitches. I’ve started creating a new Simple Dss Service RDS 2008:

The project contains one service component named DssScribble, and one Windows form, that is in charge of the drawing:

At the start of the service component, it creates the form, and connects to it using ports and its dispatcher queue:

protected override void Start() { base.Start(); // Add service specific initialization here. WinFormsServicePort.Post(new RunForm(CreateForm)); } private Form CreateForm() { return new CanvasForm(_mainPort, linePort, TaskQueue); }

Each line the user draw in the window is posted to service component, to be distributed to any interested subscriber:

[ServiceHandler(ServiceHandlerBehavior.Exclusive)] public IEnumerator<ITask> PostLineHandler(PostLine postLine) { SendNotification<PostLine>(_submgrPort, postLine); yield break; }

The lines are received in the NewLineHandler (the way to connect two components is via VPL, we’ll see later). The linePort is the port to communicate with the WinForm instance:

[ServiceHandler(ServiceHandlerBehavior.Exclusive)] public IEnumerator<ITask> NewLineHandler(NewLine newLine) { linePort.Post(newLine.Body); newLine.ResponsePort.Post(DefaultUpdateResponseType.Instance); yield break; }

The service component has no state. I use a class to transport the information of new lines to draw and distribute:

[DataContract()] public class DssScribbleState { } [DataContract()] public class DssScribbleLine { [DataMember()] public int fromX; [DataMember()] public int fromY; [DataMember()] public int toX; [DataMember()] public int toY; public DssScribbleLine() { } public DssScribbleLine(int fromX, int fromY, int toX, int toY) { this.fromX = fromX; this.fromY = fromY; this.toX = toX; this.toY = toY; } }

Drawing with VPL

The Visual Programming Language is a fascinating tool, with so many features. You can drag and drop service components from the toolbar. When the tool is launched, you can search the DssScrible service component in the left bottom toolbox:

I dropped two instances of DssScribble service component, and renamed them to DssScribble1 and DssScribble2:

I connected the instances, so they listen their notifications. Every line produced in one instance, is transmitted to the other. In this way, what you draw in a window is viewed in the other.

The connection is between the PostLine notification to a NewLine operation:

To launch the application, go to Run -> Start menu. A windows appears, showing the log messages:

Few moments, and the two instances of DssScribble service are created, each one shows a form. You can draw in any form, and the mouse drawn lines are replayed in the other form:

That is my self-portrait: my left eye with a tear, because the emotion of distributed scribbling…. ;-)

Running on distributed nodes

VPL IDE has a node designer. At the right, there is a node branch on the Project tree. Each node represents a DSS node hosting DSS service components. I added two nodes, the first containing the instance and diagram of the current application (diagrams are services that can be viewed during the debug phase in your browser, you’ll love that featuer). I dragged and dropped DssScribble2 instance to the second node:

Note the right bottom panel: I set the properties for Windows0 node, in the same localhost machine, but with other ports (Windows node will run at 50000/50001 ports). In other project, I tried to have a remote machine, and the distribution of the application worked without problems. You can define any number of nodes, each in a machine of your office, put on them DssScribble instances, related each other, and play scribbling with your coworkers (after work, please…. ;-)

In order to run the application in a distribute way, you must compile it. Use the Build -> Compile as a Service menu option. You must set a directory where to generate the code and solution. Then, VPL tool compile the solution and generate packages to deploy. You can open the generated solution with Visual Studio, if you are curious (the code is a very interesting, a very clever one, implementing, I suppose, an state or stack machine, look at Decrement and Increment methods).

Each node must have a package deployer service, running on another port (the same one for all nodes, default value is 55555). Go to run bin directory of Microsoft Robotics Developer Studio, and run the rundeployer.cmd:

 

Note: if you want to compile the service again, you must stop the deployer node: this is using the bin directory and the compilation conflicts with the package deployer host.

Now, we are ready to deploy our killer app: execute Run -> Run on distributed nodes. The system warns about to have deployers running:

And the magic begins: each node is contacted, and the application is deployer to them. A DSS node is started in each node, and its corresponding service instances are created. In this example, you will have two new old-DOS windows, running each DSS node. This is my second node window:

Now, each node show its scribbling window, and the fun begins.

Notes

I’m not using partners in this example. Instead, I’m put notification of events, in this case, new lines, so other components can receive such notification. You can write other components, other than scribble windows, to receive and process the lines. One idea is to use the coordinates of a new line to control a robot motor, or to persists the lines produced with a new persistence component. To control a robot motor, you should play with VPL, without rewriting the service component code.

Conclusions

Using VPL you can compose new applications from preexisting components. You can create your own activities, too. And using node diagrams (in VPL or using the DSS Manifest Editor) with the packager deployer, we can distribute the application in many physical nodes, in any convenient manner. But you must define each node and component: there is no provisioning of have a grid, so we can clone and distribute a working component. A gridified DSS application can be written, but in my opinion, it should be created with custom code. Idea to explore: write a custom control application that, using package deployer, deploys tasks to nodes in a grid.

All these tools opened my brain. Writing DSS service component is a new way of thinking and programming, something I want to explore for years: how to write objects with one-way message processing, in a distributed and parallel way (AjAgents was motivated by that objective). I envision some system that model Minsky ideas about the human mind.

Well, it’s a new brave world. Dorothy: we are not in Kansas anymore… ;-)

Angel “Java” Lopez
http://www.ajlopez.com/en

The Shocking Blue Green Theme. Blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.

Join 65 other followers