Angel \”Java\” Lopez on Blog

June 27, 2008

The early history of Smalltalk

Filed under: Programming Languages, Smalltalk, Software Development — ajlopez @ 5:42 pm

Today I wanted to comment a text, in my opinion, a “must be read” by all those of us dedicated to software development. It is a writing of Alan Kay, about the early history of Smalltalk:

Smallhistory.pdf

It is an excellent text to read, and that has influences simply beyond Smalltalk. It is a history of how some ideas were arising, within the American community of software development. How the programming with objects was arising, and how the form to interact with the present computers was taking form.

Read, for example, how Kay detects  some germinal ideas (data along with behaviour) in developments of the Air Force, that today we would see very remote of the OOP. Read on his contact with Lisp, and Seymour Papert. Read on the internal problems of Xerox, the competition with DEC, how some ideas were almost generated by chance. Read on Simula, Euler, IPL predecessor of Lisp. Read on the Sketchpad of Sutherland (who I found by first time in some historical revision of the Scientific American). It is a delicious and enlighten reading, at least for me.

Kay has been having an idea for years, that I share: the idea that the machine, and computing in general, must serve to us to expand our human capacities. Excellent idea. It is what of some form also today we are reaching using Internet. Today, branches of the knowledge and human actions, have been leveraged by software, the hardware and everything what it has happened in our profession in the last decades.

Years ago, people guessed that the space trips were going to revolutionize human history. That has still not happened. But of some form subproduct of the cold war and the space race, the development of the computing science (we remember its modern beginnings in the second war, and the appearance of the cybernetics impelled by military subjects) and of Internet, is what it has caused a change, that is reaching to great part of the humanity.

To read the history of Kay is indispensable to be understanding what it has happened. Somebody that has said ” the best form to dominate the future, is inventing it” (approximated phrase, reads the text, to see where it arose exactly).

Angel ” Java” Lopez
http://www.ajlopez.com/en

June 22, 2008

Distributed Agents and Fractals using DSS/VPL

Last week I wrote the base of an application running distributed agents over DSS/VPL, interchanging arbitrary messages, with automatic load balancing. You can read the details at:

Distributed Agents using DSS/VPL

Today, I extended the example with a new project, Fractal:

You can download from my Skydrive.

It has two DSS Service Components, one is the Calculator: it calculates a sector of the Mandelbrot fractal. The other component is the Renderer, that has a form to control and show the results of the calculation. The message that has the info of the sector to calculate is:

public class SectorInfo : MessagePayload { public double RealMinimum { get; set; } public double ImgMinimum { get; set; } public double Delta { get; set; } public int FromX { get; set; } public int FromY { get; set; } public int Width { get; set; } public int Height { get; set; } public int MaxIterations { get; set; } public int MaxValue { get; set; } }

Another class is the message that returns the calculation:

public class Sector : MessagePayload { public int FromX { get; set; } public int FromY { get; set; } public int Width { get; set; } public int Height { get; set; } public int[] Values { get; set; } }

The calculator splits the sector to calculate, if it too big. It can calculate all in only one step, but it is interesting to split the sector to generate more messages:

 

private void Calculate(AgentMessage msg) { LogInfo("Entering Calculator with Action: " + msg.Action); SectorInfo sectorInfo = (SectorInfo) msg.Payload; LogInfo(String.Format("X {0} Y {1} Width {2} Height {3}", sectorInfo.FromX, sectorInfo.FromY, sectorInfo.Width, sectorInfo.Height)); if (sectorInfo.Width > 100 && sectorInfo.Height > 100) SplitSector(sectorInfo); else CalculateSector(sectorInfo); }

As in the previous example, you can run this from a VPL diagram, FractalVpl1:

There is only one renderer, and two calculators agents. If you launch this VPL program, a window appears. This is the initial window content (after pressing the Calculate button):

You can drag and move the mouse to select a new region, or use the buttons to zoom in and out. Reset backs to the initial position. You can resize the window and invoke Calculate again. New colors button changes the color palette in use.

There is another VPL program, FractalVpl2, that it can be used to run the same example in a distributed way. It has a diagram with two AgentHosts:

and two nodes:

You must compile the VPL example and run as distributed nodes, using the rundeployer.cmd (see my previous post for details).

These some of the drawings the system produces:

 

Enjoy!

Angel “Java” Lopez
http://www.ajlopez.com/en

June 21, 2008

Messages everywhere

Recently, I wrote a post about Message Passing Interface:

Message Passing Interface, CCR, DSS, and Pure MPI.NET

I used to pass message between agents in my example of a web crawler:

Distributed Agents using DSS/VPL

The passing of messages between components, agents, objects, is a feature that deserves more study. I guess we could write new kinds of applications, using one-way message passing, so we could abandon the call-a-method current way of doing things. Let’s explore first, some message passing applications (not only one-way style).

In the Wikipedia entry about Message Passing:

http://en.wikipedia.org/wiki/Message_passing

we can read Alan Kay opinion:

Alan Kay has suggested that he placed too much emphasis on objects themselves and not enough on the messages being sent between them. Message passing enables extreme late binding in systems.

If you develop systems with Smalltalk, Self or alikes, you’ll notice that the message is a first citizen, in many implementations, a full object, not only a way of call a method.

There is another place for use message passing. For 25 years, QNX operating systems uses the message passing paradigm to run a real time kernel.

I found this interview at Dr. Dobb’s to Sebastien Marineau-Mes, VP of engeneering at QNX:

Real Time OS in Today s World

Sebastien talks about the use of QNX in current market, and the challenge that multi-core machine could create on legacy code.

Remember: all the QNX kernel is based on message passing, although its messages are not one-way, and the passing is optimized to not incurr in loss of performance (more details, see QNX at Wikipedia). I see that many of these challenges and opportunities could be translated to the use, not only to multi-core, but to “multi-machines” in a grid. There many forces that are conspiring to bring these topics to current arena:

- We have a better understanding of agents, message passing and other ideas

- Normal hardware is cheap

- Each year, there are more application with higher level needs of scalabity (the user base will be the whole Internet in the near future, for non-trivial app)

- Many application must interact with other application in the same enterprise or in the net, and messages are an effective way to do that.

- In order to face that challenges, we must begin to abandon n-tier-only apps, to a cloud, grid or “something alike” schema.

I could imagine languages and technologies, based on message passing features. That is one of the reasons I’ve been involved exploring simples and crazy ideas with AjMessages, AjAgents, and Distributed Software Services during last months. I hope to write more about these subjects:

- Another example of distributed agents using DSS/VPL

- AjMessages ported to DSS (I had an implementation last year, but I never published it, I published the WCF only version)

- One-way messages implemented as primitives in AjTalk (Smalltalk-like interpreter)

- Deferred/Concurrent/LazyEvaluation/SomethingAlike implemented in AjBasic (using CCR?)

- Blog about a better-finished app running in a grid of DSS hosts (my team was working hard on this, last months).

Incidentally, you can read more about use cases applied in the real world using CCR/DDS, in Arvindra Sehmi’s post:

CCR/DSS Use Cases in the Enterprise

So many ideas…. only one life…. Should I begin to parallized myself? ajlopez-in-a-grid…. ;-)

Angel “Java” Lopez
http://www.ajlopez.com/en

June 15, 2008

Distributed Agents using DSS/VPL

In this post, I explore some ideas to implement distributed agents, leveraging the features from Decentrilized Software Services (DSS) and Visual Programming Language (VPL), included in Microsoft Robotics Developer Studio (I’m working with CTP 2.0 version). You can download the source code from my Skydrive:

AjDssAgents-0.1.zip

In my post:

Web Crawler example using DSS (Decentralized Software Services)

I wrote DSS service components orchestrated from VPL. In that example, there are a Dispatcher, a Resolver, a Downloader, and a Harvester components.

Let’s suppose we have many machines to run the web crawling process. We want to deploy and run MANY downloaders and harvesters, in a grid of machines, using automatic load balancing. The problem with VPL orchestration is that it doesn’t support load balancing out of the box. Then, I wrote an example where the components communicate each other, as agents, using special messages.

An agent, in this example, is a DSS service component, capable of receiving and processes appropiate incoming messages. It can send outgoing messages to other components. Instead of sending a message to one of the other components, an agent specify in the message the type of agent to which the message is forwarded.

Another specialized component, AgentHost, is in charge of receive such outgoing messages, and it forward them to a local or remote agent, according to its type.

The solution

The solution has three projects:

AjDssAgents contains the generic agent contract and types, and the concrete AgentHost implementation.

DecrementAgent and WebCrawler contain simple agents to use in the example. The web crawler code is similar to the implementation described in the post mentioned above.

The message

Agents interchanges messages, objects of type AgentMessage:

[DataContract] public class AgentMessage { [DataMember] public string From { get; set; } [DataMember] public string To { get; set; } [DataMember] public string Action { get; set; } [DataMember] public object Payload { get; set; } }

The From field indicates the origin of the message (I’m not using this field yet). The To field is the physical address (DSS address) of the target agents, or its logical type. In the current example, I’m using only logical types. Why a logical type? If a message has a To with value “WebCrawler/Dispatcher”, it will be forwarder to one agent that has that logical type.

How an agent knows what other agents are running and what are their logical types? It doesn’t. The component that keeps that information is the local singleton AgentHost. Each agent post their outgoing message to the local AgentHost, so, this components selects a target agent and forwards the message to it.

The Agents

Each agent is a DSS service component, with an address assigned when it is created. During the start of the agent, it sends to its local AgentHost a DSS message, indicating its address and its logical type (i.e., WebCrawler/Dispatcher). This is the way an AgentHost knows the agents that are running in its local DssHost process. See the starting code for the Dispatcher agent in WebCrawler example:

 

protected override void Start() { base.Start(); // Add service specific initialization here. _state.AgentType = "WebCrawler/Dispatcher"; host.NewNode newNode = new host.NewNode(new host.AgentInfo() { Address = this.ServiceInfo.Service, AgentType = _state.AgentType }); _hostPort.Post(newNode); }

The type of the agent is keep in its state.

This is a typical code, from Dispatcher agent, showing the treatment of an incoming message and the production of outgoing messages:

 

[ServiceHandler(ServiceHandlerBehavior.Concurrent)] public IEnumerator<ITask> PostMessageHandler(generic.PostMessage postMessage) { if (postMessage.Body.Action.Equals("Dispatch")) Dispatch(postMessage.Body); else if (postMessage.Body.Action.Equals("Resolve")) Resolve(postMessage.Body); postMessage.ResponsePort.Post(DefaultSubmitResponseType.Instance); yield break; } private void Dispatch(AgentMessage msg) { LogInfo("Entering Dispatcher with Action: " + msg.Action); LogInfo("URL: " + msg.Payload); DownloadTarget target = new DownloadTarget(); target.Uri = (string) msg.Payload; target.Depth = 1; AgentMessage postmsg = new AgentMessage() { Action = "Resolve", To = _state.AgentType, Payload = target }; host.PostMessage post = new host.PostMessage(postmsg); _hostPort.Post(post); } private void Resolve(AgentMessage msg) { LogInfo("Entering Dispatcher with Action: " + msg.Action); DownloadTarget downloadtarget = (DownloadTarget)msg.Payload; LogInfo("URL: " + downloadtarget.Uri + ", Depth: " + downloadtarget.Depth); DownloadTarget target = ProcessUrl(downloadtarget); if (target != null) { AgentMessage agentmsg = new AgentMessage() { To = "WebCrawler/Downloader", Action="Download", Payload = downloadtarget }; host.PostMessage postmsg = new host.PostMessage(agentmsg); _hostPort.Post(postmsg); } }

The AgentHost

There is one and only one AgentHost per running DssHost. The AgentHost receives new agent information (address and logical type), and keeps that information in its state.

It receives messages from local agents, and then it forward them to other local agents, or to a remote AgentHost. In the later case, it serialize the payload to a string, using XML serialization (a generic object cannot be send using the DSS generated proxies). This is the structure of a remote message:

 

[DataContract] public class RemoteAgentMessage { [DataMember] public string From { get; set; } [DataMember] public string To { get; set; } [DataMember] public string Action { get; set; } [DataMember] public string PayloadTypeName { get; set; } [DataMember] public string Payload { get; set; } }

Note that the remote message has an string Payload (XML serialization of the original payload), and its qualified type, so the target host can deserialize the payload to reconstruct the original object.

An AgentHost supports subscription. Other AgentHosts can subscribe to new agent informations. In general, if you have three machine, you start one AgentHost in each machine, and subcribe each agent host to the others. In this way, any AgentHost has all the information about the running agents, local and remote ones.

A distributed Web Crawler VPL example

The WebCrawlerVpl2 VPL example contains the diagram:

There are one Dispatcher, two Downloaders and two Harverters agents. The Dispatcher launch the initial URL to crawl, and keeps a list of downloaded URLs. A Downloader reads the content of each page to crawl. A Harvester examines that content and gets new links to download.

Note that they are two AgentHost, and they are related so each one sends new agent information to the other.

All these agents and components are distributed in two nodes:

Windows node will run on localhost:50000/50001, and Windows0 node uses localhost:50002/50003 as address. You can modify these settings, add more agents and nodes, without changing the code.

To run the distributed app, you must compile using the Build -> Compile as a Service option in VPL menu. Note: you must change the settings in VPL properties, now they are pointing to local directories in my machine:

VPL will show the compiling process:

After compiling the VPL example, go to MRDS DOS prompt, change to the bin directory, and launch the rundeployer.cmd:

I run the deployer in my local machine. If you plan to run the example in remote machines, you must start the deployer in each one.

Now, we are ready to run the web crawler. Select the Run -> Run on distributed nodes option, and the application will start. A dialog prompt for the URL to crawl. After entering a valid URL, the process begins to retrieve the pages in the site. You can see the first AgentHost state at:

http://localhost:50000/agenthost

There are three local agents and two remote ones.

In the other side, there is another AgentHost:

http://localhost:50002/agenthost0

See the difference: here are two local nodes, and three remote ones.

To see the advance of the process, point your explorer to

http://localhost:50000/console/output

Conclusions

With these ideas, we can implement grid-alike applications, running in many nodes. We lost the VPL orchestration, we can’t draw the road of the messages. But we gain load balancing and dynamic deploying. With some additional effort, we can write a controlling service, to starts and deploy the system in a new remote machine, even in the middle of a running process. The serialization of arbitrary objects is possible, but with custom serialization.

I could add subscription to messaging, in a future version. That is, an agent could receive some messages that are not for it, according to some subscription criteria. The suscriptions could be kept by the AgentHosts. When an AgentHost route an outcoming message, it could forward it to any interested local or remote agent.

Angel “Java” Lopez
http://www.ajlopez.com/en

June 9, 2008

Lisp-like interpreter using DSS and VPL

My sunday project was to write some primitives of a Lisp-like interpreter using DSS Service Components. The core interpreter was derived from my previous work on AjLisp, a Lisp interpreter (I had never published the code before). I write it using Visual Studio 2008 and the CTP v2.0 of Microsoft Robotics Developer Studio. I published the code on my Skydrive space as DssLisp.zip.

The solution

It has two projects: a class library named AjLisp (a revamped version of my original interpreter in VB.NET):

and a C# Dss Service project, with has many implemented service components:

It has no manifest. It is designed to use from VPL diagrams.

Most of these components receive or return SymbolicExpression, the base type of my interpreter. See the messages for Rest service Execute operation:

[DataContract()] public class ExecuteRequest { [DataMember] public SymbolicExpression Expression; } [DataContract] public class ExecuteResponse { [DataMember] public SymbolicExpression Result; }

To use this components, you must adjust the references and directories in the DSS project (it points to my local directories, and to my local Microsoft Robotics Developer Studio install).

Using VPL

There are three VPL programs included in the examples. They use the compiled components of the solution. I had a problem with these VPL programs. I couldn’t run them from the menu option Run. You must use Build -> Compile as a Service, and then, use the option Run -> Run Compiled Services. I guess the origin of the problem is that the diagrams maps special messages, not only primitive types or Strings. In order to successful compile, you must adjust the properties of diagrams in each VPL project:

The first one DssLispVpl1 calculate (first ‘(a b c)):

 

There are two components LispParser and LispToString, that help to convert from string to SymbolicExpression and from SymbolicExpression to string.

The second one is DssLispVpl2:

It uses LispAppend component to append two predefined lists.

The third one is a mini-interpreter DsspLispInterpreter:

I must improve the error treatment. Now, if an exception occurs inside the service process, no Fault response is generated, and a causality is raised.

I could implement some generic contracts, and reuse it in many components, for example, a GenericBinaryOperation component.

Problems

I failed to write an Activity, to implement a Reverse function. The activity called itself recursively, but the generated code returns at the last recursive invocation. The problem is caused by my use of a merge to return a value. It’s weird to explain clearly, after some hours trying to do the job, I abandoned the code. Another stone in the way was the fact that activities only accept predefined types in its messages and results. But in this case, I found a solution: generate the code, and modified the generated solution to accept my SymbolicExpression type as inputs and outputs.

One interesting thing I wrote, is the component Uncons, that has TWO expressions in its result response: it takes a list, and returns the first and the rest of that list. Having two members in the response, you can processes each one in parallel branchs of the diagram.

Conclusion

As in previous posts, I found the VPL orchestration as very powerful and interesting, but lacks the support of a generic object, and has problems to manage custom types. I think you can implements any functional languages primitives and make a VPL diagram for each program you want. VPL would have problems with recursion (or I didn’t found a workable solution to manage it). This exercise could be expanded to other ideas:

- Manage a pipeline processing a generic message (I discovered that we can have a generic object as a DataMember)
- Have load balance as a component
- Serialize/Deserialize objects between nodes
- Implement some workflow or application intergration patterns in VPL

Angel “Java” Lopez
http://www.ajlopez.com/en

 

The Shocking Blue Green Theme. Create a free website or blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.

Join 57 other followers