Angel \”Java\” Lopez on Blog

January 29, 2008

Grid Computing in the browser

Filed under: .NET, C Sharp, Grid Computing — ajlopez @ 9:35 am

Daniel Vaughan has published a very interesting project (LGPL license) at Codeproject:

Legion: Build your own virtual super computer with Silverlight

Daniel is one of the 40 CodeProject MVPs for 2008. He is a software developer based in Canberra Australia, and Prague in the Czech Republic. (Thanks to Arvindra Sehmi, that sends me the article’s link).

It’s a project that uses Silverlight, the new Microsoft technology that runs in the browser, exposing .NET framework to JavaScript and other languages (in the 2.0 version).

According to the CodeProject article:

Legion is a grid computing framework that uses the Silverlight CLR to execute user definable tasks. It provides grid-wide thread-safe operations for web clients. Client performance metrics, such as bandwidth and processor speed, may be used to tailor jobs. Also includes a WPF Manager application.

Recently, I posted about Agents in a grid. Legion puts the agent code inside the browser, using Silverlight as a host environment. The server implements a JsonGridService that is accesible via web services from client side. It serializes the results using JSON (JavaScript Object Notation). One example method from that web services (JsonGridService.asmx.cs):

 

[WebMethod] [ScriptMethod(ResponseFormat = ResponseFormat.Json)] public TaskDescriptor StartNewJob(Agent agent) { try { TaskDescriptor descriptor = GridManager.GetDescriptor(agent); return descriptor; } catch (Exception ex) { HandleException(ex.Message, agent, ex); } return null; }

An instance of TaskDescriptor has a Type and a Job. The Type is a string describing the full name of the class and assembly to load and run in the client. The compiled assembly must reside in the ClientBin directory in the web server application. Job instance is a bit more complex: it’s a message, containing an arbitrary object to process in the client.

The code to run in the browser must inherits from SlaveTask. The client obtains a TaskDescriptor, creates an object according to the Type in TaskDescriptor, and runs it:

 

static void LoadAndRunTask() { TaskDescriptor info = gridService.StartNewJob(CreateAgent()); if (!info.Enabled) { return; } log.Debug("LoadAndRunTask() for Job Id " + info.Job.Id); Type type = Type.GetType(info.TypeName, true, true); if (slaveTask != null) { /* Detach last task. */ slaveTask.Complete -= taskBase_Complete; slaveTask.ProgressChanged -= taskBase_ProgressChanged; } slaveTask = (SlaveTask)Activator.CreateInstance(type); slaveTask.Initialise(info); OnTaskChanged(EventArgs.Empty); slaveTask.Complete += taskBase_Complete; slaveTask.ProgressChanged += taskBase_ProgressChanged; Thread newThread= new Thread( delegate { slaveTask.RunInternal(); }); newThread.Priority = ThreadPriority.Lowest; newThread.Start(); // ThreadPool.QueueUserWorkItem( // delegate // { // slaveTask.RunInternal(); // } // ); progressTimer.Enabled = true; }

 Note the use of Type.GetType to load the type. Silverligth uses the ClientBin directory on server side as one of the source for assembly loading. That’s the trick. Activator.CreateInstance creates the instance of the SlaveTask, and a new thread is launched to run the RunInternal method on that instance (for some reason, Vaughan commented the code that used the ThreadPool; I guess that he prefers to manage the priority in code).

In the article, more artifacts are described, as a WPF application, the Legion Manager, that allows monitoring of the grid. But now, let’s examine some concepts and alternatives.

Gridifyng the browser

The great idea, from Vaughan, is to use the browser as a host application for grid node tasks. This idea allows the use of any machine as a grid node (altought Vaughan’s approach needs the support of Silverlight). In the following paragraph, we’ll back to basic, to analyze the full picture implied in this kind of solution.

In a grid computing application, we must resolve the following problems (I’m simplifying the scenario landscape: we could have inter node communications too):

- How to program the task to run (languages, technologies…)

One of the options to the first problem, is to use a specialized language, dedicated to grid computing. Another alternative is to use a main stream language and technology, like Java or .NET. An another one: use scripting/dynamic languages, now supported in Java 6 and in .NET DLR.

- How to inject code to the grid node

If the host node application is a browser, we can program in Java (applets reloaded!), or .NET (now with Silverlight), or even in Flash. To inject the code, the technology at browser can remotely load .jars or compiled assemblies, or it can receive string with source code in dynamic languages, and run it as is.

- How to send task data to the grid node

Well, the data can be serialized, JSON is one method, XML is another one. But, now, another question emerges: who is in charge in sending the data? The server can send the data, but this implies that the client has some listening method. Or the client can poll the server, asking for new task data to process. I don’t know if an Java applet can use a ServerSocket, or if a Silverlight code can open a listening socket. One option to explore, is to have a WCF duplex channel in Silverlight client. Today, the sure option is: the client poll the server. That is the way Legion works.

- How to send back result from the node to a server

This question is an easier one. The data is send using JSON, XML, any serialization technology, to a web service in the server.

Conclusion

Each year, the browser is a more powerful application. Gridifying the browser, using the browser execution capabilities, it’s a promising idea, that deserves more exploration. Security issues, serialization, and the decision about push/pull model on message, are the points to research in more detail.

The next time you open the browser, ask its name. It could be “Legion”…. ;-)

Angel “Java” Lopez
http://www.ajlopez.com/en
http://www.msmvps.com/lopez

January 18, 2008

QCon London 2008

I’ve received an email, from Floyd Marinescu, announcing this “mega” conference. They’ll cover all-about-software-development: agile, DSLs, SOA, SaaS, Cloud computing (I guess grid computing related), .NET, Java, Ruby, Rich Client, Scrum, XP, F#, REST…. It will be the DisneyWorld of all software developers! ;-)

Here’s the announce:

You are receiving this email because you opted-in to receive notifications related to QCon London, from the QCon London website. QCon London is taking place in just 8 weeks, March 12-14 and the conference editorial is nearing final. Since our last update in December, a number of important changes have been made to the conference:

  • Google, Yahoo!, Amazon, Salesforce.com now presenting in the Cloud Computing Track
  • MySpace.com, eBay, Magus confirmed in the Architectures you’ve always wondered about Track
  • XP founder Kent Beck, and Analysis Patterns & Refactoring Author Martin Fowler keynoting
  • Banking Architectures Track adds sessions from Merrill, Betfair, Credit Suisse
  • GoF Patterns Author & Eclipse Architect Erich Gamma presenting
  • and much more…

QCon is an enterprise software development conference for team leads, architects, and project managers covering Architecture & Design, Java, .NET, Ruby, SOA, Agile methodologies and other timely topics such as DSLs, cloud computing, and bank/finance architectures. Last year’s QCon London had almost 500 registrants, and was covered very heavily in blog space see QCon London blogger coverage & key take aways).  

The track themes for QCon London & track hosts are as follows:
Architectures You’ve Always Wondered About -  Hosted by Redmonk’s James Governor
The last 2 QCon’s featured: Amazon, Linked-In, Yahoo!, eBay, Second Life, and Orbitz architecture case studies
Domain Specific Languages in Practice – Hosted by ThoughtWorks’ Neal Ford
Takes DSLs to the next level by practical applications and tools that are useful today.
Effective Design
Translate a set of abstract ideas into working and functional software in an effective way.
Architectures in Agile – Hosted by POSA author Kevlin Henney
How to integrate the trade-offs related to aspects such as performance, security, scalability and modifiability into an agile process
Banking: Complex high volume/low latency architectures  – Hosted by John Davies & Alexis Richardson
The latest innovations as well as time-proven best practices that architects of banking & finance systems need to know.
The Cloud as the New Middleware Platform – Hosted by EAI Patterns Author Gregor Hohpe
Is the internet becoming the computer?
Implementing Scrum and XP- Hosted by Patterns Almanac author Linda Rising
Leading practitioners will present and explain how Scrum and XP are implemented in the most effective way.
Java Emerging Technologies – Hosted by Java Concurrency Author/spec lead Brian Goetz
The previous QCon covered: JRuby, Grails, Server-side OSGi, DSL development, Batch Processing
.NET in the Enterprise – Hosted by Matt Deacon
.NET has brought Microsoft’s platforms into many business-critical applications, back-office, and server-side solutions.
Programming Languages of Tomorrow
Erlang, F#,, Intentional and Scala. How can we best leverage them in our next software project?
SOA, REST and the Web – Hosted by InfoQ’s Lead SOA Editor Stefan Tilkov
REST & SOA, Internet Scale Integration, REST & WS Myths
Browser & Emerging Rich Client Technologies – Hosted by InfoQ lead RIA/Java editor Scott Delap
Silverlight vs. JavaFX/Consumer JRE vs. Adobe Flex/AIR vs. Google’s Ajax RIA stack
The Rise of Ruby
Learn how to best take advantage of what Ruby has to offer
The first two QCon were well received, below are some comments from bloggers who attended our most recent QCon:

  • David Forbes – Exhilirated after gorging on brain candy this week, I have a moment to reflect on what just happened. QCon was the right place to be. I can’t imagine where else I would have rather been. If I had the week to do again, I would probably march right down to the Westin…again.
  • Denis Bregeon – I was very happy with it. Most of the talks tickled my imagination and that is the primary thing I was looking for. Many others gave me details on more technical subjects that I wanted to learn about.
  • Srini Penchikala – I was at the QCon conference in San Francisco last week. It was a great experience to be there. I learned a lot not only from the presentation speakers and panelists but also from the attendees who came from different countries (England, Syria, Australia to name a few) and companies.
  • Alex Olaru – Great conference: excellent speakers, very relevant topics, just enough product pushing without it becoming annoying. All in all a conference I would highly recommend to any architect or project manager.
  • Ola Bini – Last week I attended QCon San Francisco, a conference organized by InfoQ and Trifork (the company behind JAOO). It must admit that I was very positively surprised. I had expected it to be good, but I was blown away by the quality of most presentations.
  • Martin van Vliet – All in all, this was a good conference and more than enough reason to look forward to the next QCon, next year in London.
  • Pete Lacey – A wonderful conference made better by being able to meet many people face to face for the first time…

See also past QCon/JAOO talks available online on InfoQ:

Registration for the 3 day conference is 1180£ until Feb 22nd, and 1220£ after March 11th. The conference will be held at the Queen Elizabeth II Center, like last year.  Join us at QCon London!

Angel “Java” Lopez
http://www.ajlopez.com/en

January 3, 2008

Agents in a Grid

Filed under: AjAgents, AjMessages, Grid Computing — ajlopez @ 9:31 am

Recently, I was involved in the development of an application that runs on a grid of diskless machines. I was an exciting project, and now, after the rush of shipping it, I want to write down some ideas to explore.

In this post, I use the term “agent”, that is a bit fuzzy. I won’t define precisely the concept, I want to use as a base term to refine in a future. By now, let explore some basic (naive?) ideas to better understanding of the problems related to agents and grid applications. Yes, some ideas presented here could be viewed as naive, but I feel that this is a needed exercise, to grasp the key concepts and problems to be solved in these kind of applications. At the end of this article, I’ll present some implementation suggestions.

I described some applications to run in a grid in my previous post:

Grid Computing Programming

Agent concepts

In this post, agent is a piece of software, with behaviour and state. It runs inside an agent host, a host application that provides the base services so an agent can “live” and run. An agent will be represented as in this drawing:

Agent communication patterns

There is a lot of literature about agents communication, from simple techniques to elaborated ones as contracts, bids, and more. We can have agents with beliefs, desires and intention. In this post, an agent is simpler: only an object with state, that can send messages. It can receive stimula from other agents and from the host environment.

The simplest communication pattern is composed of an agent sending a message to another agent:

Some notes:

- The sender agent knows the receiver agent. That is, some form of identity must be implemented. The message is not send to anyone: the sender intention is to send the message to a determined receiver.

- The message carries data, and it must be understood by the receiver, possibly invoking one of its methods.

- The sender doen’t wait for a response. It’s not interested in a return message or data.

- The agents could reside in different physical machines in the grid, in a transparent way.

During its life an agent can send many messages to differents target agents, each one is a known agent for the sender:

Sometimes, the sender agent will receive a message from the receiver agents, notifying some work done, or informing some processed data. Depending on the application, the response message could carry information to identify the original message:

In this case, the sender must be prepared to receive the response in an asynchronous way. This could be an interesting problem to solve: an agent can send many messages, and it would be better if it can still run without receiving all the responses in time. For example, in an board game application, one agent could delegate some tree branch exploring to other agents, and, after some time, it should be able to take a decision, without all the information.

Clouds in the grid sky

Now, another case: an agent could be interested in send a message, without forward it to a determined receptor. Instead, it could be send to a cloud of agents, so anyone interested in the message could have the opportunity of processing it.

(Ok, I’m a bit odd creative in my drawings… ;-)

This feature could be implemented using these strategies:

- An agent could send a message to a blackboard system.

- An agent could send a message to host application, indicating a topic (like in a message queue), so any subscribed agent will receive the message. An variant: only some subscriptors receive the message, depending on application settings.

- An agent could send a message targeting some service provider definition. A service provider is an agent that declares, at the beginning of its life, its capabilities and service that is capable to provide. 

As example, we could take a chess/board game application (I prefer the game of go). An agent in such application can send a message reclaiming to solve some attack position. In a blackboard system, it will publish the request in the blackboard. In a topic system, it deposits the message on topic “attackers”. In a service provider strategy, it sends the message to one or many AttackService provider.

As in the previous pattern, an agent can receive an asynchronous response, now from “the cloud”:

 

Agent Duplication

An agent has behaviour and state. If the agent must fork its jobs, it could take the way of duplicating itself:

It’s a bit strange action, but it could be useful, depending on application at hand.

Agents and the Grid

Each agent can be hosted in a grid node. The message sending mechanism must be able to send a remote message to another node. The host application keeps a list of agents by identity, and knows which one is local or remote. An acid test: a grid application with agents must be able to run all in one machine, or in grid, without changing the code or algorithm.

 

As in another grid implementations (discussed in my previous post), a central server is in charge of distributing the tasks between the grid nodes. That server could receive tasks from outside application or even, from other grid. “Grid as a Service” is a new buzzword to apply.

Moving agent

An agent can initiate its activities in one node. But at point, it can decide to continue the work in another node (the host application in that node could take that decision, too). Then, its state should be send from one node to another.

Note that the agent behaviour (perhaps implemented in compiled code) doesn’t travel.

Injecting behaviour

The behaviour of each kind of agent could be expressed in compiled code (.jar files in Java, assemblies in .NET). Other alternatives are possible: the behaviour could be specified in a specific scripting agent language.

If the behaviour is expressed in compiled form, one or more servers have the responsability of storing and distributing that components:

Implementation ideas

Most of these ideas can be implemented in any appropiate language/technology, like Java and .NET, that supports multi threading, remote invocation, message serialization, etc…

Last months I was working on my pet projects AjMessages and AjAgents, more info at previous posts:

AjMessages: a message processor

Agents using Concurrency and Coordination Runtime (CCR)

(I have a new project, named AjGrid, not published yet). For this post, I guess that AjAgents could be an appropiate implementation. AjMessages now has remote support, but it’s more oriented to a pipeline processing: it’s difficult to implement in such disposition some of the above ideas.

I must add some features to AjAgents (actually, AjAgents is a local, proof of concept implementation):

- Configuration: Load and creation of agents in runtime, from a configuration file, or on the fly, in the middle of a run.

- Remote assembly: So a grid node can be injected with new agents.

- Agent identification: To identify an agent in a unique way (an UID could be a solution).

- Message transport: Windows Communication Foundation is a candidate.

But another approach is to take Decentralized System Services (DSS) from Microsoft Robotics Studio. An agent can be implemented as a DSS Service, running in a DSS Host. The intermachine communication would be implemented using DSSP as protocol.

Stay tuned, to see some code in the coming weeks.

Angel “Java” Lopez
http://www.ajlopez.com/en

The Shocking Blue Green Theme Blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.

Join 56 other followers