Web Crawler using the new AjAgents

I wrote a new implementation of my project AjAgents, described in:

AjAgents: a new implementation

Now, I want to describe in this post, how I used it to write (again) a Web Crawler. Previous versions:

Web Crawler Using AjAgents and AjSharp
Distributed Web Crawler using AjMessages

This new version of the web crawler use the new interface:

    public interface IAgent<T>
    {
        void Post(Action<T> action);
    }

Instead of write a type that inherits or implements an agent, the agent is a wrapper applied to a simple type. 
The new implementation of AjAgents, and a web crawler example (Project AjAgents.WebCrawler), can be downloaded from:
http://code.google.com/p/ajcodekatas/source/browse/#svn/trunk/AjAgents
In one of my previous implementation, I had:
 
Now, I dropped dispatcher. I have three classes:
 

The resolver receives a link to download. It decides to process or not (maybe, the link was already processed; or it is a link that is not in the original address to process). If the link is accepted, the info is sent to the downloader. This object downloads the content, and then send it to a harvester (it could be forward to other objects, too). The harvester examines the content, and extracts new links to process, each one is sent to the resolver.
The above classes are .NET types. But the references they have are of type Agent<Downloader>, Agent<Harvester>, Agent<Resolver>. The code to build  an initial group of interconnected objects/agents is in Program.cs:
       static Agent<Resolver> MakeAgent()
        {
            Resolver resolver = new Resolver();
            Harvester harvester = new Harvester();
            Downloader downloader = new Downloader();
            Agent<Resolver> aresolver = new Agent<Resolver>(resolver);
            Agent<Harvester> aharvester = new Agent<Harvester>(harvester);
            Agent<Downloader> adownloader = new Agent<Downloader>(downloader);
            resolver.Downloader = adownloader;
            harvester.Resolver = aresolver;
            downloader.Harvester = aharvester;
            return aresolver;
        }
 
AjAgents.WebCrawler console application accepts a parameter, with the initial link:
AjAgents.WebCrawler http://ajlopez.zoomblog.com
Output:
 
This is a proof of concept example. I would improve it:
– Agent<IDownloader>, load balancing its work with many Agent<Downloader> instances. 
– Agent<T> instantiated in different machines, implementing a distributed web crawler.
Keep tuned!
Angel “Java” Lopez

  
http://www.ajlopez.com 
http://twitter.com/ajlopez