Web Crawler using the new AjAgents

I wrote a new implementation of my project AjAgents, described in:

AjAgents: a new implementation

Now, I want to describe in this post, how I used it to write (again) a Web Crawler. Previous versions:

Web Crawler Using AjAgents and AjSharp
Distributed Web Crawler using AjMessages

This new version of the web crawler use the new interface:

    public interface IAgent<T>
        void Post(Action<T> action);

Instead of write a type that inherits or implements an agent, the agent is a wrapper applied to a simple type.

The new implementation of AjAgents, and a web crawler example (Project AjAgents.WebCrawler), can be downloaded from:


In one of my previous implementation, I had:

Now, I dropped dispatcher. I have three classes:

The resolver receives a link to download. It decides to process or not (maybe, the link was already processed; or it is a link that is not in the original address to process). If the link is accepted, the info is sent to the downloader. This object downloads the content, and then send it to a harvester (it could be forward to other objects, too). The harvester examines the content, and extracts new links to process, each one is sent to the resolver.

The above classes are .NET types. But the references they have are of type Agent<Downloader>, Agent<Harvester>, Agent<Resolver>. The code to build  an initial group of interconnected objects/agents is in Program.cs:

       static Agent<Resolver> MakeAgent()
            Resolver resolver = new Resolver();
            Harvester harvester = new Harvester();
            Downloader downloader = new Downloader();
            Agent<Resolver> aresolver = new Agent<Resolver>(resolver);
            Agent<Harvester> aharvester = new Agent<Harvester>(harvester);
            Agent<Downloader> adownloader = new Agent<Downloader>(downloader);
            resolver.Downloader = adownloader;
            harvester.Resolver = aresolver;
            downloader.Harvester = aharvester;
            return aresolver;


AjAgents.WebCrawler console application accepts a parameter, with the initial link:

AjAgents.WebCrawler http://ajlopez.zoomblog.com


This is a proof of concept example. I would improve it:

Agent<IDownloader>, load balancing its work with many Agent<Downloader> instances.

Agent<T> instantiated in different machines, implementing a distributed web crawler.

Keep tuned!

Angel “Java” Lopez



28 thoughts on “Web Crawler using the new AjAgents

  1. futuristic evening dresses by aidan mattox

    There are traditions you’ll be able to follow or break, means that as much as you. When you find which colours look best upon you and use them consistently so as to you peer better, feel great and also have more confidence. Whichever fabric, length, color or style dress the caretaker with the bride chooses, for any the sunshine wedding, she should dress formal and go with dazzle.

  2. quinceanera dresses big

    Hey! I know this is sort of off-topic however I needed to ask.
    Does managing a well-established blog like yours require a massive amount work?

    I’m completely new to running a blog but I do write in my diary on a daily basis. I’d like to start a blog so I can share my
    personal experience and thoughts online. Please let me know if
    you have any kind of suggestions or tips for brand new aspiring bloggers.

    Appreciate it!

  3. evening dresses guelph

    I do trust all of the ideas you’ve presented for your post. They’re really convincing and
    will certainly work. Nonetheless, the posts are too quick
    for newbies. Could you please prolong them a bit from next time?
    Thank you for the post.

  4. sweater dresses buy online

    Attractive portion of content. I just stumbled upon your website and in accession capital to assert that I acquire actually
    loved account your blog posts. Any way I’ll be subscribing on your augment or even I success you get admission to constantly rapidly.

  5. sweater dresses for toddlers

    Have you ever thought about publishing an ebook or guest authoring on
    other blogs? I have a blog based upon on the same
    ideas you discuss and would really like to have you share some stories/information.
    I know my visitors would appreciate your work. If you’re even remotely interested, feel free to shoot me an email.

  6. party wear jeans

    I used to be recommended this website by way of my cousin.
    I am not sure whether or not this put up is written
    by him as nobody else know such specific approximately my trouble.
    You’re incredible! Thanks!

  7. sweater dress after 40

    Howdy terrific blog! Does running a blog like this take a massive amount work?

    I’ve no understanding of computer programming however I was hoping to start my own blog soon. Anyhow, should you have any recommendations or tips for new blog owners please share. I understand this is off subject nevertheless I just had to ask. Appreciate it!

  8. fifa 13 ultimate team coin generator

    Have you ever considered creating an ebook or guest authoring on other
    websites? I have a blog based on the same ideas you discuss and would love to have
    you share some stories/information. I know my subscribers would enjoy your work.
    If you are even remotely interested, feel free to send me
    an e mail.

  9. Kenneth

    My partner and I stumbled over here by a different web page and thought I might as well check things
    out. I like what I see so now i am following you. Look forward to going over your web page for a second time.

  10. http://www.hygienecertificate.net

    I absolutely love your blog and find many
    of your post’s to be exactly I’m looking for. Does one
    offer guest writers to write content available for you?
    I wouldn’t mind writing a post or elaborating on a number of the subjects you write with regards to here.
    Again, awesome site!

  11. Free Wifi Router Software

    Are you wondering how Amazon can sell a device that can get online without monthly fees
    or a contract that locks you in. PTCL has been, and still is one of the main telecommunications
    companies in Pakistan and is kind of interesting know how this market has evolved of the last two decades and how PTCL do to contribute to
    the grow of the market, giving to the Pakistan population a good quality service and an improved network where of course are include the Wi –
    Fi services. Caution: Before inserting or removing the SIM card, you must disconnect the router from the
    power adapter.

  12. copy shoppy

    It’s amazing to pay a quick visit this website and reading the views of all mates concerning this article, while I am also zealous of getting familiarity.

  13. Pingback: nassau bahamas

  14. jugaron

     ¿Por donde seria posible apreciar un poco mas escrito en repacion a dicho tematica?
    Me sirve bastante y lo has publicacion muy perfectamente.

  15. http://voyagemadagascarmalagasytours.blogspot.com/

    This place was built by the Las Vegas Sand Corporation, the major activities where tourists interested at Marina Bay Sands are Las Vegas style casinos, night clubs, museums,
    traditional restaurants and theaters. When to travel Africa really depends on the type of animals you wish to see or
    the activity you would like to partake in. The Peruvian Paso is considered to be the
    smoothest riding horse on earth.

  16. Regina U. Orte

    Good post. I learn something new and challenging on websites I stumbleupon everyday.
    It will always be exciting to read through content from other
    writers and use a little something from other sites.

  17. Romona R. Siedel

    I’m really enjoying the design and layout of your site.
    It’s a very easy on the eyes which makes it much more pleasant for me to come here
    and visit more often. Did you hire out a developer to create your theme?
    Excellent work!

  18. Valarie Z. Friehauf

    Great post. I was checking continuously this blog and I’m
    impressed! Extremely helpful info particularly the last part :
    ) I care for such info much. I was seeking this particular information for a very long time.

    Thank you and good luck.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s