Building a tiny web crawler

Let's crawl the web! For this recipe, we'll devise a little spider that'll look over webpages, searching for hyperlinks, and crawl them again for any other web pointers they happen to hold. It'll carry on with this process over and over again, recursively discovering more links, until the maximum depth of search is reached.

The crawler's job can incur exponential time usage as, for every single page, it'll have to initiate as many HTTP connections as hyperlinks it might be able to discover. If it had to do this sequentially, the overall time taken by the operation of crawling many levels of web indirections as it will have to explore only a single link at every step would be equal to the sum of the time it takes to process every single weblink, which would be quite high. That's why considering a concurrent approach might be the right approach in our case.

As it turns out, the web crawler deals with a host of slow internet connections...