Creating a scalable crawler with ConcurrentBag

This recipe shows how to scale workload between a number of independent workers that both produce work and process it.

Getting ready

To step through this recipe, you will need Visual Studio 2012. There are no other prerequisites. The source code for this recipe can be found in BookSamples \Chapter6\Recipe4.

How to do it...

The following steps demonstrate how to scale workload between a number of independent workers that both produce work and process it:

1. Start Visual Studio 2012. Create a new C# Console Application project.

2. In the Program.cs file add the following using directives:

   using System;
   using System.Collections.Concurrent;
   using System.Collections.Generic;
   using System.Threading.Tasks;

3. Add the following code snippet below the Main method:

   static Dictionary<string, string[]> _contentEmulation = new Dictionary<string, string[]>();
   
   static async Task RunProgram()
   {
     var bag = new ConcurrentBag<CrawlingTask>();
   
     string[] urls...