8. Data System Design Examples

Activity 8.01: Building the Complete System with Pipelines and Queues

Solution

1. Import the random and time standard libraries, as well as the Queue and Thread classes from their respective modules:

   from queue import Queue
   from threading import Thread
   import random
   import time

   We imported the modules that we will use to design our next mock system.

2. Initialize the mock dataset and put it into a queue, as shown in the following query:

   urls = ['url1-', 'url1-', 'url2-', 'url3-', 'url4-', \
   'url5-', 'url6-', 'url7-', 'url8-', 'url9-', 'url10-']
   seen = set()
   url_queue = Queue()
   for url in urls:
       url_queue.put(url)

   We created 11 mock URLs and a seen set to find duplicates. We then created a queue for our URLs and added each URL to the queue.

3. Set up queues for each of the components, as shown in the following...